I hope your question was ax2+2hxy+by2 is rotated by 900 about origin and after rotation what should be it's new equation.
to solve this question you must be aware of shifting of origin which says if the function is rotated by angle A then let it's new axis be X and Y such that
X=xcosA - ysinA
Y=xsinA + ycosA
X=-y & Y=x
now in the equation of pair of straight line replace x by Y and y by -X you will get
aY2 - 2hXY + bX2
hence the new equation is ay2-2hxy+bx2
hope I answered your question
feel free to ask any doubt related to this.....