Let;s expand this first :
y=(x-2y)^2-2x^2y-2y^2=y=(x−2y)2−2x2y−2y2=
x^2-4xy+4y^2-2x^2y-2y^2=x^2-4xy+2y^2-2x^2y=>x2−4xy+4y2−2x2y−2y2=x2−4xy+2y2−2x2y⇒
y=x^2-4xy+2y^2-2x^2yy=x2−4xy+2y2−2x2y
Now to swith to polar coordinates we do the following substitutions :
y=rsinthetay=rsinθ
x=rcosthetax=rcosθ
rsintheta=r^2cos^2theta-4r^2sinthetacostheta+2r^2sin^2theta-2r^3cos^2thetasinthetarsinθ=r2cos2θ−4r2sinθcosθ+2r2sin2θ−2r3cos2θsinθ
=>sintheta=rcos^2theta-2rsin2theta+2rsin^2theta-2r^2cos^2thetasintheta=>⇒sinθ=rcos2θ−2rsin2θ+2rsin2θ−2r2cos2θsinθ⇒
rcos^2theta-2rsin2theta+2rsin^2theta-2r^2cos^2thetasintheta-sintheta=0rcos2θ−2rsin2θ+2rsin2θ−2r2cos2θsinθ−sinθ=0