Expand the square, using the F.O.I.L. method:
3 = x^2 + 8xy + 16y^2 + 3x
Substitute rcos(theta) for x and rsin(theta) for y:
3 = (rcos(theta))^2 + 8(rcos(theta))(rsin(theta)) + 16(rsin(theta))^2 + 3(rcos(theta))
This can be written as a quadratic in r:
0 = (cos^2(theta) + 8cos(theta)sin(theta) + 16sin^2(theta))r^2 + 3cos(theta)r - 3
The coefficient for r^2 factors into a square:
0 = (cos(theta) + 4sin(theta))^2r^2 + 3cos(theta)r - 3
Use the quadratic formula:
r = (-b +-sqrt(b^2 - 4(a)(c)))/(2a)
where:
a = (cos(theta) + 4sin(theta))^2
b = 3cos(theta)
c = -3
Also, we must change the +- to only +, because negative values for r do not make sense.
r = (-3cos(theta) +sqrt(9cos^2(theta) + 12(cos(theta) + 4sin(theta))^2))/(2(cos(theta) + 4sin(theta))^2)