Multiply both sides by #cos^2x#
#6sin^2x - 4sin^2xcos^2x = cos^2x#
Use the pythagorean identity #sin^2x = 1-cos^2x#
#6(1-cos^2x) -4(1-cos^2x)cos^2x = cos^2x#
Simplify
#6 - 6cos^2x -4cos^2x +4cos^4x = cos^2x#
#6 - 10cos^2x + 4cos^4x = cos^2x#
Subsitute #cos^2x# for #z#
#6 - 10z + 4z^2 = z#
Make one side equal zero
#4z^2 - 11z + 6 = 0#
Solve using the quadratic equation
#z = (11 +-sqrt(121 -4*4*6))/8#
#z = (11 +- sqrt(25))/8 = (11+-5)/8#
#z_+ = (11+5)/8 = 16/8 = 2#
#z_(-) =(11-5)/8 = 6/8 = 3/4 #
Subsitute #z# for #cos^2x#
#cos^2(x) = 2# is invalid because the square cosine has a range of #0 <= y <= 1#
Which leaves us with #cos^2x = 3/4 rarr cosx = +-sqrt(3)/2#
The cosine has one of those values on the first quadrant when #theta = 30º#, one of those values on the second quadrant when #theta = 180º - 30º = 150º#, one of those values on the third quadrant when #theta = 180º+30º = 210º# and one of the values on the fourth quadrant when #theta = 360º - 30º = 330º#
So, the set of solutions #S# is #S = {30º,150º,210º,330º}#