We start with simplifying the square root by factoring out #x^2#:
#int sqrt(x^2 - 3 x^4)\ dx = int x sqrt(1 - 3 x^2)\ dx#.
Now we substitute #x \mapsto g(t)# with
#g(t) = \frac{1}{sqrt(3)} cos t#,
#g'(t) = - \frac{1}{sqrt(3)} sin t#, and
#g^{-1}(x) = cos^{-1} ( sqrt(3) x )#.
This yields
#[ - \frac{1}{3} int sin t cos t sqrt(1 - cos^2 t)\ dt ]_{t = cos^{-1}(sqrt(3) x)} =#
#= [ - \frac{1}{3} int sin^2 t cos t \ dt ]_{t = cos^{-1}(sqrt(3) x)}#.
At this point, we may interpret #sin t# as an inner function (with derivative #cos t#), and thus write the integral as
#[ - \frac{1}{3} int u^2 \ du ]_{u = sin t = sin cos^{-1}(sqrt(3) x) = sqrt(1 - 3 x^2)} =#
#= [ - \frac{u^3}{9} + C ]_{u = sqrt(1 - 3 x^2)} =#
#= - \frac{sqrt(1 - 3 x^2)^3}{9} + C#.
