We can use chain rule here.
Let y=sin^(-1)(g(x)), where g(x)=xsqrt(x-1)-sqrtxsqrt(1-x^2)
then (dy)/(dx)=(dy)/(dg(x))xx(dg(x))/dx
Now (dy)/(dg(x))=1/sqrt(1-g(x)^2)
as g(x)^2=(xsqrt(x-1)-sqrtxsqrt(1-x^2))^2
= x^2(x-1)+x(1-x^2)-2xsqrtxsqrt((x-1)(1-x^2))
= x^3-x^2+x-x^3-2xsqrt(x^2-x^4-x+x^3)
= x-x^2-2xsqrt(x^2-x^4-x+x^3)
and (dg)/(dx)=sqrt(x-1)+x*1/(2sqrt(x-1))-sqrtx*(-2x)/(2sqrt(1-x^2))-1/(2sqrtx)sqrt(1-x^2)
= sqrt(x-1)+x/(2sqrt(x-1))+(2xsqrtx)/(2sqrt(1-x^2))-sqrt(1-x^2)/(2sqrtx)
Hence (dy)/(dx)=(sqrt(x-1)+x/(2sqrt(x-1))+(2xsqrtx)/(2sqrt(1-x^2))-sqrt(1-x^2)/(2sqrtx))/sqrt(1-x+x^2+2xsqrt(x^2-x^4-x+x^3)