Try letting x=secthetax=secθ. Note this implies that dx=secthetatanthetad thetadx=secθtanθdθ.
Also, the bounds will change. Observe that x=sqrt2=>theta=pi/4x=√2⇒θ=π4 and x=2=>x=pi/3x=2⇒x=π3.
Then:
int_sqrt2^2dx/(x^5sqrt(x^2-1))=int_(pi/4)^(pi/3)(secthetatanthetad theta)/(sec^5thetasqrt(sec^2theta-1))∫2√2dxx5√x2−1=∫π3π4secθtanθdθsec5θ√sec2θ−1
Recall the trigonometric identity sec^2theta-1=tan^2thetasec2θ−1=tan2θ.
=int_(pi/4)^(pi/3)(secthetatanthetad theta)/(sec^5thetatantheta)=∫π3π4secθtanθdθsec5θtanθ
=int_(pi/4)^(pi/3)(d theta)/sec^4theta=∫π3π4dθsec4θ
=int_(pi/4)^(pi/3)cos^4thetad theta=∫π3π4cos4θdθ
From here, I would use the identity cos^2alpha=1/2(1+cos2alpha)cos2α=12(1+cos2α), or, in this case, cos^4alpha=1/4(1+cos2alpha)^2cos4α=14(1+cos2α)2 to further simplify the integral. It will definitely be a little messy.