Try letting x=sectheta. Note this implies that dx=secthetatanthetad theta.
Also, the bounds will change. Observe that x=sqrt2=>theta=pi/4 and x=2=>x=pi/3.
Then:
int_sqrt2^2dx/(x^5sqrt(x^2-1))=int_(pi/4)^(pi/3)(secthetatanthetad theta)/(sec^5thetasqrt(sec^2theta-1))
Recall the trigonometric identity sec^2theta-1=tan^2theta.
=int_(pi/4)^(pi/3)(secthetatanthetad theta)/(sec^5thetatantheta)
=int_(pi/4)^(pi/3)(d theta)/sec^4theta
=int_(pi/4)^(pi/3)cos^4thetad theta
From here, I would use the identity cos^2alpha=1/2(1+cos2alpha), or, in this case, cos^4alpha=1/4(1+cos2alpha)^2 to further simplify the integral. It will definitely be a little messy.