#I=intx^2sec^-1(x)dx#
Let #x=sectheta#. Then, #theta=sec^-1(x)# and #dx=secthetatanthetad theta#.
#I=intsec^2theta(theta)(secthetatanthetad theta)=intthetasec^3thetatanthetad theta#
Note that #intsec^3thetatanthetad theta=intsec^2thetacolor(white).d(sec theta)=1/3sec^3theta#. Then, we can easily perform integration by parts on #I# by letting:
#u=theta" "=>" "du=d theta#
#dv=sec^3thetatanthetad theta" "=>" "v=1/3sec^3theta#
So:
#I=1/3thetasec^3theta-1/3intsec^3thetad theta" "" "" "color(red)star#
We can solve this integral with integration by parts again, now letting:
#u=sectheta" "=>" "du=secthetatantheta d theta#
#dv=sec^2thetad theta" "=>" "v=tantheta#
So:
#I=1/3thetasec^3theta-1/3(secthetatantheta-intsecthetatan^2thetad theta)#
Letting #tan^2theta=sec^2theta-1# this becomes:
#I=1/3thetasec^3theta-1/3secthetatantheta+1/3intsectheta(sec^2theta-1)#
#color(white)I=1/3thetasec^3theta-1/3secthetatantheta+1/3intsec^3thetad theta-1/3intsecthetad theta#
#color(white)I=1/3thetasec^3theta-1/3secthetatantheta-1/3lnabs(sectheta+tantheta)+1/3intsec^3thetad theta#
Earlier (#color(red)star#) we noted that #I=1/3thetasec^3theta-1/3intsec^3thetad theta#, so we can set this to be equivalent to the expression we just found.
#1/3thetasec^3theta-1/3intsec^3thetad theta=1/3thetasec^3theta-1/3secthetatantheta-1/3lnabs(sectheta+tantheta)+1/3intsec^3thetad theta#
Rearranging:
#-2/3intsec^3thetad theta=-1/3secthetatantheta-1/3lnabs(sectheta+tantheta)#
Which gives:
#-1/3intsec^3thetad theta=-1/6secthetatantheta-1/6lnabs(sectheta+tantheta)#
Plugging this into #color(red)star# yields:
#I=1/3thetasec^3theta-1/6secthetatantheta-1/6lnabs(sectheta+tantheta)#
We can now return to our original variable #x# using #x=sectheta#, #tantheta=sqrt(sec^2theta-1)=sqrt(x^2-1)# and #theta=sec^-1(x)#:
#I=1/3x^3sec^-1(x)-1/6xsqrt(x^2-1)-1/6lnabs(x+sqrt(x^2-1))+C#