Note that in general for any n >=1:
sec^(2n+1) x = sec^(2n-1)xsec^2x
and:
sec^2x = d/dx tanx
so we can integrate by parts:
int sec^(2n+1)xdx = int sec^(2n-1)d(tanx)
int sec^(2n+1)xdx = sec^(2n-1)xtanx - int tanx d(sec^(2n-1)x)
int sec^(2n+1)xdx = sec^(2n-1)xtanx - (2n-1)int tan^2xsec(2n-1)x
Use now the trigonometric identity:
tan^2x = sec^2x -1
to get:
int sec^(2n+1)xdx = sec^(2n-1)xtanx + (2n-1)int (1-sec^2x)sec^(2n-1)x
and using the linearity of the integral:
int sec^(2n+1)xdx = sec^(2n-1)xtanx + (2n-1)int sec^(2n-1)xdx- (2n-1) int sec^(2n+1)xdx
the integral now appears on both sides and we can solve for it:
2nint sec^(2n+1)xdx = sec^(2n-1)xtanx + (2n-1)int sec^(2n-1)xdx
int sec^(2n+1)xdx = (sec^(2n-1)xtanx)/(2n) + (2n-1)/(2n)int sec^(2n-1)xdx
So, for n=2:
int sec^5xdx = (sec^3xtanx)/4 + 3/4int sec^3xdx
and for n=1
int sec^3xdx = (secxtanx)/2 + 1/2int secxdx
Now:
int secx dx = int secx (secx+tanx)/(secx+tanx) dx
int secx dx = int (sec^2x+secxtanx)/(secx+tanx) dx
int secx dx = int (d(secx+tanx))/(secx+tanx)
int secx dx = ln abs(secx+tanx) +C
Putting the partial results together:
int sec^5xdx = (2sec^3xtanx + 3secxtanx + 3ln abs(secx+tanx))/8 +C