Rewriting the denominator:
int(3x^2+5x-1)/(x^2+2x-1)^2dx=int(3x^2+5x-1)/{(x^2+2x+1)-2}^2dx=int(3x^2+5x-1)/{(x+1)^2-2}^2dx
Let u=x+1, implying that x=u-1 and du=dx.
=int(3(u-1)^2+5(u-1)-1)/(u^2-2)^2du=int(3u^2-u-3)/(u^2-2)^2du
Now let u=sqrt2sectheta. This implies that u^2-2=2sec^2theta-2=2tan^2theta and that du=sqrt2secthetatanthetad theta.
=int(3(2sec^2theta)-sqrt2sectheta-3)/(2tan^2theta)^2(sqrt2secthetatanthetad theta)
=sqrt2/4int(6sec^3theta-sqrt2sec^2theta-3sectheta)/tan^3thetad theta
Multiplying through by cos^3theta/cos^3theta:
=1/(2sqrt2)int(6-sqrt2costheta-3cos^2theta)/sin^3thetad theta
=3/sqrt2intcsc^3thetad theta-1/2intcotthetacsc^2thetad theta-3/(2sqrt2)int(1-sin^2theta)/sin^3thetad theta
Note that 3/sqrt2-3/(2sqrt2)=3/(2sqrt2):
=3/(2sqrt2)intcsc^3thetad theta-1/2intcotthetacsc^2thetad theta+3/(2sqrt2)intcscthetad theta
Find the method for integrating csc^3theta here.
For the second integral, let v=cottheta so dv=-csc^2thetad theta.
The integral of csctheta is well known. For its derivation, see here.
=3/(2sqrt2)(-1/2)(cotthetacsctheta+lnabs(cottheta+csctheta))+1/2intvdv+3/(2sqrt2)lnabs(csctheta-cottheta)
Note that 1/2intvdv=1/2(v^2/2)=v^2/4=cot^2theta/4.
=-3/(4sqrt2)cotthetacsctheta-3/(4sqrt2)lnabs(cottheta+csctheta)+3/(4sqrt2)lnabs(csctheta-cottheta)+1/4cot^2theta
Our substitution was u=sqrt2sectheta, implying that costheta=sqrt2/u, which is a right triangle where the side adjacent to theta is sqrt2, the hypotenuse is u, and the side opposite theta is sqrt(u^2-2).
Thus, csctheta=u/sqrt(u^2-2) and cottheta=sqrt2/sqrt(u^2-2).
Also note that -3/(4sqrt2)lnabs(cottheta+csctheta)+3/(4sqrt2)lnabs(csctheta-cottheta)=3/(4sqrt2)lnabs((csctheta-cottheta)/(cottheta+csctheta))=3/(4sqrt2)lnabs((1-costheta)/(1+costheta)).
=-3/(4sqrt2)(sqrt2u)/(u^2-2)+3/(4sqrt2)lnabs((1-sqrt2/u)/(1+sqrt2/u))+1/4(2/(u^2-2))
=-3/4(u/(u^2-2))+1/2(1/(u^2-2))+3/(4sqrt2)lnabs((u-sqrt2)/(u+sqrt2))
=(-3u+2)/(4(u^2-2))+3/(4sqrt2)lnabs((u-sqrt2)/(u+sqrt2))
With u=x+1:
=(-3x-1)/(4(x^2+2x-1))+3/(4sqrt2)lnabs((x-sqrt2+1)/(x+sqrt2+1))+C