We use,
#(a-b)^2=a^2-2ab+b^2#
and #a^2-b^2=(a+b)(a-b)#
to factorise the denominator
#x^4-8x^2+16=(x^2-4)^2#
#=(x+2)^2(x-2)^2#
So,
#(3x^2+12x-20)/(x^4-8x^2+16)=(3x^2+12x-20)/((x+2)^2(x-2)^2)#
#=A/(x+2)^2+B/(x+2)+C/(x-2)^2+D/(x-2)#
#=(A(x-2)^2+B(x+2)(x-2)^2+C(x+2)^2+D(x-2)(x+2)^2)/((x+2)^2(x-2)^2)#
So,
#3x^2+12x-20=A(x-2)^2+B(x+2)(x-2)^2+C(x+2)^2+D(x-2)(x+2)^2#
Let #x=2#, #=>#, #16=16C#, #=>#, #C=1#
Let #x=-2#, #=>#, #-32=16A#, #=>#, #A=-2#
Coefficients of #x^3#, #0=B+D#
And #-20=4A+8B+4C-8D#
#-20=-8+8B+4-8D#
#-16=8B-8D#
#B-D=-2#
So, #B=-1# and #D=1#
#int((5x^3+2x^2-12x-8)dx)/(x^4-8x^2+16)#
#==int(-2dx)/(x+2)^2+int(-1dx)/(x+2)+int(1dx)/(x-2)^2+int(1dx)/(x-2)#
#=2/(x+2)-ln(∣x+2∣)-1/(x-2)+ln(∣x-2∣)+C#