I think the easiest way to get started, would be to note that sin^2(x) = 1-cos^2(x)
So looking at this integral, we have intsin^5(x)* cos^2(x) dx = intsin(x) * (sin^2(x))^2 * cos^2(x)dx = intsin(x) * (1-cos^2(x))^2 * cos^2(x)dx
Why do it like this?
Because you usually want to look ahead at what is going to happen, after you have found the derivative of u.
In this case, using the fact that sin^2(x) = 1-cos^2(x), I can rewrite and use u' to get rid of either one of trigonometric functions. Had I chosen to get rid of all instances of cos(x), I would have ended up with a square root. Getting rid of sin(x) on the other hand, you can already see from now that I won't get any nasty square roots, and it looks like it's just gonne be adding together powers of u. That is much simpler to do.
So, let's continue :)
u = cos(x)
(du)/dx = -sin(x) iff dx = (du)/-sin(x)
int(sin(x)*(1-u^2)^2*u^2)/-sin(x)du = -intu^2*(1-u^2)^2du
=-intu^2*(1+u^4-2u^2)du
=-intu^2+u^6-2u^4du
= -intu^2du-intu^6du-int-2u^4du
=-1/3u^3-1/7u^7+2/5u^5+k
=2/5cos^5(x)-1/3cos^3(x)-1/7cos^7(x)+k
Here's a double check: https://www.wolframalpha.com/input/?i=derivative(2%2F5cos%5E5(x)-1%2F3cos%5E3(x)-1%2F7cos%5E7(x)%2Bk)
