Remember that
#cos^3(x) = cos^2(x)*cos(x)#
#cos^2(x) = 1 - sin^2(x)#
#cos^3(x) = (1-sin^2(x))cos(x)#
so
#I = int_(pi/8)^((11pi)/12)cos^3(x) - sin(x)dx#
equals
#int_(pi/8)^((11pi)/12)(1-sin^2(x))cos(x)dx - int_(pi/8)^((11pi)/12)sin(x)dx#
On the left integral if we say #u = sin(x)#, we have
#du = cos(x) dx#
So rewriting (and
#int_(sin(pi/8))^(sin((11pi)/12))(1-u^2)du - int_(pi/8)^((11pi)/12)sin(x)dx#
Both integrals are standard so all we gotta do now is evaluate them
#(sin(x)-sin^3(x)/3+cos(x))|_(pi/8;(11pi)/12)#
#(sqrt(3)-1)/(2 sqrt(2))-(sqrt(3)-1)^3/(48 sqrt(2))-(1+sqrt(3))/(2 sqrt(2)) - (sqrt(125/144+161/(144 sqrt(2))))#
#I ~= 2.001#