We know that ,
#"Distance between Polar Co-ordinates:"A(r_1,theta_1)and B(r_2,theta_2) # is
#color(red)(D=sqrt(r_1^2+r_2^2-2r_1r_2cos(theta_1-theta_2))...to(I)#
We have , #P_1(1,pi/6) and P_2(5,(3pi)/4)#.
So , #r_1=1 , r_2=5 , theta_1=pi/6 and theta_2=(3pi)/4#
#=>theta_1-theta_2=pi/6-(3pi)/4=(2pi-9pi)/12=-(7pi)/12=-105^circ#
#=>cos(theta_1-theta_2)=cos(-105^circ)#
#=>cos(theta_1-theta_2)=cos(105^circ)to[becausecos(-theta)=costheta)#
#"Using : " color(red)((I)# we get
#D=sqrt(1^2+5^2-2(1)(5)cos105^circ)#
#=>D=sqrt(1+25-10*cos105^circ)#
#=>D=sqrt(26-10cos105^circ)#
#=>D~~5.34679#