Since cos(pi-x)=-cos(x),
cos(frac{5pi}{6})=-cos(pi/6).
cos(pi/6)=frac{sqrt{3}}{2} can be seen from the geometry of a 30,60,90 degree triangle, which is half of an equilateral triangle. Therefore,
cos(frac{5pi}{6})=-frac{sqrt{3}}{2}.
Since sin^{-1}(-x)=-sin^{-1}(x), and sin^{-1}(frac{sqrt{3}}{2})=frac{pi}{3} (which can be seen from the same 30,60,90 degree triangle),
sin^{-1}(cos(frac{5pi}{6}))=sin^{-1}(-frac{sqrt{3}}{2})
=-sin^{-1}(frac{sqrt{3}}{2})=-frac{pi}{3}