Given that, #tanA=ntanB rArr sinA/cosA=n*sinB/cosB.#
#rArr sinA/sinB=n*cosA/cosB.....................................<<1>>.#
Also given that, #sinA=msinB rArr sinA/sinB=m.....................<<2>>.#
Comparing #<<1>> and <<2>>,# we get,
# m=n*cosA/cosB," giving, "cosB=n/m*cosA......<<3>>.#
#<<2>> rArr sinB=1/m*sinA...................................<<2'>>.#
Now, using #<<2'>> and <<3>># in #cos^2B+sin^2B=1,# we get,
#rArrn^2/m^2*cos^2A+1/m^2*sin^2A=1.#
#rArr n^2cos^2A+sin^2A=m^2.#
#rArr n^2cos^2A+(1-cos^2A)=m^2.#
#rArr n^2cos^2A-cos^2A=m^2-1, i.e.,#
#rArr (n^2-1)cos^2A=(m^2-1).#
# rArr cos^2A=(m^2-1)/(n^2-1),# as desired!
Enjoy Maths.!