cos[tan^-1sin(cot^-1x)] is equal to

The correct answer is Option (1) → \( \sqrt{\displaystyle\frac{x^2+1}{x^2+2}} \)

Let \( \cot^{-1}x=\theta \), so \( \cot\theta=x \) with \( 0\lt\theta\lt\pi \), and \( \sin\theta\gt 0 \):

    \( \displaystyle \sin\theta=\frac{1}{\mathrm{cosec}\,\theta} \)

    \( \displaystyle =\frac{1}{\sqrt{1+\cot^2\theta}} \)

    \( \displaystyle =\frac{1}{\sqrt{1+x^2}} \)

Now let \( \tan^{-1}\left(\frac{1}{\sqrt{1+x^2}}\right)=\phi \), so \( \tan\phi=\frac{1}{\sqrt{1+x^2}} \) with \( 0\lt\phi\lt\frac{\pi}{2} \):

    \( \displaystyle \cos\phi=\frac{1}{\sqrt{1+\tan^2\phi}} \)

    \( \displaystyle =\frac{1}{\sqrt{1+\frac{1}{1+x^2}}} \)

    \( \displaystyle =\sqrt{\frac{1+x^2}{2+x^2}} \)

    \( \displaystyle \cos[\tan^{-1}\{\sin(\cot^{-1}x)\}]=\sqrt{\frac{x^2+1}{x^2+2}} \)

Shortcut: Put x = 0: \( \cot^{-1}0=\frac{\pi}{2} \), \( \sin\frac{\pi}{2}=1 \), \( \tan^{-1}1=\frac{\pi}{4} \), \( \cos\frac{\pi}{4}=\frac{1}{\sqrt{2}} \). Option (1) gives \( \sqrt{\frac{1}{2}} \); Option (2) gives \( \sqrt{2} \), and Options (3) and (4) give 0. Only Option (1) matches.