The value of \(\sin\left[2\cot^{-1}\left(\displaystyle\frac{-5}{12}\right)\right]\) is:
Answer & explanation
Correct answer: option 2
The correct answer is Option (2) → \(\frac{-120}{169}\)
Let \(\theta = \cot^{-1}\left(\frac{-5}{12}\right)\). The principal value range of \(\cot^{-1}\) is \((0, \pi)\), and the value is negative, so \(\theta\) lies in \(\left(\frac{\pi}{2}, \pi\right)\), the second quadrant.
\( \displaystyle \cot\theta = \frac{-5}{12} \)
The hypotenuse of the 5, 12 triangle is \(\sqrt{25+144} = 13\). In the second quadrant \(\sin\theta \gt 0\) and \(\cos\theta \lt 0\):
\( \displaystyle \sin\theta = \frac{12}{13},\quad \cos\theta = \frac{-5}{13} \)
By the double-angle formula:
\( \displaystyle \sin 2\theta = 2\sin\theta\cos\theta = 2\times\frac{12}{13}\times\frac{-5}{13} \)
\( \displaystyle = \frac{-120}{169} \)