Simplify $\tan^{-1} \left[ \frac{a \cos x - b \sin x}{b \cos x + a \sin x} \right]$, if $\frac{a}{b} \tan x > -1$

$\tan^{-1} \left( \frac{a}{b} \right) - x$

The correct answer is Option (2) → $\tan^{-1} \left( \frac{a}{b} \right) - x$ ##

We have,

$\tan^{-1} \left[ \frac{a \cos x - b \sin x}{b \cos x + a \sin x} \right] = \tan^{-1} \left[ \frac{\frac{a \cos x - b \sin x}{b \cos x}}{\frac{b \cos x + a \sin x}{b \cos x}} \right] = \tan^{-1} \left[ \frac{\frac{a}{b} - \tan x}{1 + \frac{a}{b} \tan x} \right]$

$= \tan^{-1} \frac{a}{b} - \tan^{-1} (\tan x) = \tan^{-1} \frac{a}{b} - x$