If $\int \frac{1}{a^2 \sin ^2 x+b^2 \cos ^2 x} d x=\frac{1}{12} \tan ^{-1}(3 \tan x)+C$, then the maximum value of $a \sin x+b \cos x$, is

$\sqrt{40}$

We have,

$I=\int \frac{1}{a^2 \sin ^2 x+b^2 \cos ^2 x} d x$

$\Rightarrow I =\int \frac{\sec ^2 x}{b^2+a^2 \tan ^2 x} d x$

$\Rightarrow I =\frac{1}{a} \int \frac{1}{b^2+(a \tan x)^2} d(a \tan x)$

$=\frac{1}{a b} \tan ^{-1}\left(\frac{a}{b} \tan x\right)+C$

∴  $a b=12$ and $\frac{a}{b}=3 \Rightarrow a^2=36 \Rightarrow a= \pm 6$

∴  $a b=12 \Rightarrow b= \pm 2$

Thus, we have

$a \sin x+b \cos x= \pm(6 \sin x+2 \cos x)$

We know that

$-\sqrt{a^2+b^2} \leq a  \sin x+b \cos x \leq \sqrt{a^2+b^2}$ for all $x$

∴  $-\sqrt{40} \leq \pm 6 \sin x \pm 2 \cos x \leq \sqrt{40}$ for all $x$.