$\int \frac{d x}{\sqrt{(x-a)(b-x)}}=$

$2 \sin ^{-1} \sqrt{\left(\frac{x-a}{b-a}\right)}+c$

Put $x=a^2 \cos ^2 \theta+b \sin ^2 \theta$ the given integral becomes.

$I=\int \frac{2(b-a) \sin \theta \cos \theta d \theta}{\left\{\left(a \cos ^2 \theta+b \sin ^2 \theta-a\right)\left(a \cos ^2 \theta+b \sin ^2 \theta-b\right\}^{\frac{1}{2}}\right.}$

$=\frac{2(b-a) \sin \theta \cos \theta d \theta}{(b-a) \sin \theta \cos \theta}=\left(\frac{b-a}{b-a}\right) \int 2 d \theta =2 \theta+c=2 \sin ^{-1} \sqrt{\left(\frac{x-a}{b-a}\right)}+c$

Hence (1) is the correct answer.