Let f(x) be a function defined by 

$f(x)=\left(a b-a^2-2\right) x-\int\limits_0^x\left(\cos ^4 t+\sin ^2 t-2\right) d t$

If f(x) is a decreasing function for all $x \in R$ and $a \in R$, where a is independent of x, then

none of these

We have,

$f(x) =\left(a b-a^2-2\right) x-\int\limits_0^x\left(\cos ^4 t+\sin ^2 t-2\right) d t$

$\Rightarrow f'(x)=\left(a b-a^2-2\right)-\left(\cos ^4 x+\sin ^2 x-2\right)$

For f(x) to be decreasing, we must have

f'(x) < 0 for all $x \in R$

$\Rightarrow\left(a b-a^2-2\right)-\left(\cos ^4 x+\sin ^2 x-2\right)<0$ for all $x \in R$ and for all $a \in R$

$\Rightarrow\left(a b-a^2-2\right)-\left(\cos ^4 x-\cos ^2 x-1\right)<0$ for all $x \in R$ and for all $a \in R$

$\Rightarrow\left(a b-a^2-2\right)<\left(\cos ^2 x-\frac{1}{2}\right)^2-\frac{5}{4}$ for all $x \in R$ and for all $a \in R$

$\Rightarrow\left(a b-a^2-\frac{3}{4}\right)<\left(\cos ^2 x-\frac{1}{2}\right)^2$ for all $x \in R$ and for all $a \in R$

$\Rightarrow a b-a^2-\frac{3}{4}<0$ for all $a \in R$            $\left[∵ 0 \leq\left(\cos ^2 x-\frac{1}{2}\right)^2 \leq \frac{1}{4}\right]$

$\Rightarrow 4 a^2-4 a b+3>0$ for all $a \in R$

$\Rightarrow 16 b^2-48<0$

$\Rightarrow b^2<-3<0 \Rightarrow-\sqrt{3}<b<\sqrt{3} \Rightarrow b \in(-\sqrt{3}, \sqrt{3})$