Let $f(x)=-x^3+a x^2+b x+5 \sin x \cos x$ be a decreasing function on the set R. Then, a and b satisfy

$a^2+3 b+15<0$

We have,

$f(x)=-x^3+a x^2+b x+\frac{5}{2} \sin 2 x$

$\Rightarrow f'(x)=-3 x^2+2 a x+b+5 \cos 2 x$

For f(x) to be decreasing on R, we must have

$f'(x)<0$ for all $x \in R$

$\Rightarrow -3 x^2+2 a x+b+5 \cos 2 x<0$ for all $x \in R$

$\Rightarrow -3 x^2+2 a x+b+5<0$ for all $x \in R$

$\Rightarrow 3 x^2-2 a x-b-5>0$ for all $x \in R$

$\Rightarrow 4 a^2-4 \times 3(-b-5)<0$                    [∵ Disc < 0]

$\Rightarrow a^2+3 b+5<0$