Let $f(x)=x^3+a x^2+b x+5 \sin ^2 x$ be an increasing function on the set R. Then, a and b satisfy

$a^2-3 b+15<0$

$f(x)=x^3+a x^2+b x+5 \sin ^2 x$ is increasing on R

$\Rightarrow f'(x)>0$ for all $x \in R$

$\Rightarrow 3 x^2+2 a x+b+5 \sin 2 x>0$ for all $x \in R$

$\Rightarrow 3 x^2+2 a x+(b-5)>0$ for all $x \in R$

$\Rightarrow (2 a)^2-4 \times 3 \times(b-5)<0 \Rightarrow a^2-3 b+15<0$