Let $f(x) = x^3 + ax^2 + bx + 5\sin^2 x$ be an increasing function on the set R. Then

$a^2 – 3b + 15 < 0$

Given, $f(x) = x^3 + ax^2 + bx + 5\sin^2 x$ is increasing on R.

Then $f'(x)> 0$ for all $x ∈R$

$⇒3x^2+2ax+b+5(2\sin x\cos x)>0$ for all $x ∈R$

$⇒3x^2+2ax+b+5\sin2x>0$ for all $x ∈R$ ....(i)

$3x^2+2ax+b-5<3x^2+2ax+b+5\sin2x<3x^2+2ax+b+5$

$⇒3x^2+2ax+(b-5)<0$

[as $ax^2+bx+c>0⇒a>0$ and $D<0$]

so, $4(a^2-3b+15)<0,∀\,x∈R$

$⇒a^2 – 3b + 15 < 0,∀\,x∈R$