Three small identical bodies each of mass m are moving in circular orbit around a fixed point with same angular velocity under their gravitational interaction. If the separation between any two bodies is R, the total energy possessed by the system is given by

$-\frac{3 G M^2}{2 R}$

In the figure, $O A=\frac{2}{3} R \cos 30°$

Net force towards O on the mass A

$2\left(\frac{GM^2}{R^2}\right) \cos 30^{\circ}=M \omega^2\left(\frac{2}{3} R \cos 30^{\circ}\right)$

$\Rightarrow \omega^2=\frac{3 GM}{R^3}$

Total energy 3[KE + PE] = $3\left[\frac{1}{2} Mv^2+\left(\frac{-G M . M}{R}\right)\right]$

$=3\left[\frac{1}{2} M\left(\frac{2}{3} R \cos 30\right)^2 \frac{3 G M}{R^3}-\frac{G M^2}{R}\right]=-\frac{3}{2} \frac{G M^2}{R}$