Consider a thin uniform spherical layer of mass M and radius R, if we consider a solid sphere of mass M and radius R, then the potential energy of gravitational interaction of matter forming this solid sphere is

$-\frac{3}{5} \frac{GM^2}{R}$

$d U=-\frac{G m d m}{r}$

$\Rightarrow d U=-\frac{G\left(\frac{4}{3} \pi r^3 \rho\right)\left(4 \pi r^2 d r \rho\right)}{r}$

$\Rightarrow d U=-\frac{16 \pi^2 G \rho^2}{3} r^4 d r$

$\Rightarrow U=-\frac{16}{3} \pi^2 G\left(\frac{M}{\frac{4}{3} \pi R^3}\right)^2 \int\limits_0^R r^4 d r$

$\Rightarrow U=-\left(\frac{16}{3} \pi^2 G\right)\left(\frac{M^2}{\frac{16}{9} \pi^2 R^6}\right)\left(\frac{R^5}{5}\right)$

$\Rightarrow U=-\frac{3}{5} \frac{G^2}{R}$