We assume the value of the acceleration due to gravity to be constant for small height and the gravitational potential energy of a body of mass \(m\) at a height say \(y\) from the Earth's surface in a Earth-body system is \(U=mgy\) (note that the Earth also has a potential to fall towards the body because gravitational potential energy is the shared property of both the body and the Earth).
We know from the conservation of mechanical energy that the work done by the gravitational force on a body is the negative of the change in potential energy of the body that is, \(W = -\Delta U = U_1 - U_2\).
In gravitational force attracts the body towards the centre of the Earth which is given by
\[F = - \frac{{G{M_E}{\kern 1pt} m}}{{{r^2}}} \tag{1} \label{1}\]
where \(M_E\) is the mass of the Earth. The negative sign is due to the force towards the centre of the Earth that is the force is attractive. The body undergoes a very small displacement of magnitude \(dr\) away from the Earth and therefore the small work \(dW\) done by the Earth's gravitational fore is
\[dW = - \frac{{G{M_E}{\kern 1pt} m}}{{{r^2}}}dr\]
If \(r_1\) is the initial position of the body and \(r_2\) is the final position after the particular displacement directly away from the Earth, the total work done from \(r_1\) to \(r_2\) can be obtained by integrating the above equation within the limits \(r_1\) and \(r_2\) so,
\[\begin{align*} W &= - G{M_E}{\kern 1pt} m\int\limits_{{r_1}}^{{r_2}} {\frac{1}{{{r^2}}}dr} = - G{M_E}{\kern 1pt} m\left( {\frac{1}{{{r_1}}} - \frac{1}{{{r_2}}}} \right)\\ \text{or},\quad W &= - \left( {\frac{{G{M_E}{\kern 1pt} m}}{{{r_1}}} - \frac{{G{M_E}{\kern 1pt} m}}{{{r_2}}}} \right) \tag{2} \label{2} \end{align*}\]
We know that the work done by the gravitational force is \(W = - ({U_2} - {U_1}) = -\Delta U\) which is the initial potential energy minus the final potential energy. Rewriting Equation \eqref{2}, you'll obtain \(W = -(U_1 - U_2) = \Delta U\) and hence you can write the expression(formula) for the gravitational potential energy as
\[U = - \frac{{G{M_E}{\kern 1pt} m}}{{{r}}} \tag{3} \label{3}\]
Since \(r_1 < r_2\), the work done by the gravitational force in Equation \eqref{2} is negative which is as it should be because the force acts in the opposite direction of the displacement. The negative sign in Equation \eqref{3} is essential because the work done by the gravitational force is the negative of the change in potential energy and if \({U_1} = - {{G{M_E}{\kern 1pt} m}}/{{{r_1}}}\) and \({U_2} = - {{G{M_E}{\kern 1pt} m}}/{{{r_2}}}\), the work done by the gravitational force is \(W = - \left( { - {{G{M_E}{\kern 1pt} m}}/{{{r_2}}} + {{G{M_E}{\kern 1pt} m}}/{{{r_1}}}} \right) = - \left( {{{G{M_E}{\kern 1pt} m}}/{{{r_1}}} - {{G{M_E}{\kern 1pt} m}}/{{{r_2}}}} \right)\) which is the same as Equation \eqref{2}.
Note that the body is outside the Earth and we can rewrite Equation \eqref{3} in terms of the Earth's radius \(R_E\) and height \(h\) above the Earth's surface as \(r = R_E + h\):
\[U = - \frac{{G{M_E}{\kern 1pt} m}}{{{R_E} + h}}\]
In the above equation the gravitational potential energy decreases as the height \(h\) decreases that is becomes more negative and the gravitational potential energy increases as the height \(h\) increases that is becomes less negative. The gravitational potential energy is zero when the body is infinitely far from the centre of the Earth at \(h = \infty \). Even if the expression for the gravitational potential energy is negative in Eq. \eqref{3}, the only physically significant quantity is the difference in potential energies.
What is gravitational potential energy of a system of a point mass and a spherical shell?
From this discussion we'll know that the Newton's law of gravitation is equally valid for larger bodies having spherical mass distributions. Consider that a point mass \(m\) is outside a spherical shell at a distance \(r\) from the centre of the shell. Let the spherical shell has total mass \(M\), radius \(R\) and total area \(A\). Here we find the gravitational potential energy of the system of the spherical shell and the point mass.
To do that we consider an infinitesimally thin element of the spherical shell (ring element) of mass \(dM\) and thickness \(dl = Rd\theta\) (angle should be measured in radians) on the spherical shell such that every particle in the ring is at a distance \(l\) from the point mass as shown in Figure 2. The gravitational potential energy of the ring element and point mass system is
\[dU = - \frac{{GmdM}}{l} \tag{4} \label{4}\]
Note that the shell element in Figure 2 is made somewhat larger for clarity but it's a very thin shell element. The area of the shell element is \(dA = 2\pi R^2 \sin \theta d\theta\). Note that the area of the shell element is the circumference \(2 \pi R\sin \theta\) times the thickness \(dl = Rd \theta\). Now the ratio of the mass of the shell element to the total mass of the shell is equal to the area of the shell element to the total area of the shell, and you'll get,
\[\begin{align*} \frac{{dM}}{M} &= \frac{{dA}}{A} = \frac{{2\pi {R^2}\sin \theta d\theta }}{{4\pi {R^2}}}\\ {\rm{or,}}\quad dM & = \frac{1}{2}M\sin \theta d\theta \tag{5} \label{5} \end{align*}\]
Putting the value of \(dM\) from Eq. \eqref{5} into Eq. \eqref{4}, you'll get
\[dU = - \frac{{GmM\sin \theta d\theta }}{{2l}} \tag{6} \label{6}\]
In Figure 2 \(OA = R\cos \theta\) and therefore, \({l^2} = {(r - R\cos \theta )^2} + {(R\cos \theta )^2}\) or \({l^2} = {r^2} - 2rR\cos \theta + {R^2}\) and
\[\begin{align*} ldl = rR\sin \theta d\theta \\ \text{or},l = \frac{{rR\sin \theta d\theta }}{{dl}} \tag{7} \label{7} \end{align*}\]
You can rewrite Eq. \eqref{6} by substituting the value of \(l\) from Eq. \eqref{7} and you'll obtain
\[dU = - \frac{{GmMdl}}{{2rR}} \tag{8} \label{8}\]
The gravitational potential energy of the system of the spherical shell and the point mass is obtained by integrating the above expression from \(r - R\) to \(r + R\):
\[U = - \frac{{GmM}}{{2rR}}\int\limits_{r - R}^{r + R} {dl} = - \frac{{GmM}}{r} \tag{9} \label{9}\]
The above equation shows that the gravitational potential energy of the system of the spherical shell and the point mass is the same as if the whole mass of the spherical shell were concentrated at the centre of the shell. Since the gravitational force is \(F = - dU/dr\), you can obtain the expression for the force as
\[F = GmM\frac{{d{r^{ - 1}}}}{{dr}} = -\frac{{GmM}}{{{r^2}}} \tag{10} \label{10}\]
If the point mass in Figure 2 is inside the spherical shell at a distance \(r\) from its centre, the limits of integration while integrating Eq. \eqref{8} change from \(R-r\) to \(R+r\) and the total gravitational potential energy of the system becomes
\[U = - \frac{{GmM}}{{2rR}}\int\limits_{R - r}^{R + r} {dl} = - \frac{{GmM}}{R} \tag{11} \label{11}\]
In Eq. \eqref{11} there is \(R\) instead of \(r\) so the gravitational potential energy when the point mass is inside the shell is constant and the force on the point mass is zero.
What is gravitational potential energy of a system of a point mass and a solid sphere?
The Eq. \eqref{9} gives the gravitational potential energy of the system of the spherical shell and the point mass outside the shell. But you know that a solid sphere is the combination of large number of spherical shells concentric at the center of the sphere. If one such shell has mass \(M_i\), then the total gravitational potential energy of the system of the solid sphere and the point mass outside the sphere is
\[U = -\frac{{Gm\sum\limits_{i = 1}^n {{M_i}} }}{r} \tag{12} \label{12}\]
which shows that the gravitational potential energy of the system of the solid sphere and the point mass outside the sphere is the sum of the gravitational potential energies due to all possible spherical shell and point mass systems. In Eq. \eqref{12}, the quantity \({\sum\limits_{i = 1}^n {{M_i}} }\) is the sum of the masses of all possible spherical shells of the solid sphere in which each spherical shell makes the corresponding spherical shell and point mass system and produces the corresponding gravitational potential energy.
But if the point mass is inside the solid sphere, the gravitational potential energy of the interaction of the solid sphere and the point mass is not the same expression as given by Eq. \eqref{12}. Consider a point mass \(m\) inside a solid sphere of mass \(M\) and radius \(R\) as shown in Fig7.4.
The distance of the point mass from the center of the sphere is \(r\). In this situation the gravitational potential energy of the interaction of the solid sphere and the point mass is equal to the gravitational potential energy of the interaction of the inner sphere of radius \(r\) and the point mass plus the gravitational potential energy of the interaction of the outer spherical shells having radius greater than \(r\) and the point mass. First the mass of the inner sphere of radius \(r\) is \({M_{{\rm{is}}}} = \rho \frac{4}{3}\pi {r^3}\) where \(\rho \) is the density of the solid sphere. Note that mass is the product of volume and density. Now the gravitational potential energy of the interaction of the point mass \(m\) and the inner sphere is
\[{U_1} = - \frac{{Gm{M_{{\rm{is}}}}}}{r} = - \frac{4}{3}Gm\rho \pi {r^2}\]
Now consider a particular spherical shell of radius \({r'}\) such that \(r' > r\) and the thickness of the shell is \(dr'\). The volume of the shell is \(dV = 4\pi {{r'}^2}dr'\) and therefore its mass is \(4\rho \pi {{r'}^2}dr'\). The gravitational potential energy of the interaction due to all the outer shells (\(r < r'\)) and the point mass is found by integrating the gravitational potential energy of the interaction between the single shell and the point mass within the limits from \(r' = r\) to \(r' = R\). First the gravitational potential energy of the interaction between the spherical shell and point mass is
\[dU_2 = - \frac{{Gm(4\pi {{r'}^2}dr'\rho )}}{{r'}} = - 4Gm\rho \pi r'dr'\]
And the total gravitational potential energy of the interaction of all the outer shells and the point mass is
\[{U_2} = - 4Gm\rho \pi \int\limits_r^R {r'dr'} =- 2Gm\rho \pi ({R^2} - {r^2})\]
Finally the expression(formula) for the gravitational potential energy of the interaction between the solid sphere and the point mass is the sum of \(U_1\) and \(U_2\) and hence,
\[\begin{align*} U &= {U_1} + {U_2}\\ & = - \frac{4}{3}Gm\rho \pi {r^2} - 2Gm\rho \pi ({R^2} - {r^2})\\ & = - 2Gm\rho \pi \left( {\frac{{3{R^2} - {r^2}}}{3}} \right)\\ &= - G\frac{4}{3}\pi {R^3}m\left( {\frac{{3{R^2} - {r^2}}}{{2{R^3}}}} \right)\\ {\rm{or,}}\quad U &= - GMm\left( {\frac{{3{R^2} - {r^2}}}{{2{R^3}}}} \right) \tag{13} \label{13} \end{align*}\]
At the center of the solid sphere, \(r = 0\) and Eq. \eqref{13} gives \(U = - {{2GMm}}/{{3R}}\) but \( - {{GMm}}/{{R}}\) is the gravitational potential energy when the point mass is on the surface of the solid sphere. Therefore, the gravitational potential energy of the interaction of the solid sphere and the point mass at the centre of the sphere is \(-2/3\) times the gravitational potential energy on the surface of the sphere. Another case is when \(r = R\), \(U = - {{GMm}}/{R}\) which is again the gravitational potential energy when the point mass is on the surface of the solid sphere. The force of the interaction between the solid sphere and the point mass inside the sphere can be obtained by using \(F = -dU/dr\) and so,
\[F = - \frac{{GMm}}{{{R^3}}}r \tag{14} \label{14}\]
Eq. \eqref{14} shows that the force on the point mass is directly proportional to its distance from the centre of the sphere.