Create an origin at the center of the hoop, Y= vertically down,
X= to the right. Assume that the balls are in the order A, B, C,
on the hoop (from left to right). Further name the angles from the
center of the each ball to the Y-axis (vertical) as Ta, Tb, Tc,
respectively. The Y coordinate of the c.g. of the 3-ball system then is:
Ycg = d*(a^3 cos(Ta) + b^3 cos(Tb) + c^3 cos(Tc)) / (a^3 + b^3 + c^3).
The density constant, k, can be ignored as would appear as a linear
term in both the numerator and denominator.
As Bractals points out, the idea is to maximize Ycg. This is equivalent
to saying that gravity will pull the 3-ball system as low as possible on
the hoop. The 3 unknowns are Ta, Tb, and Tc. However they are
related. Note that the radius of each ball forms a chord on the hoop.
Draw an isosceles triangle using the radius of the ball as the base,
each of the other sides is length d. The angle opposite the base can
then be shown to be arcos (1-(r^2)/2/d^2) where r is the
radius of the ball in question. Therefore for any position of the three
balls, as placed in order, above:
Ta = Tb – arcos(1-(a^2)/2/(d^2)) – arcos(1-(b^2/2/(d^2)) &
Tc = Tb + arcos(1-(b^2)/2/(d^2)) + arcos(1-(c^2)/2/(d^2))
(angles are measured counter clockwise from the 6 o'clock position
(on the Y-axis) on the hoop)
Substituting these into the equation for Ycg, the only variable is Tb.
I'll leave the calculus to someone else, but I did create an Excel
spreadsheet with the above formulae, and using Solver to maximize
Ycg it seems to work for some easily understood cases:
a=b=c=X à Ta=-Tc, Tb=0, Ycg is smaller for increasing X
a=c=X, b=Y --> Ta=-Tc, Tb=0
a=0, b=c à Ta=Tb=-Tc