Create an origin at the center of the hoop, Y= vertically down,

X= to the right. Assume that the balls are in the order A, B, C,

on the hoop (from left to right). Further name the angles from the

center of the each ball to the Y-axis (vertical) as Ta, Tb, Tc,

respectively. The Y coordinate of the c.g. of the 3-ball system then is:

Ycg = d*(a^3 cos(Ta) + b^3 cos(Tb) + c^3 cos(Tc)) / (a^3 + b^3 + c^3).

The density constant, k, can be ignored as would appear as a linear

term in both the numerator and denominator.

As Bractals points out, the idea is to maximize Ycg. This is equivalent

to saying that gravity will pull the 3-ball system as low as possible on

the hoop. The 3 unknowns are Ta, Tb, and Tc. However they are

related. Note that the radius of each ball forms a chord on the hoop.

Draw an isosceles triangle using the radius of the ball as the base,

each of the other sides is length d. The angle opposite the base can

then be shown to be arcos (1-(r^2)/2/d^2) where r is the

radius of the ball in question. Therefore for any position of the three

balls, as placed in order, above:

Ta = Tb – arcos(1-(a^2)/2/(d^2)) – arcos(1-(b^2/2/(d^2)) &

Tc = Tb + arcos(1-(b^2)/2/(d^2)) + arcos(1-(c^2)/2/(d^2))

(angles are measured counter clockwise from the 6 o'clock position

(on the Y-axis) on the hoop)

Substituting these into the equation for Ycg, the only variable is Tb.

I'll leave the calculus to someone else, but I did create an Excel

spreadsheet with the above formulae, and using Solver to maximize

Ycg it seems to work for some easily understood cases:

a=b=c=X à Ta=-Tc, Tb=0, Ycg is smaller for increasing X

a=c=X, b=Y --> Ta=-Tc, Tb=0

a=0, b=c à Ta=Tb=-Tc