The upper ball is trying to tip over the tube while the
weight of the tube and the lower ball are trying to keep
it from tipping over. Both balls exert an equal horizontal
force (say F) on the tube, but in opposite directions.
Therefore, the tube will tip over if
F*[l + (u+l)*sin(theta)] > F*l + T*(r+t) (1)
where u and l are the radii of the upper and lower balls
respectively and T, r, and t are the weight, internal radius,
and thickness of the tube respectively. Theta is the angle
between the table top and the line determined by the ball's
centers and is given by
cos(theta) = (2*r-w)/w (2)
where w = u+l (also, one third of the length of the tube).
To determine the force F, we look at the horizontal and
vertical forces exerted on the upper ball
F = B*cos(theta)
U = B*sin(theta)
where B is the force exerted on the upper ball by the lower
ball along the line determined by their centers and U is the
weight of the upper ball. The equations in (3) combine to give
F*sin(theta) = U*cos(theta) (4)
Combining the inequality of (1) with equations (2) and (4) we get
that the tube will tip over if
U*(2*r-w) - T*(r+t) > 0 (5)
We have the following relation between the weights
U 3 4*u^3
--- = ------------------------ = --------------- (6)
T 3*w*PI*[(r+t)^2 - r^2] 9*w*t*(2*r+t)
Combining equation (6) with inequality (5) we get that
the tube will tip over if
9*w*t*(2*r+t)*(r+t) - 4*u^3*(2*r-w) < 0 (7)
This is just a quadratic inequality in r.
Plugging in u = 2.6 we get the tube staying upright if
r < 4.90437... or r > 7.96488...
Plugging in u = 3.4 we get the tube tipping over if
3.41938... < r < 25.54468...
Therefore, since the physical problem requires r < w = 6
(or both the balls will come to rest touching the table),
r minimum = 3.41938... and r maximum = 4.90437...