The spheres are 6 units apart. The triangle containing the centers of the spheres and the edge of the circle has lengths 4-5-6. The radius of the circle is the height to the side of length 6.
By Heron's formula the triangle has area 15sqrt(7)/4 and so the radius is 5sqrt(7)/4/