Given a circular billiards table with center O,
radius r, and no pockets. A cue ball (with a
radius negligible compared to r) is placed at
point A such that 0<|OA|≤r. The ball is struck
and bounces off the cushion twice before
returning to point A.

How was the direction of the shot determined?

Call the point where the ball first strikes the cushion P.

Call angle OPA theta. (O being the center of the circle and A the location of the ball)

Angle OAP is the complement of twice theta due to two symmetries: the reflection symmetry at P and the symmetry of the diagram as a whole with the second reflection point symmetric with the first, and AO extended being perpendicular to the path of the ball between the two bounces(reflections).

Erect a perpendicular from O to AP, at a point we shall call X.

Let the radius be 1 wlog.

OX = sin(theta)

Since angle OAP = angle OAX is the complement of twice theta, angle AOX is twice theta, and the hypotenuse of the small triangle is sin(theta) / cos(2*theta). That hypotenuse is OA, and as the radius for these calculations is 1, this is the ratio of the distance of the ball to the radius of the table.

We then just need to solve for theta in sin(theta) / cos(2*theta) = |OA|/r and the value 90° - 2*theta will be angle OAP that we need to aim away from the center when sending the ball on its way.

DECLARE FUNCTION asin# (x#)
DEFDBL A-Z
CLS
pi = ATN(1) * 4
FOR dist = .1# TO .90001# STEP .05#
   th = 20 * pi / 180
   DO
     prev = th
     th = (asin(dist * COS(2 * th)) + th) / 2
   LOOP WHILE ABS((th - prev) / th) > .000000001#
   PRINT USING "#.## ###.#####"; dist; 90 - 2 * th * 180 / pi
NEXT dist

FUNCTION asin (x)
  IF ABS(x) = 1 THEN
   asin = pi / 2 * SGN(x)
  ELSE
   asin = ATN(x / SQR(1 - x * x))
  END IF
END FUNCTION

solves the equation iteratively and produces the following table:

          angle (°)
       relative to center 
|OA|/r    of table

0.10      78.74320
0.15      73.46478
0.20      68.54471
0.25      64.02425
0.30      59.91054
0.35      56.18788
0.40      52.82781
0.45      49.79634
0.50      47.05860
0.55      44.58132
0.60      42.33409
0.65      40.28979
0.70      38.42454
0.75      36.71751
0.80      35.15062
0.85      33.70817
0.90      32.37655

These results agree with what I found using Geometers' Sketchpad.

I don't know how convenient this would be in the pool hall.

Perhaps Bractals has a more practical method in mind.