A laser is fired from the origin along the line y=ax and reflects off the mirror.

Find the equation of the line along which the reflection travels.

Lines

y = ax

and

y = mx + b

meet at their simultaneous solution:

ax = mx + b

x(a-m) = b

x = b / (a-m); y = ab / (a-m)

The slope of the mirror line is m, and that of the light ray is a. That means that the upward angles are arctan(m) and arctan(a) respectively. Depending on how you draw the picture (whether the mirror is above or below the origin), you either must add to the angle of m the difference between the angle of m and the angle of a, or subtract from angle m the difference between the angle of a and the angle of m.  Either way, the upward angle of the reflected ray is twice that of the mirror from which is subtracted the angle of the original ray, so the formula for the angle is

2 arctan(m) - arctan(a).

Taking the tangent of this angle gives the slope of the required line. The point-slope form can be used as we know the point of intersection and the slope:

y = tan(2*arctan(m) - arctan(a)) * (x - b/(a-m)) + ab/(a-m)

The following program tests this out successfully:

Private Sub cmdStart_Click()
 Cls
 scl = Me.ScaleHeight / 22: y0 = Me.ScaleHeight / 2: x0 = y0
 Line (x0, 0)-(x0, 2 * y0)
 Line (0, y0)-(2 * x0, y0)
 a = Val(txtA): m = Val(txtM): b = Val(txtB)
 p1y = a * (-10): p2y = a * 10
 Line (x0 - 10 * scl, y0 - p1y * scl)-(x0 + 10 * scl, y0 - p2y * scl)
 p1y = m * (-10) + b: p2y = m * 10 + b
 Line (x0 - 10 * scl, y0 - p1y * scl)-(x0 + 10 * scl, y0 - p2y * scl)
 newM = Tan(2 * Atn(m) - Atn(a)): newB = a * b / (a - m)
 p1y = newM * (-10 - b / (a - m)) + newB
 p2y = newM * (10 - b / (a - m)) + newB
 Line (x0 - 10 * scl, y0 - p1y * scl)-(x0 + 10 * scl, y0 - p2y * scl)
End Sub