The simplest way to approach this is through geometry. Since the angle of both hands is moving at a constant rate, the relative angle between them is also changing at a constant rate. So, we can imagine that the hour hand doesn't move at all, and the minute hand moves (radially) at a speed of its original rate less the normal rate of the hour hand.
The problem, then, is to determine at what point the distance between the tips is increasing at the greatest rate. Draw an imaginary segment between the two hands; call this distance d. Since the minute hand is moving at a constant rate relative to the 'stationary' hour hand, the moment when d is increasing most rapidly is simply when the tips are moving directly away from each other. This occurs when d forms a right angle with the minute hand.
Thus, we have a right triangle with the hour hand (4) as the hypotenuse, the minute hand (3) as one leg, and we want to find the length of the other leg. Pythagoras helps us out here:
d = √(4² - 3²)
d = √7
We can also solve the real-time problem the "classic" way, using calculus.
Let the angle between the hands be Θ. Then the distance between the tips is √(25 - 24 cos Θ).
Differentiating, the rate of increase is 12 dΘ/dt sin Θ / √(25 - 24 cos Θ).
Differentiating again, this is a maximum when cos Θ (25 - 24 cos Θ) = 12 sin2Θ, and hence when 12 cos2Θ - 25 cos Θ + 12 = 0, or (3 cos Θ - 4)(4 cos Θ - 3) = 0.
We cannot have cos Θ > 1, so the maximum is when cos Θ = 3/4 and the distance is √(25 - 24×3/4) = √7, the same answer we got through the geometric analysis.