Sam: I don't know the numbers.

Pat: I don't know the numbers.

Sam: I don't know the numbers.

Pat: I don't know the numbers.

.

.

.

.

Sam and Pat continue like this until one of them realizes that it is impossible for either of them to determine the numbers. What is the smallest possible N for which this can happen?