If a diagram is drawn, showing a typical case (one in which x is large enough so that the vertices of the inscribed square lie on the sides of the circumscribing square), call the distance from one vertex of the inscribed square to the nearest vertex of the circumscribing (full) square, less 1, y (the subtraction of 1 making it the distance to the nearest vertex of the cut square). Then the similarity of triangles gives:

1/(x-y-2) = (1+y) / (x-y-1)

y comes out, using a quadratic formula:

y = (x - 2 + sqrt(x^2 - 4*x))/2

indicating that x needs to be at least 4 for this situation to apply, otherwise we'd get complex numbers.

In this case, x >= 4, the area of the inscribed square is

(x-y-1)^2 + (1+y)^2

Substitution for y in terms of x, and simplification, is left as an exercise for the reader.

When x<4 (but still, of course x>=2), the area is simpler:

2*(x-2)^2

as the inner square is at a 45° angle to the outer, with the inner corners of the four corner squares at the centers of the inner square's sides, so the sides of the angled square are sqrt(2) times the sides of the square formed by those inner corners.

Hopefully, when x=4 the two formulae agree:

The latter formula is easy: 2*(4-2)^2 = 8

By the first formula:

y = (4 - 2 + sqrt(0)) / 2  = 1

area = (4-1-1)^2 + (1+1)^2 = 4+4 = 8

voila.