If OO' is small enough then the squares' overlap will be a hexagon. Let E be the intersection of CD and C'D'. In the case that triangle OEO' is equilateral, find the area of the overlap.
If OO' is large enough then the square's overlap will be a quadrilateral. Let F be the intersection of AD and A'D'. In the case that triangle OFO' is equilateral, find the area of the overlap.