The equations are symmetric in x and y, so I will try to use that to my advantage.
Start by squaring each side:
x^2 + 2xy + y^2 = 529
x^2 + 12*y + y^2 + 12*x + 2*sqrt[(xy)^2 + 12*y^3 + 12*x^3+ 144xy] = 1089
Rearrange and group into terms of (x^2+y^2), (x+y), and (xy):
(x^2+y^2) = 529-2xy
(x^2+y^2) + 12(x+y) + 2*sqrt[(xy)^2 + 12(x+y)(x^2+y^2-xy) + 144xy] = 1089
Now substitute (x^2+y^2) = 529-2xy and x+y=23 into the last equation to get an equation in terms of xy:
529-2xy + 12*23 + 2*sqrt[(xy)^2 + 12*23*(529-3xy) + 144*xy] = 1089
Isolate the sqrt expression and simplify:
2*sqrt[(xy)^2 + 146004 - 828xy + 144*xy] = 1089 - 529 + 2xy - 276
sqrt[(xy)^2 - 684*xy + 146004] = 142 + xy
Square each side and simplify:
(xy)^2 - 684*xy + 146004 = (xy)^2 + 284xy + 20164
968xy = 125840
xy = 130
Then the system xy=130, x+y=23 yields two solutions (x,y) = (10,13) and (13,10).