Fold corner A down to lie somewhere along side CD. Call x the portion of the way from C to D.

Corner B is outside of the original square and is now one side of a small triangle whose opposite side is along the remainder of side BC.

(A) Where would A be folded to make this small triangle isosceles?

(B) Find a formula in terms of x for the perimeter of this small triangle.

(C) Where would A be folded to maximize the area of this small triangle?

Let E and H be the points that corners
A and B are folded into respectivly.
Let F and G be the intersections of the
crease with sides AD and BC respectively.
Let I be the intersection of line EH 
with side BC. Let s be the side length
of the square and x = |EC|.

Let beta = angle DAE. Then

                |DE|     |DC|-|EC|
   tan(beta) = ------ = -----------
                |AD|        |AD|

             = (s-x)/s.

Since F lies on FG (the perpendicular
bisector of AE, |AF| = |FE|.

Let alpha = angle DFE. Looking at
triangles DFE and AFE we see that

   alpha = 2*beta.

Therefore,

   cos(alpha) = cos(2*beta)

                 1 - tan(beta)^2
              = -----------------
                 1 + tan(beta)^2

                    x*(2s - x)
              = ------------------      (1)
                 2s^2 - 2sx + x^2

   sin(alpha) = sin(2*beta)

                   2*tan(beta)
              = -----------------
                 1 + tan(beta)^2

                    2s*(s - x)
              = ------------------      (2)
                 2s^2 - 2sx + x^2

Note that right triangles FDE, ECI, and
GHI are similar. Thus,

   |EC| = x and |EI| = x/cos(alpha),

   |GH| = |GI|cos(alpha), and

   |HI| = |GI|sin(alpha).

To tie FDE and ECI together,

   |EI| + |IH| = |EH|

        or

   x/cos(alpha) + |GI|sin(alpha) = s

        or

             s*cos(alpha) - x
   |GI| = -----------------------       (3) 
           cos(alpha)*sin(alpha)

-------------------------------------------

(A) If triangle GHI is isosceles, then

       cos(alpha) = sin(alpha)

                  or

       x*(2s - x) = 2s*(s - x)

                  or

       x^2 - 4sx + 2s^2 = 0

                  or

       x = s*[2 - sqrt(2)].

-------------------------------------------

(B) Perimeter(GHI) = |GH| + |HI| + |IG|

    Combining this with (1)-(3) gives

    Perimeter(GHI) = x.

-------------------------------------------

(C) Maximize Area(GHI):

    Area(GHI) = |GH|*|HI|/2

                 |GI|^2*cos(alpha)*sin(alpha)
              = ------------------------------
                               2

    Combining this with (1)-(3) gives

                 sx^3 - x^4
    Area(GHI) = ------------            (4)
                 8s^2 - 4sx

    Maximum area when

      (8s^2 - 4sx)*d/dx(sx^3 - x^4) -

      (sx^3 - x^4)*d/dx(8s^2 - 4sx) = 0

             or

      (8s^2 - 4sx)*(3sx^2 - 4x^3) -

      (sx^3 - x^4)*(-4s) = 0

             or

      4sx^2*(3x^2 - 10sx +6s^2) = 0 

             or

      x = s*[5 - sqrt(7)]/3             (5)
      
    Combining (4) and (5) gives

                      s^2*[316 - 119*sqrt(7)]
    Max. Area(GHI) = -------------------------
                               54

QED
-------------------------------------------

Note the above results checked with
Geometer's Sketchpad.