In retrospect, it looks like I misinterpreted the parameters of the problem, since my solution has one triangle vertex on one of the square's vertices and two vertices on the square's edges.

All three triangle vertices are supposed to be on edges of the square, so I solved the wrong problem.

Here is is anyway:

Some triangles can have the right angle coincident with one of the vertices of the square, with the short sides coincident with two adjacent sides of the square.  The smallest of these can be infinitely small:  (3ε, 4ε, 5ε).  The area, 6ε^2 approaches 0 in the limit.

The longest of these is (3/4 , 1 , 5/4), with an area of 3/8.  But we can make a larger 3-4-5 triangle fit.

Make one of the acute angles coincident with one of the vertices of the square.

Call the triangle ABC, with C being the right angle.

Call the square WXYZ.

Another family of triangles has A coincident with W and B on segment YZ.

The line y = tx passes from (0,0) to (1,t).  Let that segment be of length 4k.

The line perpendicular to the first line, through (1,t) is:

(y-t)/(x-1) = -1/t

(y-t) = -(1/t)(x-1) = (1-x)/t

y = (1-x)/t + t

And if the y value = 1, the x value is:

1 = (1-x)/t + t

(1-x)/t = 1-t

(1-x) = t-t^2

(x-1) = t^2-t 

x = t^2 - t + 1 

The length of the first line segment is:  √(t^2 + 1)

The length of the second line segment is: √((1-t)^2 + (t^2-t)^2 )

√((1-t)^2 + (t^2-t)^2 ) = √(t^2 - 2t + 1 + t^4 - 2t^3 + t^2 )

   = √(t^4 - 2t^3 + 2t^2 - 2t + 1)

These two lengths are in a 3:4 ratio

3√(t^2 + 1) = 4√(t^4 - 2t^3 + 2t^2 - 2t + 1)

9(t^2 + 1) = 16(t^4 - 2t^3 + 2t^2 - 2t + 1)

16t^4 - 32t^3 + 23t^2 - 32t + 7 = 0

   which has zeros at {1/4 , 7/4}

For our problem, 0<t<1, so t=1/4 is a solution.

So the slope of the second line is -4, running from (1, 1/4), this line contacts y=1 at 1 - (1/4)(3/4) = 13/16

This triangle has vertices at (0,0), (1, 1/4), (13/16, 1)

The side lengths are (3√17)/16, (4√17)/16, (5√17)/16

The area is 6*17/256 = 51/128

I believe this is the largest.