Take a polygon with area S1 and pick a number r in [0,1/2]. Take vertex A that connects sides AB and AC and add points M and N on these sides so that AM/AB=AN/AC=r. Cut corner A along MN. Cut all other corners the same way.
After repeating these steps infinite times we will get a figure with an area S2. Let's F(r)=S2/S1. It's clear that F(0)=1 and F(½)=0.
(a) What is this function for square?
(b) What is this function for equilateral triangle?
(c) Is it possible to get a circle from a square or from an equilateral triangle this way?
(d) Is it possible that this function is universal for all triangles, or for all rectangles, or for all polygons?
Rather than try to work on an entire polygon, I a going to look at the effects of the cutting at just one corner. The main goal is to find a function (of r) which calculates the fraction of a corner that eventually gets cut.
To start, take any corner BAC and apply a set of transforms which place A at the origin, B at (0,1) and C at (1,0). This can be done using transforms which preserve ratios of areas (translate, rotate, stretct/compress, shear). Let M be the midpoint of BA and N be the midpoint of AC.
Then apply the cutting so that R is on AM with AR/BA = r and also that T is on AN with AT/AC = r.
Take M' as the midpoint of MR and L as the midpoint of RT. Next apply the cutting to corner TRM: Take R' on RM' so that R'R/RM = r and take P on RL so that RP/RT = r.
Now consider triangles ART, AMN, RR'P, and RM'L. Since the same cutting produced ART from AMN and RR'P from RM'L, then the ratio follows: area(ART)/area(AMN) = area(RR'P)/area(RM'L).
The area of AMN = 1/8 and the area of ART = (r^2)/2. To find the area of RM'L, first construct point L' on AR so that L'L is perpendictular to AR. The coordinates of L are (r/2,r/2) so then the coordinates of L' are (0,r/2). That makes L'R = r/2 and LL' = r/2
The coordinates of M are (0,1/2) and the coordinates of R are (0,r) which makes the coordinates of M' (0,1/4 + r/2), which makes RM' = 1/4 - r/2 and L'R' = 1/4. The area of RM'L can be calculated as area(LL'M') - area(LL'R). The area of LL'M' = r/16 and the area of LL'R is r^2/8 which makes the area of RM'L = r/16 - r^2-8.
Using the area ratio above, the area of RR'P then can be calculated as ((r^2)/2)*(r/2 - r^2). The area of the analogous triangle created at corner T is the same.
The areas of the four triangles created by the third generation of cuts (at corners R', P, etc) vs the area of the two in the second generation of cuts (corners R, T) share the same ratio as the second and first generations of cuts, which means the area of a single cut decreases geometrically with the ratio (r/2 - r^2). In the nth generation, there are 2^(n-1) triangles, each with area ((r^2)/2) * (r/2 - r^2)^(n-1).
The sum of the areas is a geometric series with initial term ((r^2)/2) and ratio 2*(r/2 - r^2). The sum of that geometric series is ((r^2)/2)/(1-2*(r/2 - r^2)) = r^2/(2-2r-4r^2). That area represents the amount of the corner AMN which is removed, so a general function can be obtained by simply taking (r^2/(2-2r-4r^2))/(1/8) = (4r^2)/1-r-2r^2) since ratio of area preserving transformes were used to create AMN.
The function which calcultes the fraction of a corner's removal is f(r) = (4r^2)/(1-r-2r^2). So to calculate the total removal, take the sum area of all the triangles created by a vertex and the two midpoints of the sides that form it and multiply that area by f(r).
Notice that while f(0) = 0 as expected, f(1/2) = 1, which is equivalent to simply clipping all the corners once. This illustrates a change in behavoir of r<1/2 and r=1/2. Even when r=.499999999, the midpoints of the sides stay in the final figure, but when r=1/2, those midpoints are removed after the second generation of cuts.