I probably missed something that would have made a geometric solution easier. That's usually why I end up doing some crazy algebra to solve these. But, when everything comes together the solution is very satisfying.
My first reasoning was the triangle can be placed with its incenter the unit circle, then two sides of the triangle can be placed on the line x=1, thus I=(0,0), A=(a,0), B=(b,0) with b<0<a
For part 1) I will prove C, I and C' are colinear.
Side AC touches the incircle at ((1-a^2)/(1+a^2) , 2a/(1+a^2))
Likewise, for BC replace a with b
The lines through these are y=(a^2-1)x/(2a)+(a^2+1)/(2a)
and replace a with b
They cross at C=((1-ab)/(1+ab) , (a+b)/(1+ab))
The circumcenter of AIB Is the intersection of the perpendicular bisectors of the sides of triangle AIB, in this case, using IA and IB:
These lines are y= -x/a + (1+a^2)/(2a) and replace a with b.
They cross at C'=((1-ab)/2 , (a+b)/2)
C, I and C' can be seen to lie on the line y=(a+b)x/(a+b)
For part 2) we need points A' and B' so we need the perpendicular bisector of IC. This line is messy:
y = (ab-1)x/(a+b) +[(a^2+1)(b^2+1)]/[(2(a+b)(1+ab)]
A' is the intersection of the pbs of IB and IC
A' = ((b^2+1)/[2(ab+1)] , a(1+b^2)/[2(ab+1)])
for B' interchange b and a:
B' = ((a^1+1)/[2(ab+1)] , b(1+a^2)/[2(ab+1)])
Since I=(0,0) the coordinates of A' can be taken as the vector IA' etc.
The sum of the vectors is the sum of the components. Setting the sum to zero yields a system that on clearing the denominators is
(1-ab)(1+ab)+(b^2+1)+(a^2+1)=0
and
(a+b)(1+ab)+a(1+b^2)+b(1+a^2)=0
The second equation becomes (a+b)(1+ab)=0
so either a=-b or b=-1/a
The second case yields no solutions as subbing into the first equation yields only (a^2+1)^2=0
The first case, though, yields (a^2-3)(a^2+1)=0
So a=sqrt(3) (or a=-sqrt(3))
b=-sqrt(3)
Plugging in we find C=(-2,0)
So A=(1,sqrt(3)), B=(1,-sqrt(3)) and C=(-2,0)
is an equlateral triangle with sides length 2sqrt(3) times the radius of the incircle.
Analytic solution
