This seemed too complicated to analyze, so I did the simulation over ten million trials (after debugging using 10,000 or a million trials each time through).

After randomly selecting five points in a unit square, the program tests each of those points for being a vertex of the outlining figure. It does this by taking the direction to each of the other points. It represents each direction both as it's originally calculated and also 360° higher so that the differences can be fully calculated all the way around. It places them in increasing order so that it can test if there's a gap of at least 180° between one pair of successive directions, which would indicate this is in fact a vertex as all the directions would lie within the other 180° or less.

hits=0; clc

tic

cts=[0 0 0 0 0];

for trial=1:10000000

  for i=1:5

    x(i)=rand; y(i)=rand;

  end

  ctV=0;  % count vertices

  

  for p0=1:5

    wh=0; theta=[];

    for p1=1:5      

      if p1~=p0

        dx=x(p1)-x(p0);

        dy=y(p1)-y(p0);

        wh=wh+1;

        theta(wh)=atan2d(dy,dx);

      end      

    end

    theta=sort([theta theta+360]);

    theta=theta-theta(1);

    f180p=false;

    for i=1:length(theta)-1

      if abs(theta(i+1)-theta(i))>180

        f180p=true;

        break

      end

    end

    if f180p 

      ctV=ctV+1;

    end

  end

  if ctV==4

    hits=hits+1;

  end

  cts(ctV)=cts(ctV)+1;

end

toc

cts

trial

cts/trial

resulting in

Elapsed time is 279.890894 seconds.

cts =

           0           0     1041764     5556342     3401894

trial =

    10000000

ans =

  0         0          0.1041764        0.5556342          0.3401894

  

where cts shows the number of trials giving 1, 2, 3, 4 and 5 vertices respectively; trial shows the number of trials done; ans shows each of the counts divided by the number of trials.  During debugging, when figures showed up with only two vertices, I could tell there was still a bug, as that has a probability of zero, all the five points lying on a straight line.

 

We expect half the shown digits of each statistic to be significant so the approximate probabilities are:

triangle      10.4 %
quadrilateral 55.56 %
pentagon      34.0 %

Initially I thought there was the possibility that the probability of a quadrilateral was 5/9, but results consistently showed a value slightly higher than 55.56%, rather than merely rounding to that from below.

A later simulation shows for 100 million trials:

cts =
           0           0    10414959    55558557    34026484
trial =
   100000000
ans =
     0       0      0.10414959    0.55558557   0.34026484
     
     
triangle      10.41 %     
quadrilateral 55.56 %
pentagon      34.03 %

It's on the borderline of significance for the possibility of the answer for quadrilaterals being 5/9.