If the exponents of the prime factorization of an integer N are a,b,c,..., the number of factors of N is the product (a+1)(b+1)(c+1)...  If N has exactly 6 factors then N will be the fifth power of a prime or it will be the product of a prime and the square of a prime.

By inspection p=2 is not a solution so p is odd.  Setting p=2q+1 it's clear LHS will be divisible by 4 and not 8.  So N won't be a fifth power, making N = 4r, r=prime.

Odd primes > 3 are = 1 or -1 mod 6.  Then squaring and adding 11 gives a sum divisible by 3, so r is divisible by 3.  Thus r=3.  But p^2 + 11 = 4*3 makes p=1, a non-prime.

We've excluded p=2 and p>3.  Trying p=3 gives the solution 20 which is (2^2)*5 and has the six factors 1,2,4,5,10,20.  

p=3 is the unique solution.