p^2+5pq+4q^2 factors into (p+q)*(p+4q). Any factor in common with p+q and p+4q must also be a factor of their difference 3q.
p and q are coprime then q is coprime to p+q. For p+q to have a common factor with 3q then p+q must have a common factor with 3.
Then the greatest common factor of p+q and p+4q is either 1 or 3.
(p+q)*(p+4q) is given to be a perfect square. Since the common factor is either 1 or 3 then p+q and p+4q are either both perfect squares or both 3 times a perfect square.
In the case the common factor is 1: let p+q=v^2 and p+4q=w^2. Then 3q=v^2-w^2=(v-w)*(v+w) and 3p=4w^2-v^2=(2w-v)*(2w+v)
Each of v-w and 2w-v is either 1 or 3, making four total cases to check.
v-w=1 and 2w-v=1 yields q=5/3 and p=7/3; not integers.
v-w=1 and 2w-v=3 yields q=3 and p=13; solution.
v-w=3 and 2w-v=1 yields q=11 and p=5; solution.
v-w=3 and 2w-v=3 yields q=15 and p=21; not primes.
In the case the common factor is 3: let p+q=3x^2 and p+4q=3y^2. Then q=y^2-x^2=(y-x)*(y+x) and p=4x^2-y^2=(2x-y)*(2x+y).
Each of y-x and 2x+y must equal 1. This yields q=5 and p=7, another solution.
Solution
