Observations:

1) the two hemispheres will always have a contact point that is a tangent to each of their curved surfaces.

2) Normals to theses contact points therefore  will be a radius of each hemisphere by definition.

3) These two radii will also be co-linear for all times and displacements given the problem description.

4) The above all results from the problem description and "no slipping" comment - which - actually then implies rolling contact between A & B

5) The term "nudge" implies, to me anyway", small angles and displacements, so that approximations such as sin(theta)=theta, can be used, albeit carefully.  Once can envision that this oscillation is not linear with respect to larger displacements, just like a classic pendulum is actually non-linear w.r.t. non-small displacements.

6) The only forces acting on A are: Gravity (weight), normal force through the contact point, and friction force at the contact point.  The friction force is perpendicular to the normal force.

7) The c.g. of a solid hemisphere radius r, is along the line of radial symmetry, 4*r/3/pi, or about 4/9 of the way up the "center" radius from the flat side.

8) I believe this geometry can be reduced to the 2D equivalent of 2 semi circles and get the same solution, being careful to use the c.g. location for the solid hemisphere, and not incorrectly use properties of a semi-circle.

Methodology

The easiest (least complex?) way I can envision to deduce a set of equation(s) to work with would be to sum moments on A about the "nudged" contact point. This would eliminate all moments on A except that due to the weight force.  A is displaced a small angle from the static equilibrium arrangement. The point where A & B remain in contact migrates away from the equilibrium point.  Depending on the sizes of a & b, the c.g. of A migrates differently, the amount of which is dependent on some f(a,b).  To derive this function, use the above principles and geometry from the problem.  To find the solution, the constraint would be that the relationship of the two hemisphere radii, a & b, must be such that the line of action of the force of gravity acting on A, through it's center of gravity, must be oriented such that the moment of this force about the mutual contact point on A & B has direction to try to rotate A back towards the equilibrium position.  Only if this is true can there be oscillation.