(From the MSU Problem Corner: Read the problem statement and the solution.) The answers:
Explanation: The first two parts follow as special cases (n=3 and n=4) of the last part.
The optimal circle must have equal arc length inside and outside the n-gon. To see why, take a circle with radius r, and define f(r) to be the total arc length of the circle inside the n-gon, so that 2πr - f(r) is the arc length outside the circle. Now increase the radius by an infinitesimal amount (from r to r+dr). The figure on the right illustrates (for a triangle) how the "inside-outside" area changes: The region shaded red is added, while the green-shaded region is subtracted.
The areas of these red- and green-shaded regions are approximated by their width (dr) times their (arc) lengths, so the resulting change in the area will be
(It is easily confirmed that if dr is negative, so that we are actually decreasing the radius of the circle, the expression given for dA is still correct.) Therefore, dA/dr = 0 when f(r) = πr — that is, when half of the circle's circumference is inside, and half outside.
It remains to confirm the given expression for r, which achieves the desired property f(r)=πr. This is easiest to see with the figure on the right.
In the figure, θ = 2π/n, since it is the central angle of a regular n-gon. Therefore the center-to-side distance is
x = (1/2) cot (θ/2) = (1/2) cot (π/n).
Since the arc length for a circular arc is determined by the central angle of the arc (and the circle's radius), it is clear that in order for the "inside" arc length to be the same as the "outside" arc length, the circle must intersect the n-gon so that the smaller angle shown is θ/4 = π/(2n). This makes the radius
r = x / cos (θ/4) = (1/2) cot (π/n) sec (π/(2n)).
Generalizations: If we fix the radius of the circle (at, say, 1) and vary the size of the triangle, the minimum area is achieved when the total "outside" side length of the triangle equals the total "inside" side length. This happens when the triangle's side length is 4*√(3/7) ≈2.619.
If we have a sphere and a regular tetrahedron, we can draw similar conclusions (but with more complicated computations!) by equating the inside and outside surface areas of the sphere (with the tetrahedron's size held constant) — or the tetrahedron (with the sphere's radius held constant). Similarly, we can work with a cube and a sphere, etc.