On configuration spaces of plane polygons, sub-Riemannian geometry and periodic orbits of outer billiards

Following a recent paper by Baryshnikov and Zharnitskii, we consider outer billiards in the plane possessing invariant curves consisting of periodic orbits. We prove the existence and abundance of such tables using tools from sub-Riemannian geometry. We also prove that the set of 3-periodic outer billiard orbits has zero measure.


Introduction
The classical billiard system describes the motion of a point in a plane domain subject to the elastic reflection off the boundary, described by the familiar law of geometrical optics: the angle of incidence equals the angle of reflection; see, e.g., [13,14] for surveys of mathematical billiards.
For every n ≥ 2, the billiard system inside a circle has a very special property: every point of the circle is the starting point of an n-periodic billiard orbit; this orbit is an inscribed regular n-gon. 1 Likewise, an ellipse enjoys the same property for every n ≥ 3; the orbits are inscribed n-gons of extremal perimeter length. 2 A billiard table of constant width also has the property that every point on its boundary belongs to a 2-periodic, back-andforth, billiard trajectory, and this is a dynamic characterization of the curves of constant width. The phase space of the billiard ball map is a cylinder, and the family of n-periodic orbits forms an invariant circle of the billiard ball map consisting of n-periodic points.
How exceptional is this property? More specifically, given n ≥ 3, one wants to describe plane billiards such that the billiard ball map has an invariant curve consisting of n-periodic points. The first result in this direc-tion was obtained by Innami [8]: he constructed a billiard curve, other than a circle or an ellipse, whose every point is a vertex of a 3-periodic billiard trajectory (these billiard triangles are isosceles in his example). 3 In a recent paper [1], Baryshnikov and Zharnitskii developed a new elegant approach to this problem. In a nutshell, their innovation was in switching focus from billiard curves to the space of plane n-gons. If such an n-gon, z 1 , . . . , z n , is a periodic billiard trajectory then, due to the law "the angle of incidence equals the angle of reflection", we know the directions of the billiard curve at each vertex z i . This determines an n-dimensional distribution D in the space of n-gons, called the Birkhoff distribution. One can further restrict attention to the hypersurface of n-gons with a fixed perimeter length: the Birkhoff distribution is everywhere tangent to these hypersurfaces.
It turns out that the Birkhoff distribution is non-integrable. A 1-parameter family of billiard n-gons z 1 (t), . . . , z n (t), 0 ≤ t ≤ 1, is interpreted as a curve in the space of n-gons, tangent to D and satisfying the "monodromy" condition z i (1) = z i+1 (0), i = 1, . . . , n. The problem of finding such horizontal curves belongs to sub-Riemannian geometry and non-holonomic dynamics.
The main result of [1] is that, for every n ≥ 2, the space of billiard tables, for which the billiard ball map possesses an invariant curve consisting of n-periodic points, is infinite-dimensional (this space also has an infinite codimension in the space of all billiard tables). Example 1.1 Let us illustrate this approach in the simplest case n = 2, that is, curves of constant width (for information on billiards of constant width, see [9]).
Consider the space of oriented segments, say, of length 2. A segment z 1 z 2 is "allowed" to move in such a way that the velocities of both end-points are orthogonal to the segment. Let O = (x, y) be the mid-point of the segment and α its direction. Then the space of segments has coordinates (x, y, α) and We conclude that the velocity of point O must be orthogonal to the vector (cos α, sin α), and this is the definition of the Birkhoff distribution in this case. Thus D is the kernel of the 1-form λ = cos α dx + sin α dy. This 1-form is contact: λ ∧ dλ = 0. We have identified our space of segments with the space of cooriented contact elements in the plane, a fundamental example of a contact 3-dimensional manifold (see, e.g., [4]).
The projection of this Legendrian curve on the (x, y)-plane, that is, the trajectory of the mid-point O, is a closed smooth curve ∆ with singularities (generically, semi-cubical cusps); the total winding of this curve is π. Every such curve ∆ uniquely lifts to a Legendrian curve in the space of contact elements: the missing α coordinate is recovered as the slope: cot α = −dy/dx.
In other words, we use ∆ as a guide: place a segment of length 2 so that its mid-point O is on the curve ∆ and the segment is orthogonal to ∆, and then slide the segment around ∆. If certain convexity conditions hold (for example, ∆ should not have inflection points) then the end-points z 1 and z 2 will describe a curve of constant width.
For example, if the end-points z 1 and z 2 move along a circle then ∆ degenerates to its center, and if z 1 and z 2 describe the Reuleaux triangle then ∆ is made of three 60 • arcs of a circle, see figure 1. In this paper we extend the results from [1] to outer (or dual) billiards, a dynamical system that is a close relative of the conventional, inner, billiards; see [3,13,14] for surveys and [5] for a recent interesting result.
An outer billiard table is a strictly convex compact plane domain D. Pick a point x outside D. There are two support lines from x to D; choose one of them, say, the right one from x's view-point, and reflect x in the support point. One obtains a new point, y, and the transformation T : x → y is the outer billiard map, see figure 2. If the boundary of the outer billiard table contains straight segments (for example, is a convex polygon), the map T or its inverse is not defined on the lines containing these segments; this still leaves a set of full measure on which all iterations of T are defined. We will consider only outer billiard tables with strictly convex smooth boundaries.
x T(x) Let us mention four fundamental properties of outer billiards, important for us. First, the outer billiard map commutes with affine transformations of the plane. Secondly, the outer billiard map preserves the standard area form in the plane (similarly to the billiard ball map which is an area preserving map of a cylinder). Thirdly, the outer billiard map is a twist map of the exterior of the outer billiard table D, a semi-infinite cylinder, with respect to the vertical foliation consisting of tangent half-lines to the boundary of D. And finally, an n-periodic outer billiard orbit is an n-gon (possibly, starshaped), circumscribed about the table D and having an extremal area. This is an outer analog of a variational property of periodic billiard trajectories: they are inscribed polygons of extremal perimeter length. As a consequence, if an outer billiard has a 1-parameter family of n-periodic orbits then the respective circumscribed n-gons have the same areas.
Let G n be the cyclic configuration space of the plane, that is, the space of polygons z 1 , . . . , z n such that z i = z i+1 for i = 1, . . . , n (where the index i is understood cyclically). If z 1 (t), . . . , z n (t) is a 1-parameter family of n-periodic outer billiard orbits with T (z i (t)) = z i+1 (t) then each segment z i (t)z i+1 (t) envelopes the boundary of the outer billiard table and is bisected by the tangency point. This prompts the following definition.
Denote by F the n-dimensional distribution in G n determined by the following condition: the velocities of points z 1 , . . . , z n lie in F if, for each i = 1, . . . , n, the motion of the line z i z i+1 is an infinitesimal rotation about the mid-point of the segment z i z i+1 . We call F the dual Birkhoff distribution.
The dual Birkhoff distribution is tangent to the level hypersurfaces of the area function, and we can restrict attention to non-degenerate polygons of a fixed non-zero area. On this hypersurface, F is non-integrable: the vector fields, tangent to F , and their first commutators generate the tangent space at every point (see Theorem 1).
We are interested in the outer billiard tables possessing invariant curves consisting of n-periodic orbits. Such an invariant curve gives rise to a horizontal curve z 1 (t), . . . , z n (t), 0 ≤ t ≤ 1, in G n satisfying the monodromy condition z i (1) = z i+1 (0), i = 1, . . . , n. If the outer billiard table is a circle (or an ellipse) then, for every pair (n, k) where 3 ≤ n, 1 ≤ k ≤ n/2, and k coprime with n, there is an invariant curve consisting of n-periodic orbits with rotation number k (that is, making k turns around D). We show that, for every such pair (n, k), one can perturb a circle so that the resulting outer billiard still has an invariant curve consisting of n-periodic orbits with rotation number k; the space of such deformations is infinite-dimensional and depends on functional parameters (see Theorem 2).
In Section 4, we study the simplest case of n = 3 in detail; this is the closest outer billiard analog of curves of constant width. This case illustrates the general approach by explicit computations. We also discuss a discretization of the problem in which a smooth outer billiard curve is replaced by a polygon.
We also prove that, for every outer billiard table, the set of 3-periodic orbits has zero measure (see Theorem 4). For inner billiards, this is a wellknown result having a number of different proofs [11,12,15,16]. We also give different proofs, one of them follows ideas of [16] and another those of [1]. Note that there exist area preserving twist maps of a cylinder whose set of periodic points has a non-empty interior.
Let us finish this introduction with a problem: can invariant billiard curves (inner or outer) consisting of periodic points with different periods, or the same period but different rotation numbers, coexist, unless the table is an ellipse? In particular, is it true that the only billiard of constant width, having an invariant curve consisting of 3-periodic points, is a circle?

Configuration space of plane polygons and the dual Birkhoff distribution
Let us fix some notations. The determinant of two vectors, that is, the crossproduct, is denoted by [ , ]. A generic point in the plane is denoted by z = (x, y). If z 1 = (x 1 , y 1 ) and z 2 = (x 2 , y 2 ) then z 1 dz 2 denotes the 1-form is a vector in R 2 and z = (x, y) is a point then by w∂z we mean the tangent vector u∂x + v∂y. When working with n-gons, we always understand the indices cyclically, so that i = i + n. For a polygon z 1 , . . . , z n , set this is twice the area of the polygon (counted with appropriate multiplicities and signs). Let F be the dual Birkhoff distribution on the space of n-gons defined in Section 1. Let Z = (z 1 , . . . , z n ) ∈ G n be an n-gon and W = (w 1 , . . . , w n ) a tangent vector to G n at Z; the vector w i is the velocity of the vertex z i . Lemma 2.1 F is an n-dimensional distribution. A vector W lies in F if and only if, for all i = 1, . . . , n, one has: Proof. To see that F is n-dimensional, view a polygon Z = (z 1 , . . . , z n ) ∈ G n as an ordered collection of n lines L = (z 1 z 2 , z 2 z 3 , . . . , z n z 1 ). From this dual viewpoint, G n becomes an open subset in the space L n of n-tuples of oriented lines in the plane. An infinitesimal rotation of the i-th line z i z i+1 about the mid-point of the segment z i z i+1 determines a tangent vector to L n at point L, and these vectors are clearly independent for i = 1, . . . , n. It follows that dim F = n.
For W to belong to F , the line through points z i + εw i and z i+1 + εw i+1 must pass through the mid-point (z i + z i+1 )/2; here ε is infinitesimal. Thus the linear term in ε of the determinant An n-gon is called non-degenerate if no three consecutive vertices lie on one line. The set of non-degenerate n-gons is denoted by U n . Clearly, U n is an open subset of G n .
Let us introduce the 1-forms Lemma 2.2 Let Z ∈ U n be a non-degenerate polygon. Then the 1-forms α i are linearly independent at Z and the intersection of their kernels is the fiber Proof. First, we claim that each form α i vanishes on every tangent vector , and this is zero, according to Lemma 2.1.
Suppose that a non-trivial linear combination t i α i vanishes at point Z. Then, for every test vector W = (w 1 , . . . , w n ), one has: Therefore Since the polygon Z is non-degenerate, the vectors z i+1 − z i and z i − z i−1 are linearly independent, and (1) implies that t i = 0 for all i.
Since F is n-dimensional and the 1-forms α i are linearly independent, one has: The next lemma shows that one can restrict attention to the level hypersurfaces of the area function. Lemma 2.3 Every non-zero level hypersurface of the function A is smooth, and the distribution F is tangent to these hypersurfaces.
Proof. Let Q : V → V * be a self-adjoint linear operator in a vector space V . Consider the quadratic function F (x) = Q(x) · x on V . Then every non-zero level hypersurface of F is smooth. Indeed, by Euler's formula, ∇F (x) · x = 2F (x), hence dF does not vanish on a non-zero level hypersurface. The function A is a quadratic form on the space of n-gons, and the first claim follows.
One has the equality which implies the second claim. ✷ Let us construct a system of n linearly independent vector fields tangent to F . Let Z = (z 1 , . . . , z n ) be a non-degenerate polygon. Set For these values of i, one has: Proof. To show that W k lies in F , it suffices to check that the vector which follows from the general identity, valid for any three vectors: Under identification of the space of n-gons with the space of n-tuples of lines L n , the tangent vector W k corresponds to a nontrivial infinitesimal rotation of the line z k z k+1 about the mid-point of the segment z k z k+1 . Therefore these vectors are linearly independent and hence span the dual Birkhoff distribution. ✷ Now we prove that the the dual Birkhoff distribution is completely nonintegrable.
Theorem 1 Let M 2n−1 be the hypersurface in U n consisting of non-degenerate n-gons with a fixed non-zero area. Then the tangent space of M at every point is generated by the vectors fields tangent to F and their first commutators. Thus the bracket growth type of F is (n, 2n − 1) (just like that of the Birkhoff distribution, see [1]).
Proof. Consider the vector fields ξ k = [W k−1 , W k ] (the fields W i and W j clearly commute if |i − j| ≥ 2). We want to show that these fields, along with F , span the tangent space to M at every point.
Since the fields W k belong to the kernel of each form α i , the Cartan formula implies: Recall that Z = (z 1 , . . . , z n ) with z i = (x i , y i ). Let ω i = dx i ∧ dy i , i = 1, . . . , n. We claim that dα i = 2(ω i+1 − ω i ). Indeed, Thus we want to show that the vectors span R n−1 for k = 1, . . . , n. This will follow if we show that the vectors is a non-degenerate diagonal matrix, and we are done. ✷ Remark 2.5 The dual Birkhoff distribution F on the cyclic configuration space G n is Lagrangian with respect to the following symplectic structure. The space of oriented lines in the plane, topologically, a cylinder, has a canonical area form (see, e.g., [13,14]), and the direct product of these forms is a symplectic structure in the space of n-tuples of oriented lines L n . In the proof of Lemma 2.1, we realized G n as an open subset in L n , and this provides a symplectic structure on G n . Infinitesimal rotations of distinct lines determine symplectically orthogonal tangent vectors to L n , hence F is Lagrangian. The Lagrangian property of the dual Birkhoff distribution echoes that of the Birkhoff distribution [1].

Outer billiards with invariant curves consisting of periodic points
In this section we show that, for every pair (n, k) with 3 ≤ n, 1 ≤ k ≤ n/2 and k coprime with n, there is an abundance of outer billiard tables, possessing invariant curves consisting of n-periodic orbits with rotation number k; such tables are constructed as perturbations of circles. Denote by H 2 F the space of parameterized horizontal paths [0, 1] → M (paths tangent to the dual Birkhoff distribution) whose first 2 derivatives are square integrable (one can work with a higher number of derivatives as well). For a polygon Z ∈ M, denote by H 2 F (Z) the space of paths starting at Z. Then H 2 F (Z) is a Hilbert manifold, see [10]. Let Z be a regular star-shaped n-gon with rotation number k. We have the following result.

Theorem 2
The set of (parameterized) outer billiard curves, sufficiently close to a circular one and possessing invariant curves consisting of n-periodic orbits with rotation number k, is a smooth Hilbert submanifold of codimension Proof. To fix ideas, consider the case of simple periodic n-gons, that is, the case of k = 1. At the end of the proof, we will indicate the small adjustment to be made in the case of k > 1. Start with a circular outer billiard. One has a 1-parameter family of periodic n-gons circumscribed about the circular table (the table is scaled appropriately). Let σ : G n → G n be the cyclic permutation of the vertices of a polygon: σ(z 1 , . . . , z n ) = (z 2 , . . . , z n , z 1 ). Then Z(1) = σ(Z(0)). The family (2) determines a horizontal curve γ 0 (t) in G n whose endpoints are Z(0) and Z(1). Denote by H 2 F (Z(0), Z(1)) ⊂ H 2 F (Z(0)) the set of horizontal curves with the terminal point Z(1). We want to perturb γ 0 (t) in H 2 F (Z(0), Z(1)). If a perturbation is sufficiently small and trivial in a neighborhood of end-points then we obtain a 1-parameter family of convex n-gons whose vertices trace n arcs that smoothly match together to form an invariant curve of the outer billiard map consisting of n-periodic points; the outer billiard curve is recovered as the envelope of the segments connecting the consecutive vertices.
To prove that γ 0 is not singular we use a criterion due to Hsu [7]. Denote by F ⊥ ⊂ T * M the annihilator of the dual Birkhoff distribution and let Ω be the restriction of the canonical symplectic form on the cotangent bundle to F ⊥ . Let π : T * M → M be the projection. A characteristic in F ⊥ is a curve that does not intersect the zero section and whose direction at every point lies in the kernel of Ω. By Hsu's criterion, a horizontal curve is singular if and only it is the projection under π of a characteristic curve in F ⊥ .
It is convenient for computations to use the following form of this criterion. The 1-forms α i , introduced in Section 2, generate F ⊥ : every covector in F ⊥ can be written as λ i α i where the coefficients λ i are "Lagrange multipliers". There is exactly one relation: α i = 0 (see Lemma 2.3), and to factor this relation out, we assume that λ i = 0. Consider the 2-form ω(λ) = λ i dα i on M. Then one has the following result, see [10]: the kernel of Ω at a point (x, λ i α i ) ∈ F ⊥ is isomorphically projected by the differential dπ onto the kernel of the form ω(λ) restricted to the fiber F (x) at point x ∈ M.
Finally, if the rotation number k of a periodic trajectory is greater than 1, one changes the curve (2) to Z(t) = (z 1 (t), . . . , z n (t)) with z j (t) = e 2πi(jk+t)/n , t ∈ [0, 1], and the proof proceeds along the same lines. ✷ Remark 3.1 To illustrate the importance of checking that the curve γ 0 is not singular, let us give a simple example of a rigid horizontal curve, that is, a curve that does not admit smooth perturbations in the class of horizontal curves with fixed end-points, see [10]. Consider the curve γ(t) = (t, 0, 0), t ∈ [0, 1] tangent to the distribution dz = y 2 dx in 3-space. If (x(t), y(t), z(t)) is its perturbation as a horizontal curve with fixed end-points then Since x ′ (t) > 0, we have y(t) = 0, and hence z(t) = 0, for all t; thus the perturbed curve is a reparameterization of γ.

Case study: n = 3 and discretization
Three is the smallest possible period of the outer billiard map. In this section we illustrate the general constructions of Sections 2 and 3 in the case n = 3.
Let γ be a smooth strictly convex outer billiard curve and z 1 z 2 z 3 a 3periodic orbit. Let w 1 , w 2 , w 3 be the mid-points of the sides of the triangle z 1 z 2 z 3 .

Lemma 4.1
The inscribed triangle w 1 w 2 w 3 has the property that, for every i = 1, 2, 3, the tangent line to γ at vertex w i is parallel to the side w i+1 w i+2 . Conversely, if an inscribed triangle has this property then the tangent lines to γ at its vertices form a circumscribed triangle whose sides are bisected by the tangency points. Furthermore, this property of an inscribed triangle is equivalent to having extremal area.
Proof. The first claim holds by definition of the outer billiard map and the second follows from elementary geometry. To prove the last statement, fix points w 2 and w 3 and vary w 1 . The area is extremal when the distance from w 1 to the line w 2 w 3 has an extremum, and this happens if and only if the tangent line to γ at w 1 is parallel to w 2 w 3 . ✷ It follows that we can replace circumscribed triangles of extremal areas by inscribed ones. We want to construct smooth strictly convex closed curves such that an inscribed triangle of extremal area can be turned around inside the curve.
Following the approach of Section 2, let M be the 5-dimensional manifold of triangles of area 3/2. Let w 1 w 2 w 3 be a triangle from M; denote by c its center of mass. Let u = w 1 − c, v = w 2 − c; then [u, v] = 1 and the frame (u, v) belongs to SL(2, R). The cyclic group Z 3 acts on M by the cyclic permutations of the vertices of a triangle and on SL(2, R) by the transformation T (u, v) = (v, −u − v). Extend the action of Z 3 to SL(2, R) × R 2 taking product with the trivial action on the second factor. On M, we have a 3-dimensional distribution F determined by the condition that the velocity of w i is parallel to the line w i+1 w i+2 , i = 1, 2, 3.
The fibers of the projection SL(2, R)× R 2 → SL(2, R) are transverse to the distribution F .
Proof. The first claim is obvious. To prove the second, note that a vector, tangent to the fibers of the projection SL(2, R) × R 2 → SL(2, R), is induced by a parallel translation of the triangle: all three vertices move with the same velocity. If this vector lies in F then it must be parallel to all three sides of the triangle, which is impossible. ✷ We want to construct a closed horizontal curveγ(t), t ∈ [0, 2π] in M satisfying the monodromy conditioñ where σ is the cyclic permutation of the vertices of a triangle. By Lemma 4.2, this curve projects to a smooth curve γ(t) in SL(2, R) satisfying γ(t+2π/3) = T (γ(t)). Conversely, such a curve γ(t) ⊂ SL(2, R) lifts uniquely to a horizontal curveγ(t) ⊂ M, provided a lift of the initial point is chosen, but the monodromy condition (5) may fail. But once (5) is satisfied -and assuming convexity -we obtain a desired plane curve possessing a 1-parameter family of inscribed triangles of extremal areas. Let us work out explicit formulas. Denote by γ(t) = (u(t), v(t)), t ∈ [0, 2π] a closed curve in SL(2, R) such that u(t + 2π/3) = v(t) and v(t + 2π/3) = −u(t) − v(t). Let c(t) be the curve of the center of mass of the respective triangle, that is, the projection to R 2 of a lifted horizontal curvẽ γ(t). Set:

Proposition 4.3
The velocity of the center of mass is given by the formula: The monodromy condition (5) holds if and only if or, equivalently, Proof. One has: The curve is horizontal if and only if or, in view of (9), (there are only two equations because the third is their consequence). It follows that and since [u(t), v(t)] = 1, this implies (6). Abusing notation, let T be the shift t → t + 2π/3 of the argument of the functions u(t), v(t), p(t), q(t), r(t); this defines an action of Z 3 on the space generated by these functions. One has: The monodromy condition (5) is satisfied if and only if c(t) is 2π/3-periodic. Applying (10) to (6), it follows that c ′ (t) is 2π/3-periodic, and hence c(t) is 2π/3-periodic if and only if c(t) is 2π-periodic, that is, if 2π 0 c ′ (t)dt = 0. In view of (6), this is equation (7). Finally, for every function f (t), one has Applying this to (10) yields four identities for integrals of vector-functions pu, pv, qu, qv, ru, rv: These identities imply that (7) and (8) are equivalent. ✷ To summarize, one constructs an outer billiard table having a 1-parameter family of 3-periodic trajectories from a closed curve in SL(2, R) satisfying (8), that is, satisfying two scalar equations.
It is interesting to consider the following discretization of the problem in which a smooth curve is replaced by a polygon. Let P be a convex polygon and ABC an inscribed triangle whose vertices are some vertices of P . One is allowed to slide one vertex of the triangle along an edge of P to the adjacent vertex of P , provided this edge is parallel to the opposite side of the triangle. Using a sequence of such moves, one wants to turn the triangle around inside P . For which polygons is this possible?
Consider an example. Let P be a regular k = 3n + 1-gon with vertices z 1 , . . . z k , and let A = z 1 , B = z n+2 , C = z 2n+2 . Then the segment z 1 z 2 is parallel to the segment BC, and one can move point A to z 2 . After that one can move point C to z 2n+3 , then point B to z n+3 , etc. -see figure 4 for k = 7.
One can deform a regular k = 3n + 1-gon preserving the desired property: the condition to satisfy is that the segments z i z i+1 and z i+n+1 z i+2n+1 be parallel for all i = 1, . . . , k. This gives k conditions, whereas a k-gon has 2k degrees of freedom, so it is not surprising that non-trivial deformations exist. We do not dwell on details here.
Let us finish this section with a problem. Suppose that a convex domain has the property that an inscribed triangle of extremal area can be turned around inside the domain. Find the lower and upper bounds for the ratio of

The set of 3-periodic orbits has empty interior
It is an old conjecture, with motivation in spectral geometry, that the set of n-periodic billiard trajectories in a billiard with a smooth boundary (not necessarily convex) has zero measure. For n = 2, this is easy to show, and for n = 3, this is a theorem of Rychlik [11] with a number of different proofs available [1,12,15,16]. For n ≥ 4, the conjecture is open. This section concerns the outer counterpart of this conjecture. Consider n outer billiard tables D 1 , . . . , D n with smooth boundaries and let T 1 , . . . , T n be the respective outer billiard maps. We are interested in the set F of fixed points of the composition T n • T n−1 • · · · • T 1 .
Conjecture 3 For every n ≥ 3, the set F has zero measure.
Remark 5.1 For strictly convex outer billiards with corners, Conjecture 3 fails. An example shown in figure 5 is a rounded square: the set of 4-periodic orbits has a non-empty interior. One can generalize: if a plane polygonal outer billiard has an n-periodic orbit then a whole neighborhood of this point consist of periodic points as well (with period 2n, if n is odd). According to a theorem of Culter [2], every polygonal outer billiard has periodic orbits, 4 thus one can use any (rounded) polygon to construct an open set of periodic orbits.
We prove the following particular case of Conjecture 3.   Theorem 4 Let D 1 , D 2 , D 3 be three smooth strictly convex outer billiard tables and T 1 , T 2 , T 3 the respective outer billiard maps. Then the set F of fixed points of the map T 3 • T 2 • T 1 has empty interior.
As in [11,12,15,16], this theorem implies that F has zero measure; we do not dwell on this refinement.
As a preparation to the proof of Theorem 4, let us recall two results on outer billiards. The first is an outer analog of the "mirror equation" of geometrical optics that plays an important role in the theory of inner billiards.
Let D be an outer billiard table with smooth boundary and T the outer billiard map. Let T (x) = y, see figure 6.
x l m y α β Figure 6: The differential of the outer billiard map and the mirror equation In the Cartesian coordinate system in figure 6, the differential of T at point x is given by the matrix where r = |xy|/2 and ρ is the radius of curvature of the outer billiard curve at the reflection point. If dT takes a line l to a line m, see figure 6, then one has the following "mirror equation": For a proof of Proposition 5.2, see [6]. The second result is the following "area construction", an outer counterpart of the string construction in the theory of inner billiards, that recovers a billiard table from its caustic. Proposition 5.3 Let D be an outer billiard table and C an invariant curve of the outer billiard map T . Let x ∈ C and T (x) = y. Then the area cut by the segment xy from C is the same for all x ∈ C. In other words, D can be recovered from C as an envelope of segments of constant areas.
For a proof of Proposition 5.3, see [3,13,14] or [6]. In particular, let C consist of two non-parallel lines, l and m, as in figure 6. Then the area construction yields a hyperbola for which the lines l and m are the asymptotes. If the outer billiard map T takes x to y and dT (l) = m, as in figure 6, then the hyperbola for which the lines l and m are the asymptotes is second order tangent to the outer billiard curve at the mid-point of the segment xy. Note also that of 4 possible orientations of the pair of lines l and m in figure 6, only two are consistent with the action of dT : one line is oriented toward their intersection point while the other is oriented from it.
Proof of Theorem 4. We give two proofs. The first is reminiscent of the one given in [16]. Consider figure 7. One has three outer billiard maps, T 1 , T 2 , T 3 that cyclically permute points z 1 , z 2 , z 3 : Let α i , i = 1, 2, 3, be the angles of the triangle, 2r i its side lengths and ρ i the curvature radii of the outer billiard curves at the reflection points. Set dT i = A i and let B i be matrix of rotation through angle π − α i .
Let A be the area of the triangle z 1 , z 2 , z 3 and h 2 its altitude from the vertex z 2 . Then by elementary geometry, 2r 1 sin α 3 = 2r 3 sin α 1 = h 2 , r 1 cos α 3 + r 3 cos α 1 = r 2 , A = h 2 r 2 , which makes it possible to rewrite (13) as For orbits z 1 , z 2 , z 3 in the set U, the area A remains constant. Move points z 1 and z 3 slightly along the line z 1 z 3 so that the mid-point of the segment z 1 z 3 remain the same. We obtain a family of orbits in U for which ρ 2 does not change but r 2 does, in contradiction with (14). This completes our first proof.
The second argument is a variation on the proof given in [1]. Let M 5 be the manifold of non-degenerate triangles whose oriented areas equal 1 and let F 3 be the dual Birkhoff distribution on M. If there is an open set of fixed points of the map T 3 • T 2 • T 1 then one has a 2-dimensional disc U ⊂ M tangent to F . We will show that this is impossible.
If such a disc U exists, let x and y be local coordinates in it. Then u = ∂/∂x and v = ∂/∂y are commuting vector fields on U.
As a basis of vector fields in M generating F we can choose the fields W 1 = (z 2 − z 1 )∂z 2 + (z 1 − z 3 )∂z 3 , W 2 = (z 3 − z 2 )∂z 3 + (z 2 − z 1 )∂z 1 and W 3 = (z 1 − z 3 )∂z 1 + (z 3 − z 2 )∂z 2 , see Section 2. Then where f i , g i are smooth functions on U. Without loss of generality, f 1 = 0 and g 2 = 0. Then we take a linear combination of u and v, with coefficients that are smooth functions, to obtain new fields so that [ū,v] ∈ Span(ū,v). In particular, [ū,v] is tangent to F . z 1 and z 2 having the direction of the side z 1 z 2 . Then dT (u) = v. Let w = dT (v) = dT −1 (u). Since T is an orientation preserving transformation, the vector w lies on the same sides of the lines z 2 z 3 and z 1 z 3 as the outer billiard table D. Thus w lies inside the wedge z 1 z 3 z 2 . Let A be the intersection point of the line through point z 3 in the direction of vector w with the line z 1 z 2 . It follows from the area construction (see the paragraph after Proposition 5.3) that the outer billiard curve is second order tangent to the hyperbola inscribed in the angle z 1 Az 3 and tangent to the segment z 1 z 3 in its mid-point. This hyperbola is convex outward, contradicting the convexity of D.
On the other hand, d(T 2 • T 1 • T 3 ) may be equal to the identity at a fixed point of the map T 2 • T 1 • T 3 . An example is shown in figure 8 on the right: z 1 z 2 z 3 is a unit equilateral triangle, and the three outer billiard tables are circles of radii 1/ √ 3. Then equation (14) holds, and d(T 2 • T 1 • T 3 )(z 1 ) = Id.