Symmetrical Symplectic Capacity with Applications

In this paper, we first introduce the concept of symmetrical symplectic capacity for symmetrical symplectic manifolds, and by using this symmetrical symplectic capacity theory we prove that there exists at least one symmetric closed characteristic (brake orbit and $S$-invariant brake orbit are two examples) on prescribed symmetric energy surface which has a compact neighborhood with finite symmetrical symplectic capacity.


Introduction and main results
The purpose of this paper is to study the existence of the symmetric periodic solutions of Hamiltonian systems in the presence of symmetry for the manifold and also for the Hamiltonian functions. A very famous example is the figure-eight orbit in planar threebody problem with equal masses(see [2]). It is the orbit with two different symmetries: cyclic symmetry and generalized brake symmetry. In this paper we consider the existence of symmetric orbits of smooth Hamiltonian systems with some symmetries. An important case is the existence of brake orbits on the manifolds with the brake symmetry. For this purpose, we first study the symmetrical symplectic capacity theory for the symplectic manifolds with corresponding symmetry.
Symplectic capacity is an important symplectic invariant. It was first discovered by I. Ekeland and H.Hofer in [3] and [4] for subsets of R 2n in their search for periodic solutions of Hamiltonian systems on fixed energy surfaces. We call it the Ekeland-Hofer capacity and denote it by c EH . This concept was extended to general symplectic manifolds by H.Hofer and E.Zehnder in [11] and [12]. We call it the Hofer-Zehnder capacity and denote it by c 0 . As examples of symplectic capacity, Gromov's width W G defined in ( [5]) is the smallest symplectic capacity, Hofer's displacement energy d defined in ( [9]) is also a symplectic capacity, the Floer-Hofer capacity c F H defined in ( [6]) which can be viewed as a variant of Ekeland-Hofer capacity c EH , and Viterbo's generating function capacity c V defined in ( [31]) is also a symplectic capacity. The symplectic capacities were applied to the study of many symplectic topology problems, see [12], [22], [30] and the references therein for more details.
In this paper, we introduce a symmetrical capacity on some symmetrical symplectic manifolds. For the brake symmetry case, we say that a symplectic manifold (M, ω) is brake symmetrical (ϕ-symmetric) if there is an antisymplectic involution ϕ : M → M satisfying ϕ 2 = id, ϕ * ω = −ω and the fixed point set Fix(ϕ) = ∅. It is well known that the fixed point set L of ϕ is a Lagrangian submaifold of M if it is not empty. We denote the ϕ-symmetric symplectic manifold M by (M, L, ω, ϕ). For a ϕ-symmetric symplectic manifold (M, L, ω, ϕ), in this paper we first develop a symmetrical symplectic capacity c ϕ (M) in subsection 2.1. When a symplectic manifold (M, ω) is provided with two different symmetries, for example, the ϕ-symmetry and a cyclic symmetry S, for some special cases we also introduce a capacity c ϕ,S (M). For example, in (R 2n , ω 0 ), we choose ϕ as a linear mapping N 0 : R 2n → R 2n with N 0 = −I n 0 0 I n , and an orthogonal symplectic matrix S with S m = id for some 2 ≤ m ∈ N, we introduce a symmetrical capacity c N 0 ,S (U) for (N 0 , S) invariant subset U of R 2n in subsection 3.1.
We note that for a general symplectic manifold it is not easy to determine the finiteness of its Hofer-Zehnder's capacity. There are few results about the finiteness of symplectic capacity for some special symplectic manifolds(see for example [10], [13], [17], [21]). It's also difficult for us to prove the finiteness of symmetrical symplectic capacity in general, in neighborhood U with c ϕ (U, ω) < ∞, then Σ possesses a closed brake-characteristic.
We note that for a compact ϕ-contact type hypersurface Σ in (R 2n , ω 0 ) with ϕ = N 0 , it is clear that Σ has a ϕ-invariant neighborhood U with c ϕ (U, ω) < ∞. So Σ always possesses a closed brake-characteristic.
In section 3.2, as applications of c N 0 ,S , we consider the existence of S-symmetrical brake orbits on energy hypersurface in (R 2n , ω 0 ), and get the following result.
Then there is an open neighborhood U of Σ such that c N 0 ,S (U) < ∞ and there exists a sequence λ j → 1,j → +∞, such that every energy surface Σ λ j = H −1 (λ j ) possesses a S-symmetrical brake orbit of the Hamiltonian vector field X H .
We shall note that on a fixed energy surface there may be no closed characteristic(see [7], [8] for counter examples). But for the case of (R 2n , ω 0 ) as considered in Example 2.1 below, if the the N 0 -invariant hypersurface Σ = H −1 (1) is star-shaped, Rabinowitz in 1987 [25] proved that if x · H ′ (x) = 0 for all x ∈ Σ then there exist at least one brake orbits on Σ, which has been generalized by Corollary 2.21 below in this paper. If the [28] proved that it possesses at least n geometrically distinct brake orbits. If the N 0 -invariant hypersurface Σ = H −1 (1) is convex and central symmetric, that is Σ = −Σ, Y. Long, D. Zhang and C.
Zhu in 2006 [20] proved that Σ possesses at least two geometrically distinct brake orbits.
Recently, in 2009 [19], D. Zhang and the first author of this paper proved that a convex and central symmetric hypersurface Σ ⊂ R 2n possesses at least [ n 2 ] + 1 geometrically distinct brake orbits, and if all brake orbits on Σ are nondegenerate, then Σ possesses at least n geometrically distinct brake orbits.
For brake boundary value problems of non-autonomous Hamiltonian, one can refer the papers [18], [32] and [35]. For the existence and multiplicity of closed characteristics on prescribed energy surface, one can further refer the papers [15,16,23,24,27,29,33,34] and the references therein.

Symmetrical Symplectic Capacity
From Theorem 2.4, we see that the symmetric symplectic capacity c ϕ (M, ω) satisfies all properties of the general symplectic capacity in the sense of symmetric category(c.f., [3,4,11,12]). The proof of Theorem 2.4 is similar to that as in [3,4,11,12]. We complete the proof of Theorem 2.4 via the following lemmas.

Lemma 2.5. c ϕ satisfies the properties (A) and (B).
Proof. We divided the proof into two steps.
Step 1. Proof of property (A). Define a map φ * : . This implies the property (A). Step Therefor, X Hα and X H have the same brake orbits with the same periods. It implies that ψ is also a bijection between H a (M, L, ω, ϕ) and H a (M, L, αω, ϕ). Thus the property (B) is true. 2 For the proof of property (C), we note that it is enough to prove it for N = N 0 . In fact, there exists an orthogonal symplectic matrix P satisfying That is to say P : diffeomorphism satisfying the condition in property (A), so we have A brake orbit is naturally a periodic orbit, from the definitions of c 0 and c ϕ , there holds In order to prove property (C), we need to prove c N 0 (B(1)) ≤ π. By property (A), it is enough to prove Then there is a compactly supported symplectic diffeomorphism of Z(1), ψ : Define a smooth function ρ : R 2n → R with compact support in A(2δ) ⊂ Z(1) by The flow ψ t : Z(1) → Z(1) of the Hamitonian vector field X K is compact supported symplectic diffeomorphim for every t > 0. We define ψ = ψ 1 the time-1 map. It has to a function defined on the whole space R 2n . This is possible since H is constant near the boundary of Z(1). Denote by where z = (x, y) ∈ R 2n , and K ∈ Z + is sufficiently large. It is clear that q K (z) = q K (N 0 z).
Since H ∈ H(Z(1)) satisfies m(H) > π, there is an ε > 0 such that m(H) > π + ε. We can take a smooth function f : The extension of H is now defined bȳ andH is quadratic at infinity, exactly we havē for some large R. The following crucial lemma describes the distinguished brake orbit we are looking for.
with period 1. If it satisfies Since a brake orbit is a special periodic orbit, the proof of the lemma is the same as the proof of Proposition 2 in [P 74 , [12]].
The remaining of this subsection is to find a 1-periodic brake orbit x(t) of the equation (2.14) satisfying (2.15). We simply replaceH by H in the sequel.
Denote by The space L 2 is a Hilbert space with the usual L 2 inner product ·, · 0 and associated norm · 0 . Denote by The space H s is a Hilbert space with inner product and associated norm defined by x, y s = x 0 , y 0 + 2π k∈Z |k| 2s x k , y k , (2.18) for x, y ∈ H s . Denote by X = H 1/2 , · = · 1/2 , ·, · = ·, · 1/2 .
There is an orthogonal splitting of X e 2πjJt x j } and X 0 = L 0 .
The corresponding orthogonal projections are denoted by P + , P − , P 0 . Therefore, every x ∈ X has a unique decomposition We define for x, y ∈ X a(x, y) which is a continuous bilinear form on X. The functional a : X → R, defined by so the gradient of a is We have X ⊂ L 2 , the inclusion map is compact. Its adjoint operator is defined by Lemma 2.8. j * is compact and there hold j * (L 2 ) ⊂ H 1 and j * (y) 1 ≤ y 0 .
Proof. By direct computation, we have for any y = j∈Z e 2πjJt y j ∈ L 2 , where i * is the projection map: R 2n → L 0 = {0} ⊕ R n . From the definition of L 2 and H s , we can complete the proof. 2 We next consider the functional since H vanishes in a neighborhood of the origin, and from (2.13), there is M > 0 such that so the functional b can be defined for x ∈ L 2 and hence also for x ∈ X ⊂ L 2 .
Lemma 2.9. There holds b ∈ C 1 (X, R), ∇b : X → X maps bounded sets into relatively compact sets. Moreover, Proof. We have The proof is complete. 2 Now we consider the functional We have Φ : X → R is differentiable and its gradient is given by Lemma 2. 10. Assume x ∈ X is a critical point, i.e., ∇Φ(x) = 0. Then x ∈ C 2 (S 1 ) and it is a brake orbit with 1-periodic.
Proof. Let be a critical point of Φ, and ∇H(x(t)) = j∈Z e j2πJt a j ∈ L 2 , a j ∈ R 2n .
Proof. In fact we will prove that every sequence {x j } ⊂ X satisfying ∇Φ(x j ) → 0 contains a convergent subsequence. Assume ∇Φ(x j ) → 0 so that If x j is bounded in X, then x 0 j ∈ R 2n is bounded, and from Lemma 2.9, we see that {x j } has a convergent subsequence. To prove that x j is bounded we argue by contradiction and assume x j → ∞. Define so y k = 1. By assumption, from (2.29), Since |∇H(z)| ≤ M|z|, the sequence is bounded in L 2 . Since j * : L 2 → X is compact, (P + − P − )y k is relatively compact, and since y 0 k is bounded in R 2n , the sequence y k is relatively compact in X. After taking a subsequence we can assume y k → y in X and hence y k → y in L 2 . From (2.13), we have where Q(z) = (π + ε)q(z). Since |∇H(z) − ∇Q(z)| ≤ M for all z ∈ R 2n and since ∇Q defines a continuous linear operator of L 2 , we conclude This implies that y ∈ X solves the linear equation in X y + − y − − j * ∇Q(y) = 0, y = 1.
As in Lemma 2.10 one verifies that y solves the linear Hamiltonian equatioṅ y(t) = J∇Q(y(t)).
Recall now that Q = (π + ε)q, and q(z) = (x 2 1 + y 2 1 ) + 1 K 2 n j=2 (x 2 j + y 2 j ). We see that the symplectic 2-planes {x j , y j } are filled with periodic solutions of J∇Q having periods T = 1. Since the linear equation does not admit any nontrivial periodic solutions of period 1 we conclude y(t) ≡ 0. This contradicts y = 1 and we conclude that the sequence {x k } must be bounded. 2 ∇Φ is globally Lipschitz continuous, so the gradient equatioṅ which maps bounded sets into bounded sets.
Lemma 2.12. The flow ofẋ = −∇Φ(x) has the following form where K : R × X → X is continuous and maps bounded sets into precompact sets.
By the uniqueness of the initial value problem ξ(t) = 0 so that y(t) = x · t as required. In view of (2.29) we can write (e t−s P − + P 0 + e −t+s P + )∇H(j(x · s))ds}.
Proof. From a| X − ⊕X 0 ≤ 0 and b ≥ 0 we have We shall deal with the functional on those parts of the boundary ∂Ω τ which are defined by x − + x 0 = τ or s = τ . By the construction of H there exists a constant γ > 0 such that Therefore, Recalling the definition of the quadratic form q, one verifies for Recalling that e + 2 = 2π , for Consequently there exists a constant c > 0 such that The right hand side is not positive if x − + x 0 = τ or s = τ for τ sufficiently large. The proof of the lemma is complete. 2 The subset Γ = Γ α ⊂ X + is defined by Lemma 2.15. There exist α > 0 and β > 0 such that Proof. The space X is continuously embedded in L p (S 1 ) for every p ≥ 1. Hence there is Observing that |H(z)| ≤ c|z| 3 for all z ∈ R 2n , we can take a constant K > 0 such that for all x ∈ X. Now, if x ∈ X + , then Φ(x) ≥ 1 2 x 2 − K x 3 and the lemma is now obvious for some small α > 0 and β > 0. Since Φ(ϕ t (x)) decreases in t we conclude immediately from Lemma 2.14 and Lemma 2.15 that ϕ t (∂Ω) ∩ Γ = ∅ for all t ≥ 0. But the following result tell us that ϕ t (Ω) ∩ Γ = ∅.
Proof. We shall use the Leray-Schauder degree. Abbreviating the flow by ϕ t (x) ≡ x · t, we need to verify that (Ω · t) ∩ Γ = ∅ for all t ≥ 0. We can rewrite this by requiring Recall that, by Lemma 2.12, the flow has the representation x · t = e t x − + x 0 + e −t x + + K(t, x), so that (2.39) becomes Multiplying the X − part by e −t one gets the following equivalent equations Since x ∈ Ω is represented by x = x − + x 0 + se + , with 0 ≤ s ≤ τ , we can rewrite (2.41) as follows: x + B(t, x) = 0 and x ∈ Ω, (2.42) where the operator B is defined by provided that αe + ∈ Ω, which holds true for τ > α. This finishes the proof of Lemma

2
Now we can finish the proof of Proposition 2.13. We shall apply the minimax argument.
We take the family F consisting of the subsets ϕ t (Ω), for every t ≥ 0 and define We claim that c(Φ, F ) is finite. Indeed, since ϕ t (Ω) ∩ Γ = ∅ and Φ| Γ ≥ β we conclude In the last estimate of (2.48) we have used that Φ maps, in view of Lemma 2.9, bounded sets into bounded sets. Therefore, We know already that the functional Φ satisfies the (PS) condition (Lemma2.11). Moreover, the family F is invariant under the negative gradient flow ϕ t for t > 0. Consequently the Minimax Lemma implies that c(Φ, F ) is a critical value. We deduce that there is a point x * ∈ X satisfying ∇Φ(x * ) = 0 and Φ(x * ) = c(Φ, F ) ≥ β > 0, and the proof of Proposition 2.13 is complete.

Application to the Existence of Brake Orbit
In this subsection, we use the symmetrical symplectic capacity theory developed in the previous subsection to solve the existence of brake orbits on energy surfaces.
Let (M, L, ω, ϕ) be a symmetrical symplectic manifold, and H ∈ C 2 (M, R) satisfying Suppose that the energy surface is compact and regular, i.e., dH(x) = 0 for x ∈ Σ, (2.51) and Σ ∩ L = ∅ with transversal intersections. Thus Σ ⊂ M is a smooth and compact submanifold of codimension 1 whose tangent space at x ∈ Σ is given by We define an open and bounded neighborhood U of Σ by where I = (1−ε, 1+ε) for some small ε > 0, and Σ λ = {x ∈ M|H(x) = λ} is diffeomorphic to Σ with Σ λ ∩ L = ∅ for all λ ∈ I. Indeed, the gradient ∇H = 0 in a neighborhood of Σ, in view of (2.53). The modified gradient flow ψ t 0 defined by the following equatioṅ is transversal to Σ, and there holds This means that ψ t 0 : Σ → Σ 1+t is a diffeomorphism. Since H(ϕ(z)) = H(z), we have ϕ(U) = U. Similar to Theorem 1 in P 106 of [12], we have the following result which is equivalent to Theorem 1.1. Proof. Suppose I = (1 − ρ, 1 + ρ) for some small ρ > 0. For 0 < ε < ρ, we define a smooth It is easy to see that F ∈ H(U, U ∩ L, ω, ϕ) and m(F ) > c ϕ (U). Consequently, in view of the definition of the capacity c ϕ (U), there exists a nonconstant brake orbit (T, x(t)) with 0 < T ≤ 1 of the Hamiltonian system: Moreover, Thus, in view of the definition of the function f , the value λ belongs to the set 1 − ε < λ < 1 − ε 2 or 1 + ε 2 < λ < 1 + ε. In particular |λ − 1| < ε. By rescaling, we define which has period |τ |T and satisfiesẏ (t) = X H (y(t)), hence y(t) is a brake orbit of the original Hamiltonian vector field X H on the energy surface H(y(t)) = λ. Duo to the arbitrariness of ε, 1 is the limit point of λ such that Σ λ possesses a brake orbit and so is true for all point in I. (2.54) Proof of Theorem 1.2. We follow the ideas of the proofs of Theorem 5 and Theorem 6 in P 123 of [12]. Let X be the vector field defined in Definition 2.20. Since Σ is compact and X is transversal to Σ, the map defined by Ψ(x, t) = ψ t (x) for x ∈ Σ and t < ε is a diffeomorphism onto an open neighborhood U of Σ provided ε > 0 is sufficiently small, where ψ t is the flow of X.
From L X ω = ω we conclude that if x(s) is a closed brake characteristic on Σ, then y(s) = ψ t (x(s)) will be a closed brake characteristic on Σ t = ψ t (Σ), then from Theorem 2.17, we complete the proof. 2 Consider the Example 2.1, from the results above, we have the following result. It is easy to say that if Σ is N 0 -invariant and star-shaped with center at origin, then Σ is N 0 -contact type, and Corollary 2.21 generalize the result of [25].
For further applications, we need the following lemma: Lemma 2.23. A compact hypersurface Σ ∈ S ϕ is of ϕ-contact type if and only if there exists a 1-form α on a neighborhood U of Σ and constant λ = 0 such that    dα = λ ω, α(ξ) = 0, for 0 = ξ ∈ L Σ , ϕ * (α)(x) = −α(ϕ(x)), ∀z ∈ U, that is to say The remains of the proof is similar to [12]. 2 belongs to S N 0 (see Definition 2.19) and is of N 0 -contact type.
Define y(t) = x(t/λ) we have (λT, y(t)) is a S-symmetrical brake orbit of H, and M ≤ H(y(t)) ≤ M + ε. Since M and ε are arbitrary the theorem is proved. 2 Remark. Similar to Theorem 1.2, we can prove that every compact (N 0 , S)-contact type hypersurface Σ in R 2n with Σ ∩ L 0 = ∅ possesses an S-symmetric closed brake characteristic. We note that the "figure-eight orbit" is a special case.