Bifurcation of Limit Cycles in Small Perturbations of a Class of Hyper-elliptic Hamiltonian Systems of Degree 5 with a Cusp

This paper deals with small perturbations of a class of hyper-elliptic Hamiltonian system, which is a Liénard system of the form ˙ x = y, ˙ y = Q1(x) + εyQ2(x) with Q1 and Q2 polynomials of degree 4 and 3, respectively. It is shown that this system can undergo degenerated Hopf bifurcation and Poincaré bifurcation, which emerge at most three limit cycles for ε sufficiently small.


Introduction
Consider a planar Hamiltonian system of the forṁ where P m (x) is a real polynomial in x of degree m. If the level set {H = h} contains ovals and all critical points are real, then the level sets are elliptic for m = 3, 4 and hyper-elliptic for m ≥ 5. In the progress to solve Hilbert's 16th problem, in recent years many studies have been devoted to the limit cycles bifurcation for elliptic Hamiltonian systems. To the best of our knowledge most studies in this direction concern the elliptic case. For instance, in a series of papers Dumortier and Li made a complete study on the limit cycles bifurcation for elliptic Hamiltonian systems (see [1,2,3]).
In this paper, we study a small perturbation of Hamiltonian vector field with a hyper-elliptic Hamiltonian of degree five. The topological classification of hyperelliptic Hamiltonian systems of degree five was given first by Gavrilov and Iliev in [5]. There are ten topologically different phase portraits for the hyper-elliptic Hamiltonian system. Wang and Xiao in [11] considered one of these cases and made a complete study on small perturbations of Hamiltonian vector field with a hyper-elliptic Hamiltonian having a nilpotent saddle. They showed that this system can undergo degenerated Hopf bifurcation and Poincaré bifurcation, which emerges at most three limit cycles in the plane. Furthermore they showed that the limit cycles can encompass only an equilibrium inside, i.e. the configuration (3, 0) of limit cycles can appear for some values of parameters. Recently, Yang and Han in [13] studied systems of these classes with a cuspidal loop and a homoclinic loop and they obtained new results on the lower bound of the maximal number of limit cycles for these systems.
Here we choose another class in [5] to study the number of limit cycles under a small perturbation. Consider the Hamiltonian systeṁ with Hamiltonian function H(x, y) = 1 2 y 2 + 1 3 which has a cusp point C(−1, 0), a non-degenerate center O(0, 0), a hyperbolic saddle S(2/3, 0) and a heteroclinic loop γ 4 45 (see Fig. 1). Inside γ 4 45 , all orbits {γ h } are closed, By Xiao [12] in (1) one can assume Hence according to the classification in [12], the system (H 0 ) is the case corresponds to α = 2 5 and β = 1 (up to a linear transformation). We intend to study the following Liénard system which is a perturbation of (H 0 ): Then associated to the given perturbation there exists the so-called first order Melnikov function or the Abelian integral where I k (h) = γ h x k ydx, and γ h is oriented clockwise. Here 0 < ε 1 and a, b and c are real bounded parameters. A limit cycle is an isolated periodic orbit in the set of periodic orbits. The Melnikov function I(h) is a suitable tool for studying limit cycles of system (H ε ). We recall that a limit cycle of system (H ε ) corresponds to an isolated zero of the Melnikov function I(h). Our main result is the following: from which at most three limit cycles emerges in the plane. Furthermore they show that the limit cycles can encompass only an equilibrium inside, i.e. the configuration (3, 0) of limit cycles can appear for some values of parameters.
In this paper, we intend to study a Liénard system of type (4,3) that is a small perturbation of Hamiltonian vector field with a hyper-elliptic Hamiltonian of degree five. The topological classification of hyper-elliptic Hamiltonian system of degree five was first given by Gavrilov and Iliev in [5]. There are eleven different phase portraits for the hyper-elliptic Hamiltonian system, which have the family of ovals. We study the number of limit cycles of one of these classes under a small perturbation. Consider the Hamiltonian systeṁ with Hamiltonian function which has a cusp point C(−1, 0), a non-degenerate center O(0, 0), a hyperbolic saddle S(2/3, 0) and a heteroclinic loop γ 4/45 (see Figure 1). Inside γ 4/45 , all orbits {γ h } are closed, We intend to study a perturbation of (H 0 ) of the form: x = y, Main Theorem. System (H ε ) can undergo degenerated Hopf bifurcation and Poincaré bifurcation, which emerge at most three limit cycles for ε sufficiently small. Moreover there are values of parameters (a, b, c) for which system (H ε ) can have three limit cycles. We split our main theorem into three theorems and prove them in the sequel as follows: In section 2, we consider the local stability of the equilibrium solution of system (H ε ), and we prove that system (H ε ) can undergo degenerated Hopf bifurcation which emerges at most three limit cycles in any compact region inside the heteroclinic loop γ 4 45 of Hamiltonian system (H 0 ). In Section 3, we show that Abelian integrals I(h) of system (H ε ) has the Chebyshev property, i.e. the least upper bound of number (multiplicity taken into account) of zeros of I(h) is three. This implies that system (H ε ) can undergo Poincaré bifurcation which emerges at most three limit cycles from this period annulus if I(h) is not identically zero. In section 4, we study the asymptotic expansions of the Abelian integrals I(h) at the center and the heteroclinic loop. By the asymptotic expansions of Abelian integrals I(h) at the end points of open interval (0, 4/45), we show that there exist parameter values such that I(h) has three isolated zeros .

Local stability analysis and Hopf bifurcation
In this section we will consider the local stability of equilibrium solutions of system (H ε ), and discuss the number of small limit cycles. We show that the system (H ε ) can undergo degenerated Hopf bifurcation which emerges at most three limit cycles near equilibrium O(0, 0). Clearly, system (H ε ) always has three equilibria C(−1, 0), O(0, 0) and S(2/3, 0) for each value of parameters (a, b, c). In the following lemma we give a detailed analysis of all possible dynamics of these equilibria.
Then equilibrium S(2/3, 0) is a hyperbolic saddle, equilibrium O(0, 0) is a focus and C(−1, 0) is a saddle-node or a cusp. More precisely we have: In order to prove the above lemma the following lemmas will appear to be useful (see [7,9]) Suppose that F (x) and g(x) are smooth functions in a neighborhood of the origin, and that Then the equilibrium (0, 0) of (4) is a multiple focus of multiplicity k if B j = 0, j = 1, 2, . . . , 2k, and B 2k+1 = 0. Furthermore, it is locally stable (unstable) if B 2k+1 < 0 (B 2k+1 > 0, respectively).
where f (x), g(x) are continuously differentiable functions on the open interval (α, β). Suppose that Then system (5) has no closed orbits in the strip α < x < β. Now we are ready to prove lemma 2.1. Proof of lemma 2.1. The statements (S) and (O i ) can be proved by a straightforward calculation of the eigenvalues of system (H ε ) at S(2/3, 0) and O(0, 0), respectively. To prove the statements (O ii ) − (O iv ), we shall apply Lemma 2.2. First, using the translations X = x, Y = y − ε( 1 2 bx 2 + 1 3 cx 3 + 1 4 x 4 ), we transform system (H ε ) to the following Liénard systeṁ . For convenience, we still use x, y instead of X, Y , respectively. It is clear that F (x), and g(x) are C ∞ smooth functions and g(0) = F (0) = F (0) = 0 and g (0) > 0. Let Here O(x k ) stands for terms of higher order than x k−1 . Performing a Taylor expansion of function F (α(x)) − F (x) at x = 0, we obtain . In the following we study the equilibrium C(−1, 0). Moving C(−1, 0) to the origin, the system (H ε ) becomesẋ If c − b + a − 1 = 0, then the eigenvalues of system (6) at (0, 0) are zero and µ = 0. Therefore, (0, 0) is a degenerate equilibrium for (6). In order to determine the local stability of (0, 0) we let Using this transformation system (6) becomes where By implicit function theorem, we know that, there exists a smooth function Y = ϕ(X) and a small positive number δ such that ϕ(X) + q 2 (X, ϕ(X)) = 0 for |X| < δ, where Therefore, p 2 (X, ϕ(X)) = − 5 3µ 2 X 2 + O(X 3 ). According to theorem 3.5 in [4], we obtain that the equilibrium (0, 0) is a saddle-node. This implies the statement (C i ). If c−b+a−1 = 0, then the eigenvalues of system (6) are two zeros and the linearized matrix is not zero matrix. Hence, in this case the equilibrium (0, 0) is nilpotent.
From theorem 3.5 in [4], we obtain that (0, 0) is a cusp for system (6). This implies statement (C ii ). From lemma 2.1 and Hopf bifurcation theorem, we can see that there are three surfaces that when the parameters a, b and c pass through them the equilibrium O(0, 0) can undergo a series of Hopf bifurcations for any given ε with 0 < ε 1. In fact, the Hopf bifurcation surface of codimension one is given by And in the closure of H 1 there is a curve which is a degenerate Hopf bifurcation curve of codimension two. In the closure of this curve, there is point which is a degenerate Hopf point of codimension three. Moreover, we have the following theorem. and c decreases from −5/2; (ii) a unique unstable limit cycles bifurcates from equilibrium O(0, 0) as a = 0, c < −5/2 and b increases from −5; (iii) a unique stable limit cycle bifurcates from equilibrium O(0, 0) as c < −5/2, b > −5 and a increases from zero. Therefore, as a > 0, b > −5 and c < −5/2 system (H ε ) has three limit cycles surrounding equilibrium O(0, 0), in which two limit cycles are stable and the other is unstable.
Theorem 2.4 implies that the maximum number of small amplitude limit cycles which bifurcate from equilibrium O(0, 0) of system (H ε ) is three. Next we deduce some results concerning the large limit cycles of system (H ε ). For this we state some preliminaries and related definition from [6] about concepts of resultant of two polynomials and Sturm's Theorem. Given two polynomials p, q ∈ C[x, y] say p(x, y) = a 0 x m + · · · + a m , with a 0 = 0 q(x, y) = b 0 x n + · · · + b n , with b 0 = 0 where a i , b i ∈ C[y], the resultant of p and q with respect to x denoted by Res(p, q, x) is determinant of a (m + n) × (m + n) matrix defined in terms of coefficients of p and q. One of the basic properties of resultant is that Res(p, q, x) vanishes at any common solution of p(x, y) = q(x, y) = 0 (See appendix 5.1 of [6] for details).

f m does not vanish on [a, b].
3 Then we have the following theorem. Proof. We prove the part (i) by contradiction. Suppose that system (H ε ) has a closed orbit γ surrounding E(2/3, 0). Then γ crosses line x = 2/3 and positive x-axis respectively at P (2/3, y p ), Q(2/3, y q ) and R(x r , 0), where y q < 0 < y p and x r > 2/3. Hence, the vector field of system (H ε ) at P (2/3, y p ) is (y p , ε(a + (2/3)b + (4/9)c + 8/27)y p ), and vector field of system (H ε ) at R(x r , 0) is (0, x r (x r + 1) 2 (x r − 2/3)). Since the orientation of vector field on γ at R is counterclockwise while the orientation of vector field on γ at P is clockwise. This is a contradiction. Thus part (i) is proved. Next we prove part (ii) by applying Lemma 2.3. From Lemma 2.1 we know that system (H ε ) has a weak focus of order three when a = 0, b = −5 and c = −5/2. In (H ε ) we set a = 0, b = −5 and c = −5/2, and transfer the system to the following systemẋ where 3 . Now we investigate if system (8) has a closed orbit for −1 < x < 2/3. It is clear that xg 1 (x) > 0 for −1 < x < 2/3 and x = 0. Let

Now, we consider if the equations
have a solution (u, x) with −1 < u < 0 and 0 < x < 2/3. By a straightforward computation, the equations (9)  We now calculate the Sturm's sequence of polynomial R 1 (x) and the number of sign reversal of them in the interval (0, 2/3) by Maple. We obtain that the number of sign reversal is zero. By Sturm's Theorem we have that R 1 (x) < 0 for 0 < x < 2/3. Therefore equations (9) have no solution (u, x) with −1 < u < 0 and 0 < x < 2/3. Now, according to Lemma 2.3 we deduce that system (8) does not have a closed orbit for −1 < x < 2/3. This implies part (ii). This ends the proof of Lemma.

Bifurcation of limit cycles from the period annulus
In this section we study the maximum number of limit cycles which bifurcate from the period annulus of system (H 0 ) for 0 < ε 1. We use an algebraic criterion developed in [6] to study the related Abelian integral I(h) of system (H ε ). As a matter of fact, we will show that the base functions {I 0 (h), I 1 (h), I 2 (h), I 3 (h)} in the Abelian integral I(h) form a Chebeyshev system. Hence, the Abelian integral I(h) of system (H ε ) has the Chebeyshev property, i.e. the number (multiplicity taken into account) of isolated zeros of I(h) in the open interval (0, 4/45) is at most three. Also, using the asymptotic expansions of Abelian integrals I(h) near the end points of open interval (0, 4/45), we obtain that by Poincaré bifurcation, if I(h) is not identically zero, the number of isolated zeros of I(h) in the open interval (0, 4/45) is at least three. Therefore, the maximum number of limit cycles of system (H ε ) bifurcating from the period annulus is three. Our main result in this section is the following theorem: Theorem 3.1. Consider the system (H ε ) and the related Abelian integral (3). Then, the collection {I 0 (h), I 1 (h), I 2 (h), I 3 (h)} is an extended complete Chebeyshev system on the interval (0, 4/45). Therefore, if the Abelian integral I(h) is not identically zero then it has at most three zeros, counting multiplicities, in any compact subinterval of (0, 4/45) and for all values of parameters (a, b, c). And the number of limit cycles of limit cycles bifurcating from the period annulus is at most three.
In order to prove theorem 3.1, first we recall some preliminaries, the algebraic criterion and related definition. The reader is referred to [6] for details about the criterion. . . , f n−1 } is said to be Chebeyshev system provided that any nontrivial linear combination has at most n − 1 isolated zeros on J.
(iv) The continuous Wronskian of Consider a Hamiltonian function with the following special form which is analytic in some open subset of the plane and has a local minimum at the origin. We fix that H(0, 0) = 0, then there exist a punctured neighborhood P of the origin foliated by the ovals or period annulus γ h ⊂ {H(x, y) = h}. The period annulus can be parameterized by the energy levels h ∈ (0, h 0 ) for some h 0 ∈ (0, +∞]. In the sequel, we denote the projection of P on the x-axis by (x , x r ). It is easy to verify that, under the above assumptions, xA (x) > 0 for any x ∈ (x , x r ) \ {0} and B(0) > 0. Thus there exists a smooth invertible function z(x) with x < z(x) < 0 such that A(x) = A(z(x)) for 0 < x < x r . Theorem B in [6] is as follows. Theorem B. Let us consider the Abelian integrals where, for each h ∈ (0, h 0 ), γ h is the oval surrounding the origin inside the level curve {A(x) + B(x)y 2m = h}. We define A (z(x))(B(z(x))) 2s−1 2m .
Then {I 0 , I 1 , . . . , I n−1 } is an extended complete Chebeyshev system on (0, h 0 ) if {l 0 , l 1 , . . . , l n−1 } is a complete Chebeyshev system on (0, x r ) and s > m(n − 2). The efficiency of Theorem B comes from the fact that the requirement of some functions to be complete Chebeyshev system can be verified by computing Wronskians.
The following well known result will clarify this fact (see [6]).
In order to determine if the four Wronskians have zeros on (0, 2/3) , we invoke the symbolic computations by Maple 12 to compute the resultant between two polynomials, and apply Sturm's Theorem to claim the nonexistence of zeros of a polynomial in an interval. We have the following lemma.  Proof. We now compute four Wronskians in four cases and check if they have zeros on (0, 2/3).