The transfer operator for the Hecke triangle groups

In this paper we extend the transfer operator approach to Selberg's zeta function for cofinite Fuchsian groups to the Hecke triangle groups G_q, q=3,4,..., which are non-arithmetic for q \not= 3,4,6. For this we make use of a Poincare map for the geodesic flow on the corresponding Hecke surfaces which has been constructed in arXiv:0801.3951 and which is closely related to the natural extension of the generating map for the so called Hurwitz-Nakada continued fractions. We derive simple functional equations for the eigenfunctions of the transfer operator which for eigenvalues rho =1 are expected to be closely related to the period functions of Lewis and Zagier for these Hecke triangle groups.


1.
Introduction. This paper continues our study of the transfer operator for cofinite Fuchsian groups and their Selberg zeta functions [3], [4]. For the modular groups, i.e. finite index subgroups Γ ⊂ SL(2, Z), the transfer operator approach to Selberg's zeta function [3] has led to interesting new developments in number theory, like the theory of period functions for Maass wave forms by Lewis and Zagier [9]. Obviously, it is necessary to extend this theory to more general Fuchsian groups, especially the nonarithmetic ones. One possibility for such a generalization is via a cohomological approach [1], which has been worked out for the case G 3 = SL(2, Z) recently in [2]. We concentrate on the transfer operator approach to this circle of problems and started to work out this approach in [12], [13] for the Hecke triangle groups which, contrary to modular groups studied up to now, are mostly non-arithmetic.
The transfer operator has been introduced by D. Ruelle [18] among other reasons primarily to investigate analytic properties of dynamical zeta functions. A typical 2 DIETER MAYER, TOBIAS MÜHLENBRUCH AND FREDRIK STRÖMBERG example of such a function is the Selberg zeta function, Z S (s), for the geodesic flow on surfaces of constant negative curvature, which connects the length spectrum of this flow with spectral properties of the corresponding Laplacian. It is defined by where the outer product is taken over all prime periodic orbits γ of period l(γ) of the geodesic flow Φ t : SM → SM on the unit tangent bundle of the surface M.
The period l(γ) coincides in this case with the length of the corresponding closed geodesic. If P : Σ → Σ is the Poincaré map on a section Σ of the flow Φ t Ruelle showed that Z S (s) can be rewritten as where ζ R denotes the Ruelle zeta function for the Poincaré map P defined as r P k (x) , n ≥ 1, the so called dynamical partition functions and r : Σ → R + the recurrence time function with respect to the map P, defined through Φ r(x) (x) ∈ Σ for x ∈ Σ and Φ t (x) / ∈ Σ for 0 < t < r(x).
In the transfer operator approach for the modular groups the Selberg zeta function gets expressed in terms of the Fredholm determinant of an operator L s as Z S (s) = det(1 − L s ). From this relation it is clear that the zeros of Z S (s) are directly related to the values of s for which L s has eigenvalue one. Furthermore, the corresponding eigenfunctions in a certain Banach space of holomorphic functions can be directly related to the automorphic functions of these modular groups [4]. It is expected that this approach can be extended to all cofinite Fuchsian groups. In this paper we continue to work it out for the Hecke triangle groups and their corresponding surfaces. A Poincaré map P : Σ → Σ for the geodesic flow on the Hecke surfaces M q = G q \H, q = 3, 4, 5, . . . was constructed in [13]. Thereby H denotes the hyperbolic upper half-plane and G q the Hecke triangle group generated by the isometries S : z → −1/z and T : z → z + λ q with λ q = 2 cos π q . In [13] it was shown, that the map P is closely related to the natural extension F q : Ω q → Ω q of the generating map f q : I q → I q , I q = [− λq 2 , λq 2 ] of the Hurwitz-Nakada continued fractions [14], [17], also called λ q -continued fractions or shortly λ q -CF 's. These are closely related to the Rosen λ-continued fractions [16,17,19]. For a precise description of this relationship see e.g. [13] Remark 15. We recall the necessary facts about the λ q -continued fractions in §2. Contrary to the case of the modular surfaces, where a Poincaré map P : Σ → Σ has been constructed through the natural extension of the Gauss map T G : [0, 1] → [0, 1], T g (x) = 1 x mod 1, x = 0, in the present case of the Hecke surfaces M q there TRANSFER OPERATOR FOR HECKE TRIANGLE GROUPS   3 is not a one-to-one correspondence between the periodic orbits of the map f q generating the λ q -CF 's and the periodic orbits of the geodesic flow Φ t : SM q → SM q . Indeed there exist for every G q exactly two periodic points r q , −r q ∈ I q which correspond to the same periodic orbit O of the flow Φ t . In the case q = 3 Hurwitz already showed in [8] that there exist exactly two closed f 3 -orbits which are equivalent under the action of the group G 3 = SL(2, Z) and hence lead to the same orbit of the flow Φ t . As a consequence the Fredholm determinant det(1−L s ) of the Ruelle transfer operator L s of the the Hurwitz-Nakada map f q does not correctly describe the Selberg zeta function (1) for the Hecke triangle groups G q , since it contains the contribution of the closed orbit O twice. To correct this overcounting we introduce another transfer operator, K s , whose Fredholm determinant describes exactly the contribution of this orbit O to the Selberg function. The form of this operator can be directly deduced from the λ q -CF expansion of the point r q ∈ I q . The spectrum of the operator K s can be determined explicitly, leading to regularly spaced zeros of its Fredholm determinant, det(1 − K s ), in the complex s-plane. In in Section 6.2 we will use the operator K s to show the following formula for the Selberg zeta function for Hecke triangle groups: As in the case of the modular surfaces and the Gauss map T G , the holomorphic eigenfunctions − → g of the transfer operator L s fulfil simple functional equations. In the case q = 3 it was recently shown [2] that if s = 1 2 then there is a one-to-one correspondence between eigenfunctions of L s with eigenvalue 1 and Maass waveforms, i.e. square-integrable eigenfunctions of the Laplace-Beltrami operator, on the modular surface M 3 . We therefore expect that the holomorphic eigenfunctions − → g of the operator L s with eigenvalue ρ(s) = 1 can be related for all G q and general s to the automorphic functions of these Hecke triangle groups which almost all are non arithmetic. This will extend the transfer operator approach to the theory of period functions of Lewis and Zagier [9] to a whole class of non-arithmetic Fuchsian groups. We hope to come back to this question soon.
The structure of this article is as follows: In Section 2 we briefly introduce the Hecke triangle groups G q and recall the λ q -continued fractions as discussed in [12]. Section 3 recalls the geodesic flow on the Hecke surface M q , the Selberg zeta function and the construction of the Poincaré section Σ and the Poincaré map P : Σ → Σ in [13]. The transfer operator L s for the H-N map f q : I q → I q is discussed in Section 4. We show that it is a nuclear operator when acting in a certain Banach space B of vector-valued holomorphic functions whose dimension is determined by the Markov partitions for f q and has a meromorphic extension to the entire complex s-plane. In Section 5 we define a symmetry operator, P : B → B, commuting with the transfer operator. This allows us to restrict the operator L s to the two eigenspaces B , = ±1 of P . From this we derive the scalar functional equations which the eigenfunctions of the restricted transfer operators L s, are shown to fulfil. In Section 6 we discuss the Ruelle zeta function for the H-N map f q and show that it can be expressed in terms of Fredholm determinants of the operator L s respectively the reduced operators L s, . Finally, in section 6.2 we introduce the transfer operator, K s , whose Fredholm determinant describes the contribution of the orbit, O + corresponding to the point r q and which is needed to obtain (2) above.
2. λ q -continued fractions and their generating maps.
2.1. The Hecke triangle groups. Let denote the projective linear group, where SL(2, R) denotes the group of 2×2 matrices with real entries and determinant 1 and ±1 = ±1 0 0 ±1 . We usually identify the elements of PSL(2, R) with one of its matrix representatives in SL(2, R).
For an integer q ≥ 3, the Hecke triangle group G q ⊂ PSL(2, R) is the group generated by the elements where The elements S and T q satisfy the relations Later on we also need the element The action of PSL(2, R) on H is given by Möbius transformations: One can easily verify that g z ∈ H for z ∈ H and g x ∈ P R for x ∈ P R where H = {x + iy | y > 0} denotes the upper half-plane and P R = R ∪ {∞} denotes its boundary, the projective real line. We say that two points x, y ∈ H ∪ P R are G q -equivalent if there exists a g ∈ G q such that x = g y.
An element g ∈ PSL(2, R) is called elliptic, hyperbolic or parabolic depending on whether |Tr (g)| := |a + d| < 2, > 2 or = 2. The same notation applies for the fixed points of the corresponding Möbius transformation.
In the following we identify the element g ∈ PSL(2, R) with the induced map z → g z on H. Note that the type of fixed point is preserved under conjugation, g → AgA −1 , by A ∈ PSL(2, R). A parabolic fixed point is a degenerate fixed point, belongs to P R , and is usually called a cusp. Elliptic fixed points appear in pairs, z and z with z ∈ H, and G q (z), the stabilizer subgroup of z in G q , is cyclic of finite order. Hyperbolic fixed points also appear in pairs x, x ∈ P R , where x is said to be the repelling conjugate point of the attractive fixed point x.
2.2. λ q -continued fractions and their duals. Consider finite or infinite sequences [a i ] i with a i ∈ Z for all i. We denote a periodic subsequence within an infinite sequence by overlining the periodic part and a finitely often repeated pattern is denoted by a power, where the power 0 means absence of the pattern, hence [a 1 , a 2 , a 3 ] = [a 1 , a 2 , a 3 , a 2 , a 3 , a 2 , a 3 , . . .], Furthermore, by the negative of a sequence we mean the following: Put for even q and q−3 2 for odd q.
Next we define the set B q of forbidden blocks as The choice of the sign is the same within each block and m ≥ 1. For example [2,3], We call a sequence [a 1 , a 2 , a 3 , . . .] q-regular if [a k , a k+1 , . . . , a l ] ∈ B q for all 1 ≤ k < l and dual q-regular if [a l , a l−1 , . . . , a k ] ∈ B q for all 1 ≤ k < l. Denote by A reg q respectively by A dreg q the set of infinite q-regular respectively dual q-regular sequences (a i ) i∈N .

Special values and their expansions.
The following results are well-known (see [13] and [12, §2.3]). The point x = ∓ λ 2 has the regular λ q -CF for even q and for odd q.
The regular respectively dual regular λ-CF of the point x = R q has the form for even q, for odd q ≥ 5, and for even q, for q = 3.

Moreover,
R q = 1 and − R q = S R q for even q and and R q satisfies the inequality λ 2 < R q ≤ 1. . Denote by l(x) ≤ ∞ respectively l(y) ≤ ∞ the number of entries in the above λ q -CF's. We introduce a lexicographic order "≺" for λ q -CF's as follows: If a i = b i for all 0 ≤ i ≤ n and l(x), l(y) ≥ n, we define , a n > 0 > b n if n > 0, both l(x), l(y) ≥ n + 1 and a n b n < 0, a n < b n if n > 0, both l(x), l(y) ≥ n + 1 and a n b n > 0, b n < 0 if n > 0 and l(x) = n or a n > 0 if n > 0 and l(y) = n.
We also write x y for x ≺ y or x = y. This is indeed an order on regular λ q -CF's, since Lemmas 22 and 23 in [13] imply: Lemma 2.2. Let x and y have regular λ q -CF's. Then x ≺ y ⇐⇒ x < y.

2.5.
The generating interval maps f q and f q . Denote by I q and I Rq the intervals with λ q and R q given in (2.3) and (5). The nearest λ q -multiple map · q is defined as We also need the map · q given by The interval maps f q : I q → I q and f q : R q → R q are then defined as follows: These maps generate the regular respectively dual regular λ q -CF's in the following sense: For given x, y ∈ R the entries a i and b i , i ∈ Z ≥0 , in their λ q -CF are determined by the algorithms: (0) a 0 = x q and x 1 := x − a 0 λ q ∈ I q , (1) a 1 = −1 x1 q and x 2 : . . , ( ) the algorithm terminates if y i+1 = 0. In ( [12], Lemmas 17 and 33) it is shown that these algorithms lead to regular respectively dual regular λ q -CF's in the sense of section (2.2): 2.6. Markov partitions and transition matrices for f q . As shown in [12], both maps f q and f q have the Markov property. This means that there exist partitions of the intervals I q and I Rq into subintervals whose boundary points are mapped by f q respectively f q onto boundary points. In particular the following Markov partitions for f q have been constructed in ( [12]

The orbit O(−
We denote the elements of O(− λq 2 ) for q = 2h q + 2 by respectively for q = 2h q + 3 by The φ i 's then satisfy the ordering [13] define a Markov partition of the interval I q for the map f q . This means that holds, where S • denotes the interior of the set S.
As in [12] we introduce next a finer partition which is compatible with the intervals of monotonicity for f q .
In the case q = 3 where λ 3 = 1 define for m = 2, 3, 4, . . . the intervals J m as do not fall onto a boundary point of any of the intervals J m , m ∈ N the partition given by the intervals J m has to be refined.
For even q define the intervals J ±1i as This way one arrives at the partition M(f q ), defined as which is clearly again Markovian. Consider next the case of odd q ≥ 5. Here one has φ i ∈ J 1 for 1 ≤ i ≤ κ q − 2 and φ (κq−1) ∈ J 2 for = ±. For = ± define the intervals Then it is easy to see that the partition M(f q ) defined by A useful tool for understanding the dynamics of the map f q are the transition matrices A = A i,j i,j∈F , where F is the alphabet given by the Markov partition M(f q ): for even q and {±1 1 , . . . , ±1 κq−1 , ±2 κq , ±2 κq+1 , ±3, . . .} for odd q ≥ 5.
Each entry A i,j , i, j ∈ F is given by Tables 1, 2 and 3 show the transition matrices A for q = 3, even q respectively odd q ≥ 5, as determined in [12].
A ±k,m = 1, k ≥ 3, and all m ∈ F. Table 2. The nonvanishing matrix elements of the transition matrix A = (A i,j ) i,j∈F for even q.

The geodesic flow on Hecke surfaces. Let
A geodesic γ on H is either a half-circle based on R or a line parallel to the imaginary axis. The pair of base points of γ are denoted by γ ± ∈ P R such that the geodesic flow Φ t : SH → SH along the oriented geodesic γ satisfies lim t→±∞ φ t = γ ± . We identify an oriented geodesic γ on H with the pair of its base points (γ − , γ + ).
3.1. A Poincaré map for Φ t : SM q → SM q and its Ruelle zeta function. The Hecke surfaces M q , hyperbolic surfaces of constant negative curvature −1, are given for q = 3, 4, . . . as the quotient which we sometimes identify with the standard fundamental domain of G q , with sides pairwise identified by the generators in (3).
If π : H → M q denotes the natural projection map z → G q z, then the geodesic flow Φ t on H projects to the geodesic flow on M q which we denote by the same symbol Φ t . The geodesic γ = πγ is a closed geodesic on M q if and only if γ + and γ − are hyperbolic fixed points of a hyperbolic element g γ * ∈ G q . In [13] a Poincaré section Σ and a Poincaré map P : Σ → Σ have been constructed for the geodesic flow on the Hecke surfaces M q using λ q -CF expansions. For this the authors construct a mapP :Σ →Σ withΣ ⊂ ∂F q × S 1 where ∂F q denotes the boundary of the standard fundamental domain (3.1) and S 1 denotes the unit circle.
The induced map P : Σ → Σ on the projection Σ := π 1 Σ ⊂ SM q defines a Poincaré map for the geodesic flow on the Hecke surface M q .
To be more precise, let γ be a geodesic corresponding to an elementz ∈Σ such that its base points γ ± ∈ R have the regular and dual regular λ q -CF expansions ThenP(z) corresponds to the geodesic g (γ − , γ + ) = (g γ − , g γ + ) for g ∈ G q such that its base points have the regular and dual regular expansions corresponding to a k−fold shift of the symbol sequences determined by the entries in the λ q -CF 's of γ − and γ + . But these shifts also correspond to the action of the map f k q , that is From these relations we deduce that the periodic orbits of the mapP are determined by the periodic orbits of the map f q which, in turn, are determined by the points x ∈ I q with a periodic regular λ q -CFexpansion x = [[0; a 1 , . . . , a n ]]. The base points γ ± of the corresponding closed geodesic γ are given by . . , a n , a 1 ]] and γ + = [[0; a n , . . . , a 1 ]] .
Hence it follows from [13] that there is a one-to-one relation between the orbits of the points x ∈ Q with periodic regular λ q -CF expansion, i.e. the periodic orbits of the map f q : I q → I q , and the periodic orbits of the mapP. For the Poincaré map P : Σ → Σ this relation is also bijective except for the periodic orbits under f q of the two points ±r q , which correspond to the same periodic orbit of P : Σ → Σ as shown in [12, Theorem 2.5.1]. Because of (3.1) the period of a periodic orbit O of P is smaller than the one of the corresponding periodic orbit of f q if the λ q -CF expansion of a point x = [[0; a 1 , . . . , a n ]] in the periodic orbit of f q contains the block [(±1) k ] for some k > 0.
A prime periodic orbit is a periodic orbit which is not obtained by traversing a shorter orbit several times. Analogously a periodic point x = [[0; a 1 , . . . , a n ]] is said to be prime if n is the shortest period length of the sequence a 1 , . . . , a n . Consider now a prime periodic orbit γ = (γ − , γ + ) of the geodesic flow determined by the prime periodic point x = S γ − = [[0; a 1 , . . . , a n ]] ∈ I q . The period l(γ ) of γ is given by the well-known formula l(γ * ) = 2 ln λ where λ is the larger one among the two real positive eigenvalues of the (appropriately chosen) hyperbolic element g ∈ G q whose attracting fixed point is x ∈ I q .
But f n q (x * ) = x * and hence f n q defines a hyperbolic element g * ∈ G q . A straightforward calculation then shows that where r(x) = ln f q (x). SinceP k (z * ) =z * for some k ≤ n forz * ∈Σ corresponding to x * , the period l(γ * ) can also be written as Observe here, that r(z * ) = ln f q (x * ) is exactly the recurrence time function for the Poincaré mapP. The Ruelle zeta function ζ R (s) for the generating map f q of the λ q -CF expansion is defined as where we used the positivity of f q for real arguments. It is well-known that for s large enough ζ R (s) is a holomorphic function. A prime periodic orbit of period n contributes to all partition functions Z ln (s), l ∈ N.
denotes the contribution of a prime orbit O to the Ruelle zeta function then one finds by using the Taylor expansion for ln ( where r O = ln f n q (x) depends only on the orbit O and not on the specific point x ∈ O. Summing over all the prime orbits O of f q leads to the well-known formula [18] ζ Consider on the other hand the Ruelle zeta function for the mapP :Σ →Σ. We know that the prime periodic orbitsÕ of this map are in a 1 − 1 correspondence with those of the map f q : I q → I q . Let x = [[0; a 1 , . . . , a n ]] determine such a periodic orbit O. Then the corresponding periodic orbitÕ ofP is determined by the pointz O ∈Σ wherez O is the intersection ofΣ with the geodesic γ with base points (S x, −y), y = [[0; a n , . . . , Hence the contribution ZÕ(s) of the orbitÕ to the Ruelle zeta function ζ R (s) of the mapP coincides with the contribution Z O (s) of the orbit O to ζ R (s) of the map f q . Therefore the Ruelle zeta functions for the two maps are identical.
In [13] it was shown that there is a 1 − 1 correspondence between the prime periodic orbitsÕ of the mapP :Σ →Σ and the prime periodic orbits γ of the geodesic flow on S M q up to the two orbitsÕ ± determined by the endpoints S (±r q ), ∓r q . These two orbits coincide under the projection π q : SH → SM q . However, the contributions of both of these two orbits are contained in the Ruelle zeta function ζ R for the mapP respectively f q . The period of the orbit O + of the point r q under the map f q is given by κ q defined in (2.3). Define therefore the partition function Z Hence the Ruelle zeta function ζ P R (s) for the Poincaré map P : Σ → Σ of the geodesic flow Φ t : SM q → SM q has the form 3.2. The Selberg zeta function. The Selberg zeta function Z S (s) = Z Gq S (s) for the Hecke triangle group G q is defined as The inner product is taken over all prime periodic orbits γ of the geodesic flow on M q and l(γ ) denotes the period of γ (and hence the length of the corresponding closed geodesic). It is now clear that we can write Z S (s) as where the inner product is over all prime periodic orbitsÕ =Õ + of the Poincaré mapP :Σ →Σ. For s > 1 it can also be written in the form (r q ). Note that the zeros of the denominator all lie in the left half s-plane. We will show next how this function can be expressed in terms of the Fredholm determinant of the transfer operator for the map f q : I q → I q .
4. Ruelle's transfer operator for the Hurwitz-Nakada map f q : I q → I q .
For g : I q → C a function on the interval I q Ruelle's transfer operator L s for f q acts on it as follows: where r(y) = ln f q (y) and and Re (s) > 1 to ensure convergence of the series. To get a more explicit form for L s one has to determine the preimages y of any point x ∈ I q . For this recall the Markov partition I q = i∈Aκ q Φ i , A κq = {±1, . . . , ±κ q } in (2.6), determined by the intervals Φ i defined in (2.6) and the local inverses ϑ n of f q in (2.7). The Markov property of the map f q shows that the preimages of points in Φ o i can be characterized by the following lemma The same reasoning applies to the case q = 2h q + 3. Remark 1. We will define the operator L s on a space of piecewise continuous functions, hence it is enough to determine the preimages of points in the interior of the intervals Φ i . In general, points on the boundary of an interval Φ i can have more preimages than those in the interior.
We are now able to determine the sets N i,j explicitly. For this we denote by Z ≥n respectively by Z ≤−n for n = 1, 2, . . . the sets Z ≥n := {l ∈ N : l ≥ n} respectively Z ≤−n := {l ∈ Z : l ≤ −n}.
Lemma 4.2. For q = 2h q + 2 the sets N i,j are given as follows: For q = 3 the sets N i,j are given by For q = 2h q + 3 one has Proof. We will prove the case q = 2h q + 2 , the case of odd q is similar. If x]] ∈ Φ hq . All other sets N i,j are empty for 1 ≤ i ≤ h q and j ∈ A κq . That N −i,j = −N i,−j for all 1 ≤ i ≤ κ q , j ∈ A κq is obvious from Φ −i = −Φ i and the form of the maps ϑ n .
This Lemma allows us to derive explicit expressions for the transfer operator L s for the map f q : I q → I q . Using the index sets N i = j∈Aκ q N i,j we can rewrite the transfer operator L s in (4) as where χ Φi is the characteristic function of the set Φ i . If we now introduce vector valued functions g = (g) i∈Aκ q with g i := g| Φi then the operator L s can also be written as follows If g i is continuous on Φ i for all i ∈ A κq then (L s g) i is also continuous on Φ i since ϑ n (Φ i ) ⊂ Φ j for n ∈ N i,j . This implies that L s is well defined on the Banach space B = ⊕ i∈Aκ q C(Φ i ) of piecewise continuous functions on the intervals Φ i . To give explicit expressions for this operator on the space B denote for n ∈ N by L ∞ ±n,s the operator and by L ±n,s the operator Then we have Lemma 4.3. For q = 3 the operator L s is given by For q = 2h q + 3 one has (L s g) 1 = L 2,s g 2hq + L ∞ 3,s g 2hq+1 + L ∞ −2,s g −(2hq+1) + L −1,s g −2hq , (L s g) 2 = L ∞ 2,s g 2hq+1 + L ∞ −2,s g −(2hq+1) + L −1,s g −2hq , (L s g) i = L 1,s g (i−2) + L ∞ 2,s g 2hq+1 + L ∞ −2,s g −(2hq+1) + L −1,s g −2hq , 1 ≤ i ≤ 2h q + 1, respectively (L s g) −1 = L 1,s g 2hq + L ∞ 2,s g 2hq+1 + L ∞ −3,s g −(2hq+1) + L −2,s g −2hq , (L s g) −2 = L 1,s g 2hq + L ∞ 2,s g 2hq+1 + L ∞ −2,s g −(2hq+1) , (L s g) −i = L 1,s g 2hq + L ∞ 2,s g 2hq+1 + L ∞ −2,s g −(2hq+1) + L −1,s g 2−i , 1 ≤ i ≤ 2h q + 1. Unfortunately, on the space of piecewise continuous functions the operator L s is not of trace class. In fact, it not even compact.
Much better spectral properties however can be achieved by defining L s on a Banach space B = ⊕ i∈Aκ q B(D i ) with B(D i ) the Banach space of holomorphic functions on certain discs D i ⊂ C with Φ i ⊂ D i , i ∈ A κq , continuous on D i together with the sup norm. This is possible, since all the maps ϑ ±m , m ≥ 1 have holomorphic extensions to complex neighborhoods of I q with the following properties: Here D i denotes the closure of the set D i . For the proof of the Lemma it suffices to show the existence of open intervals I i ⊂ R, i ∈ A κq , which have the properties • Φ i ⊂ I i and • ϑ n (I i ) ⊂ I j for all n ∈ N i,j .
Since the maps ϑ n are conformal it is clear that the discs D i with center on the real axis and intersection equal to the open intervals I i satisfy Lemma (4.4).
Using (2.7) the two conditions on I i can also be written as ST n I i ⊂ I j and Φ i ⊂ I i for all n ∈ N i,j and all i, j ∈ A κq In the cases q = 3 and q = 4 we give explicit intervals fulfilling conditions (4). For the case q ≥ 5 we first show the existence of intervals I i satisfying the weaker condition ST n I i ⊂ I j and Φ i ⊂ I i for all n ∈ N i,j .
The existence of intervals I i satisfying (4) then follows by a simple perturbation argument. Proof. The above intervals I i , i = ±1 obviously satisfy Proof of Lemma 4.4 for q = 3, 4. The maps ϑ n are all conformal, which means that they preserve angles and generalized circles. It is now easy to see that the discs D ±1 which have diameters along the intervals I ±1 satisfy the conditions of the lemma for q = 3 and q = 4.

Next we can prove
Lemma 4.7. For q = 2h q + 2, h q ≥ 2 define the intervals I i respectively I −i for For odd q ≥ 5 we need Lemma 4.8. For q = 2h q + 3, h q ≥ 1 one has Proof. By (2.1), (2.2) and (2.3) we have Using the identities in (4.8) shows (4.8).
Proof. Since the proof of this Lemma proceeds along the same lines as in the case of even q, we restrict ourselves to the case where ϑ n (I i ) ⊂ I j for n ∈ N i,j . This happens only for the pairs (i, j) = (±k, To prove finally Lemma 4.4 one has to enlarge the intervals I i a little bit so that ϑ n (I i ) ⊂ I j for all n ∈ N i,j . In the case q = 2h q + 2, h q ≥ 2 one can take the intervals 4 ) with n i > n i−1 for 2 ≤ i ≤ h q and n 1 large enough. In the case q = 2h q + 3, h q ≥ 1 one can choose the intervals with n 2i+1 > n 2i > n 2i−1 > n 2i−2 for all 1 ≤ i ≤ h q and n 1 large enough. The existence of the discs D i , i ∈ A κq , of Lemma 4.4 shows that the operator L s is well defined on the Banach space B = ⊕ i∈Aκ q B(D i ) with B(D i ) the Banach space of holomorphic functions on the disc D i with Φ i ⊂ D i , i ∈ A κq , with the sup norm. In fact we have the following theorem.
Theorem 4.10. The operator L s : B → B is nuclear of order zero for s > 1 2 and extends to a meromorphic family of nuclear operators of order zero with poles only at the points s k = 1−k 2 , k = 0, 1, 2 . . ..

Proof.
It is now easy to verify that the operator L s can be written as a 2κ q × 2κ q matrix operator which for even q has the form

DIETER MAYER, TOBIAS MÜHLENBRUCH AND FREDRIK STRÖMBERG
For odd q it has the form As for the transfer operator of the Gauss map (cf. [10]) one shows that the operators L ∞ l,s , l = ±1, ±2, ±3 define meromorphic families of nuclear operators L ∞ l,s : B(D i ) → B(D j ) in the Banach spaces of holomorphic functions on the discs for which N i,j = Z ≥l respectively N i,j = Z ≤−l for l = 1, 2, 3. These operators have poles at the points s = s k = 1−k 2 for k = 0, 1, · · · . On the other hand the operators L l,s , l = ±1, ±2 with L l,s : B(D i ) → B(D j ) are holomorphic nuclear operators in the entire s-plane in the corresponding Banach spaces of holomorphic functions on the discs for which N i,j = {±l}, l = 1, 2. Hence the operator L s has these properties in the Banach space B = ⊕ i∈Aκ q B(D i ).

5.
The reduced transfer operators L s, , = ±1 and functional equations for their eigenfunctions.
5.1. The symmetry operator P : B → B. From the above matrix representation of the transfer operator L s it can be seen that this operator has a certain symmetry which we will discuss next. For this purpose, define the operator P : B → B as This operator is well-defined, since D −i = −D i for all i ∈ A κq and P 2 = id B . That P is indeed a symmetry follows from the following lemma. Proof. Let s > 1 2 and suppose that f ∈ B. To extend ϑ n to the complex discs D i . we use the convention (n + z) 2s := ((n + z) 2 ) s It is then easy to see that if and only if (L s ) −i,−j = L ∞ −l,s . Combining these two observations, the fact that L s P f (z) = P L s f (z) follows immediately. Since both the operators P L s and L s P are meromorphic in the entire s-plane with only poles at the points s = s k , k = 0, 1, . . . they coincide there.
This allows us to restrict the operator L s to the eigenspaces of the operator P which is an involution and therefore has the eigenvalues ±1. Denote these eigenspaces by B ± .
Let B κq denote the Banach space B κq = ⊕ 1≤i≤κq B(D i ) with the discs as defined earlier in Lemma 4.4. Then the transfer operator L s restricted to the spaces B ± induces the following operators L s,± in the Banach space B κq . For q = 2h q + 2, κ q = h q and − → g = (g i ) 1≤i≤κq we get: For q = 3 on the other hand we get respectively for q = 2h q + 3 > 5, κ q = 2h q + 1

Functional equations.
It is well-known that for modular groups, i.e. finite index subgroups of G 3 = P SL(2, Z), the eigenfunctions of the transfer operator L s with eigenvalue ρ = 1 fulfill simple finite term functional equations [7], so called Lewis equations, which are closely related to the period functions of Lewis and Zagier [9] for these groups. In the present case we can also derive such functional equations, but it is not clear how their holomorphic solutions are related to the period functions of the Hecke triangle groups G q for arbitrary q. In the case q = 3 it was shown in [2] that the solutions of the functional equation derived from our transfer operator L s for s = 1 2 are indeed in one to one correspondence with the Maass cusp forms for G 3 . Since the spectrum of the operator L s is the union of the spectra of the two operators L s, , = ±1, we use these two operators to derive the corresponding functional equations. In the case q = 3 their eigenfunctions − → g = (g 1 ) with eigenvalue ρ = 1 obey the equation where we used the so-called slash-action defined by For q = 2h q + 3, h q ≥ 1 one finds that g 1 = g 2hq |ST 2 + g 2hq+1 |N 3 + g 2hq |ST −1 + g 2hq+1 |N −2 respectively g 2 = g 2hq+1 |N 2 + g 2hq |ST −1 + g 2hq+1 |N −2 and hence Induction on i shows furthermore Therefore g 2hq = g 2 |P hq−1 (ST ) and g 2hq+1 = g 1 |(ST ) hq + g 2 |P hq−1 (ST ).
Inserting this into equation (13) shows that This allows to express both g 2hq and g 2hq+1 in terms of g 1 : Inserting these expressions into equation (13) gives finally the following functional equation for g 1 : Since |N 3 (1 − T ) = |ST 3 and |N −2 = | −ST −1 the "Lewis" equation for the Hecke triangle group G q , q = 2h q + 3 has the form: For q = 5 corresponding to h q = 1 this reads 6. The Selberg zeta function for Hecke triangle groups G q . We want to express the Selberg zeta function for the Hecke triangle groups G q in terms of Fredholm determinants of the transfer operator L s for the map f q . Our construction is analogous to that for modular groups and the Gauss map [3]. We start with a discussion of the Ruelle zeta function for the H-N map f q .
6.1. The Ruelle zeta function and the transfer operator for the Hurwitz-Nakada map f q . We have seen that the transfer operator L s for the map f q : I q → I q can be written as where where ϑ n1,...,n k denotes the map ϑ n1 • . . . • ϑ n k . We have seen that the set of k-tuples (n 1 , . . . , n k ) ∈ Z k with the property that [[0; n 1 , . . . , n k , x]] ∈ A q depends only on the interval I i for x ∈ I • i , the interior of the interval ).
If f j , j ∈ A κq denotes the restriction f |I j and n k = (n 1 , . . . , n k ) ∈ Z k we get On the Banach space B = ⊕ i∈Aκ q B(D i ) we get The trace of this operator on this Banach space is then given by the well-known formula for such composition operators [11] trace L k s = where z n k = [[0; n 1 , . . . , n k ]] is the unique fixed point of the map ϑ n1,...,n k : D i → D i which defines a hyperbolic element in the group G q . These points are however in one-to-one correspondence with the periodic points of period k of the map f q : I q → I q . Hence also the following identity holds Therefore we get Proof. For s large enough we have, comparing equations (3.1) and (15) Z n (s) = trace L n s − trace L n s+1 and therefore ζ R (s) = det(1−Ls+1) det(1−Ls) Z n (s). Since the operators L s are meromorphic and nuclear in the entire s-plane their Fredholm determinants also allow such a meromorphic continuation, which proves the proposition.
6.2. The transfer operator K s . As discussed earlier, there is a one to one correspondence between the closed orbits of the map f q : I q → I q and the closed orbits of the geodesic flow on the Hecke surfaces apart from the closed orbits of the two points r q = [[0; 1 hq−1 , 2]] for even q , respectively r q = [[0; 1 hq , 2, 1 hq−1 , 2]] for odd q, and −r q , which are not equivalent under the map f q , but are equivalent under the group G q and hence correspond to the same closed orbit of the geodesic flow. In the Ruelle zeta function ζ R (s) for the map f q the contributions of both the orbits of r q and −r q are included. To relate this function to the Selberg zeta function Z S (s) for the geodesic flow on the Hecke surfaces we have to subtract the contribution of one of these two orbits of f q , say of r q , to this function. This we can achieve by subtracting the contribution of the orbit O + of the point r q from the partition functions Z lκq (s) for the map f q for all l = 1, 2, . . .. Consider therefore the corresponding Ruelle function ζ O R (s) = exp( If n = lκ q we find that Z O+ lκq (s) = κ q exp(−slr O+ ) and hence ζ O+ R (s) = For q = 2h q + 3 on the other hand define Then one has Proof. Since the proof for odd q is completely analogous we restrict ourselves to the case q = 2h q + 2. Induction on i shows that for 1 ≤ j ≤ h q one has for − → g = (g) 1≤j≤hq But this shows that This proves the Lemma.
where z is the attractive fixed point of  (17) and (18) Proof. We restrict ourselves again to the case q = 2h q +2. The case of odd q is anal- L 2,s ) for q = 2h q + 3. We then arrive at the following theorem.
Remark 3. Using the explicit form of the maps which fix r q , cf. e.g [13] Remark 27 (where the upper right entry of the matrix for even q should read λ − λ 3 ) one can prove that the spectrum of the operator K s can also be written as µ n = l 2s+2n , n = 0, 1, . . . where l = 4 − λ 2 q Rλ q + 2 = 2 − λ q 2 + λ q for even q and l = 2 − λ q Rλ q + 2 = 2 − λ q 2 + Rλ q for odd q.
The Selberg zeta function Z S (s) for Hecke triangle groups G q and small q has been calculated numerically using the transfer operator L s by one of us in [21].
Besides the case q = 3, that is the modular group G 3 = SL(2, Z) [2], we do not yet know how the eigenfunctions of the transfer operator L s with eigenvalue ρ = 1 are related to the automorphic functions for a general Hecke group G q . The divisor of Z S (s) is closely related to the automorphic forms on G q (see for instance [6],p. 498). One would therefore expect that there exist explicit relationships also for q > 3 similar to those obtained for modular groups between eigenfunctions of the transfer operator L s with eigenvalue one and automorphic forms related to the divisors of Z S at these s-values.
Another interesting problem would be to understand the behavior of our transfer operator L s in the limit when q tends to ∞. In this limit the Hecke triangle group G q tend to the theta group Γ θ , generated by Sz = −1 z and T z = z + 2. This group is conjugate to the Hecke congruence subgroup Γ 0 (2), for which we have constructed a transfer operator in [7], [5]. One should understand how these two different transfer operators are related to each other. The limit q → ∞ is quite singular, since the group Γ θ has two cusps whereas all the Hecke triangle groups have only one cusp. Therefore one expects in this limit all the singular behavior Selberg predicted already in [20]. Understanding the limit q → ∞ could also shed new light on the Phillips-Sarnak conjecture [15] on the existence of Maass wave forms for general non-arithmetic Fuchsian groups.