Twisted Ruelle zeta function at zero for compact hyperbolic surfaces

Let $X$ be a compact, hyperbolic surface of genus $g\geq 2$. In this paper, we prove that the twisted Selberg and Ruelle zeta functions, associated with an arbitrary, finite-dimensional, complex representation $\chi$ of $\pi_1(X)$ admit a meromorphic continuation to $\mathbb{C}$. Moreover, we study the behaviour of the twisted Ruelle zeta function at $s=0$ and prove that at this point, it has a zero of order $\dim(\chi)(2g-2)$.


Introduction
The zeta functions of Selberg and Ruelle were first introduced in [28,29]. For a hyperbolic surface, they are defined by an Euler-type product over the prime closed geodesics, i.e., the closed geodesics which are not multiples of shorter geodesics, and in terms of the lengths of these geodesics. Twisting these Euler products by a representation of the fundamental group of the surface defines twisted versions of the Selberg and Ruelle zeta functions. The goal of this paper is to prove that these twisted zeta functions, associated with a non-unitary representation of the fundamental group, admit a meromorphic continuation to C. In addition, we study the behaviour of the twisted Ruelle zeta function near the origin and prove that it is related to a topological invariant, the Euler characteristic of the surface.
Let Γ be a discrete, torsion-free, cocompact subgroup of PSL 2 (R) and let X = Γ\H 2 be the associated compact hyperbolic surface with fundamental group π 1 (X) ∼ = Γ. We define the twisted dynamical zeta functions, associated with a finite-dimensional, complex representation χ : Γ → GL(V χ ) of Γ, which is not necessary unitary. For s ∈ C, the twisted Selberg zeta function, associated with χ, is given by the infinite product unitary representation in terms of Betti numbers (see also the recent work [7] for perturbations of the hyperbolic metric on 3-manifolds). In the odd-dimensional case it turns out that the leading coefficient is in fact related to a topological invariant, the Reidemeister torsion ( [11,15,27]). This led Fried to conjecture a precise relation between the value R(0; χ) and the Reidemeister torsion for all compact locally symmetric spaces and acyclic representations χ. For unitary representations χ, Fried's conjecture was recently proven by Shen [30] (see also [5,6,8,9,23,24,26,31,32,35,36,37,40] for other contributions to Fried's conjecture).
In fact, also for quotients X = Γ\H 2 of the upper half plane, Fried obtained in [17] a relation between the value of a Ruelle zeta function at s = 0 and a topological invariant. He considered a general Fuchsian group Γ, allowing also elliptic elements. In such a case, X is a compact orbisurface and the unit sphere bundle S(X) = Γ\ PSL 2 (R) is a Seifert fiber space over X. Fried considered the twisted Ruelle zeta function R(s; ρ), associated with a unitary representation ρ of π 1 (S(X)). Note that π 1 (X) ≃ π 1 (S(X))/Z. By [17,Theorem 3], for an acyclic, unitary representation ρ of π 1 (S(X)), the value |R(0; ρ)| −1 equals the Reidemeister torsion of S(X) in the representation ρ. It is natural to ask whether one can extend these results to the case of a non-unitary representation of π 1 (S(X)). We deal with this problem the subsequent work [3].
Outline of the proof. The main tool to prove the above results is a twisted Selberg trace formula for non-unitary representations of Γ (see Theorem 3.2.4). This trace formula holds for the heat operators of a twisted Laplace operator ∆ ♯ χ on the flat vector bundle E χ over X, associated with χ. This twisted Laplacian is not self-adjoint in general, but it is elliptic with self-adjoint principle symbol and hence it has similar spectral properties (for further details see Subsections 3.1 and 3.2). A slightly weaker version of this trace formula, namely for Paley-Wiener functions applied to (∆ ♯ χ ) 1/2 instead of heat operators, was previously obtained by Müller [25,Theorem 1.1] in the more general context of rank one locally symmetric spaces. Our trace formula can, after some additional effort estimating the heat kernel, be used to obtain a resolvent trace formula (see Proposition 4.2.4) which is the key to prove Theorem A. We note that the trace formula and the functional equation for the twisted Selberg zeta function that we obtain generalize the corresponding results in Hejhal [20,Chapter III,Theorem 4.10].
Finally, Corollary C on the behavior of the twisted Ruelle zeta function follows from a detailed analysis of the functional equation for the twisted Selberg zeta function. Since the functional equation is the same as in the case of a unitary representation of Γ, the same computations as in [17] apply.
Organization of the paper. In Section 2, we review well-known theory of the geometry of hyperbolic surfaces, the Laplace-Beltrami operator, and the principal series representation of PSL 2 (R). In Section 3, we introduce the twisted Bochner-Laplace operator acting on the space of sections of a vector bundle over X and recall its spectral properties. In the same section, we derive the resolvent trace formula for the heat operator induced by the twisted Laplacian. Section 4 is the core of this article. In this section, we define the Selberg and Ruelle zeta function associated with a non-unitary representation of the fundamental group of X and prove that they admit a meromorphic continuation to C. Moreover, we prove functional equations for the twisted Selberg function, which further lead to the result for the behavior of the twisted Ruelle zeta function at the origin.

Preliminaries
In this section we set up some notation around the geometry and analysis on hyperbolic surfaces and representation theory of PSL(2, R).

Hyperbolic surfaces
We consider the upper half plane The group G = PSL(2, R) acts on H 2 by fractional linear transformations. This action is transitive and the maximal compact subgroup K = PSO(2) is the stabilizer of i ∈ H 2 , hence H 2 ∼ = G/K = PSL(2, R)/ PSO(2). Let g and k denote the Lie algebras of G and K. The G-invariant metric, induced by the restriction of the Killing form on g to the Cartan complement p of k, is the Poincaré metric Let G = N AK be the Iwasawa decomposition of G with where matrices are identified with their images in the quotient group PSL(2, R) = SL(2, R)/{±I}. Let A + ⊆ A be the subset of all elements of the above form with y > 1. We write n and a for the Lie algebras of N and A. Now, let Γ be a Fuchsian group, that is a discrete subgroup of G. We assume that Γ is cocompact, i.e., Γ\G is compact, and that Γ is torsion-free, i.e., there are no non-trivial elements of finite order. Then, X = Γ\H 2 = Γ\G/K is an orientable compact connected hyperbolic surface, and conversely every orientable compact connected hyperbolic surface arises in this way.
In this setting, every γ ∈ Γ with γ = e, where e is the identity element, is hyperbolic and hence conjugate to a unique element a γ ∈ A + . It follows that the non-trivial conjugacy classes [γ] in Γ parameterize the set of closed geodesics on X. Here a geodesic is called closed if it returns to its starting point with the same direction. The length l(γ) of the closed geodesic corresponding to a nontrivial conjugacy class [γ] is related to the element a γ by Ad(a γ )| n = e l(γ) Id n .
A non-trivial conjugacy class [γ] in Γ is called primitive if γ cannot be written as γ = γ n 0 , with γ 0 ∈ Γ and n ≥ 2. Then [γ] is primitive if and only if the corresponding geodesic is prime, i.e., it cannot be written as the multiple of a shorter closed geodesic. For every non-trivial conjugacy class [γ] we can write γ = γ n 0 with [γ 0 ] primitive. The number n = n Γ (γ) is unique, independent of the representative γ and satisfies l(γ) = n · l(γ 0 ).

The Laplace-Beltrami operator
denote the Laplace-Beltrami operator on H 2 , where we write z = x + iy ∈ H 2 . If we define the left translation of a smooth function f on H 2 by g ∈ G as Hence, ∆ descends to a differential operator ∆ on Γ\H 2 .

Unitary principal series representations
Let P = AN denote the standard parabolic subgroup of G. We identify a * C ≃ C by λ → λ(diag( 1 2 , − 1 2 )). Then ρ = 1 2 tr ad | n corresponds to 1 2 . For λ ∈ R we form the character e iλ of A given by e iλ (e H ) = e iλ(H) (H ∈ a), and extend it to a character e iλ ⊗ 1 of P = AN by letting N act trivially. We form the induced representation π λ = Ind G P (e iλ ⊗ 1) of G on the Hilbert space H λ of L 2 -sections of the homogeneous vector bundle G × P (e iλ+ρ ⊗ 1) → G/P . The representations π λ are unitary and irreducible for all λ ∈ R and are called the unitary principal series. We write Θ λ for the distribution character of π λ defined by where dg denotes a suitably normalized Haar measure on G. We denote by H K λ the one-dimensional subspace of H λ consisting of K-fixed vectors.

Harmonic analysis on hyperbolic surfaces
Following Müller [25], we describe the construction of the twisted Bochner-Laplace operator on vector bundles over X = Γ\H 2 induced from non-unitary representations χ of Γ and use it to obtain a trace formula for X.

The twisted Bochner-Laplace operator
In this section, we define the twisted Bochner-Laplace operator as introduced in [25,Section 4]. This operator acts on sections of a vector bundle over X. It is an elliptic operator which is in general not self-adjoint. Nevertheless, it has a self-adjoint principle symbol (with respect to the choice of a metric) and hence it has qualitatively similar spectral properties.
Let χ be a finite-dimensional, complex representation equipped with a flat connection ∇ Eχ . We recall the construction of the twisted Bochner-Laplace operator ∆ ♯ χ , acting on smooth sections of E χ . The second covariant derivative (∇ Eχ ) 2 is defined by , T X is the tangent bundle of X, and ∇ LC denotes the Levi-Civita connection on T X. The twisted Bochner-Laplace operator ∆ ♯ χ is defined to be the corresponding connection Laplacian on E χ , i.e. the negative of the trace of the second covariant derivative: Remark 3.1.1. The definition of the twisted Bochner-Laplace operator in the present paper is the same as in [25,Section 4] for τ being the trivial representation.
Locally, this operator is described as follows.
We consider an open subset The local expression above is independent of the choice of the basis of flat sections of E χ | U , since the transition maps comparing flat sections are constant. By (3.1) and (3.2), the twisted Bochner-Laplace operator acting on C ∞ (U, E χ | U ) is given by where ∆ denotes the Laplace-Beltrami operator on X.
Let now E χ be the pullback to X = H 2 of E χ . Then, With respect to the isomorphism (3.4), it follows from (3.3) that the lift ∆ ♯ χ of ∆ ♯ χ to E χ takes the form where ∆ is the Laplace-Beltrami operator on H 2 . If we choose a Hermitian metric on E χ , then ∆ ♯ χ acts in L 2 (X, E χ ) with domain C ∞ (X, E χ ). However, it is not a formally self-adjoint operator in general.
Hence, ∆ ♯ χ is an elliptic, second order differential operator with the following spectral properties: its spectrum is discrete and contained in a translate of a positive cone C ⊂ C such that R + ⊂ C. This fact follows from classical spectral theory of elliptic operators, under the assumption of the compactness of the manifold. We refer the reader to [33], and also [25,Lemma 2.1]. Moreover, the direct sum of all generalized eigenspaces is dense in L 2 (X, E χ ).

The trace formula
For the derivation of the trace formula, we follow the classical work of Wallach [38], and its extension to non-unitary representations of Γ by Müller [25]. This trace formula will be the basic tool to prove the meromorphic continuation of the twisted dynamical zeta functions in the next section.
We denote by spec(∆ ♯ χ ) ⊆ C the (discrete) spectrum of ∆ ♯ χ . For µ ∈ spec(∆ ♯ χ ), we write L 2 (X, E χ ) µ for the corresponding generalized eigenspace. We define the algebraic multiplicity m(µ) of µ as m(µ) We want to utilize the heat operator e −t∆ ♯ χ , induced by ∆ ♯ χ , as an integral, trace class operator and derive a corresponding trace formula. In [25], a Selberg trace formula for non-unitary twists is derived for particular test functions φ, the Paley-Wiener functions on C (see [25, p. 2079]). Since the exponential function φ(λ) = e −tλ 2 , λ ∈ C does not belong to this space, we use the extended results obtained in [35, p. 171-173]. With this, we conclude that the heat operator e −t∆ ♯ χ is an integral operator with smooth kernel, i.e., there exists a smooth section which is the spectral side of our (pre)-trace formula. Let H t be the kernel of the heat operator e −t ∆ , that is the operator induced by the self-adjoint Laplacian ∆ acting in L 2 ( X Note that every γ ∈ Γ, γ = e, is hyperbolic, so the summation is over hyperbolic conjugacy classes of Γ. We therefore refer to the first term in the RHS of (3.9) as the identity contribution and to the second term as the hyperbolic contribution.

Remark 3.2.2.
The trace formula in [39] is obtained for kernel functions that are compactly supported which is not true for the heat kernel. However, the heat kernel is an admissible function in the sense of Gangolli [18, p. 407]. This is because the heat kernel belongs to the Harish-Chandra L 1 -Schwartz space C 1 (G) (see [18, Proposition 3.1 and p. 411]).
Let us first simplify the hyperbolic contribution. The character Θ λ of π λ can be evaluated on the K-biinvariant function H t in terms of the spherical Fourier transform H t (λ) of H t : where v ∈ H K λ , with v = 1 and φ λ (g) = π λ (g)v, v denotes the associated spherical function (see e.g. [21, Chapter IV] for details).  Proof. By [2, eq. (2.11)], where Ω is the Casimir element in the center of the universal enveloping algebra of sl(2, C). By [22,Proposition 8.22], Recall now from Subsection 2.3 that ρ corresponds to 1 2 . We now turn to the identity contribution in (3.9). The function H t on G belongs to the Harish-Chandra L q -Schwartz space. Hence, the Fourier inversion formula [19,Theorem 3] can be applied to H t (see also Theorem 1 in [1] for K-biinvariant test functions in the Harish-Chandra L q -Schwartz space for 0 < q ≤ 2). By [21, Theorem 7.5 (i)] (see also [21, eq. (28) and (29), p. 42]), we have with the c-function c(λ) given by (note the change of variables λ vs. λ 2 compared to [21]) (3.12) |c(λ)| −2 = λπ tanh λπ, λ ∈ R.

Twisted dynamical zeta functions on compact hyperbolic surfaces
In this section we prove meromorphic continuation of the twisted Selberg and the Ruelle zeta function, obtain their functional equations and study the behaviour of the twisted Ruelle zeta function at s = 0.

Definition and convergence
Let χ : Γ → GL(V χ ) be a finite-dimensional, complex representation of Γ. For s ∈ C, we define the twisted Selberg zeta function, associated with χ, by the infinite product It is proven in [14, Theorem 3.1] that the product in (4.1) converges absolutely for s in some right half plane in C. To keep the paper self-contained, and also because we need parts of the computation later on, we include a proof of this result: Proof. Let · be the operator norm associated to a fixed norm on V χ . By [40, p. 10], there exists a c ≥ 0 such that Hence, for Re(s) ≫ 0, by (4.1), we get tr(χ(γ)) e −sl(γ) 1 − e −l(γ) .   Proof. (4.6) follows easily by differentiating (4.3).
where c is as in (4.2). We remark also that one can consider the notion of the critical exponent of χ, in order to eliminate the dependence on the norm · on V χ in the estimates (4.2). The critical exponent c χ is defined as c χ := inf{c ≥ 0, such that (4.2) holds}. Note that for χ being unitary, we have that c χ = 0. For more details we refer the reader to [40, p. 10].

Meromorphic continuation
The trace formula (3.13) motivates to consider a shift of the operator ∆ ♯ χ by 1 4 : We now explicitly calculate the hyperbolic contribution on the RHS of (3.13).
Proof. Since Ad(a γ )| n = e l(γ) Id n , we have Moreover, the integral in the hyperbolic contribution in the RHS of (3.13) is just the Fourier transform of the function λ → e −tλ 2 . Hence, substituting (4.11) in (3.13), we get (4.10).

of trace class, and so is
We observe that for Re(s j ) ≫ 0 Hence, Lemma 4.2.2 (see [4,Chapter 2]). There exist coefficients c j such that tr e −tA ♯ χ has an asymptotic expansion Proof. As t → ∞, the integrand in the RHS of (4.13), decays exponentially. As t → 0, we use Lemma 4.2.2 and the Taylor series expansion of e −t(s1−1/2) 2 and e −t(s2−1/2) 2 to conclude that the integral converges absolutely. Hence, we can interchange summation and integration and (4.13) follows.

Proposition 4.2.5.
The logarithmic derivative L(s; χ) of the Selberg zeta function Z(s; χ) extends to a meromorphic function in s ∈ C with singularities given by the following formal expression: where (λ j = 1 4 + µ 2 j ) j∈Z + ⊆ C are the eigenvalues of ∆ ♯ χ counted with algebraic multiplicity.
The LHS of (4.14), that is the spectral side of the resolvent trace formula, is meromorphic since the resolvent (A ♯ χ + (s − 1/2) 2 ) −1 is meromorphic, and the simple poles where −(s − 1/2) 2 equals an eigenvalue of A ♯ χ give exactly the singularities of the first term in (4.18).
In order to establish meromorphicity of I(s, s 2 ; χ) and obtain the singularities of the second term in (4.18), we recall the expression (4.16) and use the following identity (see e.g. [22, p. 401]): Then, we get Since one can interchange summation and integration on the RHS of the equation above, we obtain the following expression To compute the integral, note that for a 2 = b 2 we can write and use the following integral formula, which is easily derived using the residue theorem: for Re(a) > 0.
The result now follows by substituting k = |n|−1 2 ∈ Z + . Theorem 4.2.6. Let X = Γ\H 2 be a compact hyperbolic surface and let χ : Γ → GL(V χ ) be a finite-dimensional, complex representation of Γ. Then, the twisted Selberg zeta function Z(s; χ) admits a holomorphic continuation to C with zeros given by the following formal product: Proof. The theorem follows from Proposition 4.2.5 by integration and exponentiation. We only need to prove that the residues of L(s; χ) are positive integers. This follows from (4.18) and the Gauss-Bonnet formula where g ≥ 2 is the genus of the surface.

Remark 4.2.7.
Since the twisted Bochner-Laplace operator ∆ ♯ χ is in general not self-adjoint, the eigenvalues λ j = 1 4 + µ 2 j need not be real and > 0 and hence, some of the zeros 1 2 ± iµ j and −k of Z(s; χ) could coincide. Therefore, it is not possible in general to deduce more precise information about the degree of the zeros of Z(s; χ) from Theorem 4.2.6. Corollary 4.2.8. The twisted Ruelle zeta function R(s; χ) admits a meromorphic continuation to C with zeros and singularities given by the following formal product: In particular, R(s; χ) has a zero of order (2g − 2) dim(V χ ) at s = 0.
Proof. The statement follows from Theorem 4.2.6 and Lemma 4.1.5.  where the integral is a complex line integral along any curve from 0 to s − 1 2 . We remark that the value of the integral in (4.21) depends on the chosen curve from 0 to s − 1 2 , but the RHS of (4.21) does not, because the residues of r tan πr are integer multiples of 1 2π and dim(V χ ) Vol(X) = 2π(2g − 2) dim(V χ ). Proof. Write again s = s 1 and consider the transform s → 1 − s in (4.14). Then, the LHS of (4.14) remains invariant under this transform. The RHS of (4.14) is as in (4.15), where I(s, s 2 ; χ) is now given by (4.20) and H(s, s 2 ; χ) is given by (4.17). (Note that we cannot use the identity (4.16) for I(s, s 2 ; χ) here since it is only valid for Re(s − 1 2 ) ≫ 0 and hence only holds for s and not for 1 − s.) Thus, if we consider the transform s → 1 − s, then the RHS of (4.14) will give Using the classical identity this can be rewritten as We integrate both sides of (4.24) over s and exponentiate the result to get (4.21). Moreover, by [17, p.153], the latter integral in (4.28) is of order 1 2 log 1 2 + O(ǫ). Hence, as ǫ → 0 (4. 29) exp(A(ǫ)) ∼ 2πǫ.
In total, we find that exp −2π