Transverse Dynamics of Two-Dimensional Traveling Periodic Gravity–Capillary Water Waves

Abstract

We study the transverse dynamics of two-dimensional traveling periodic waves for the gravity–capillary water-wave problem. The governing equations are the Euler equations for the irrotational flow of an inviscid fluid layer with free surface under the forces of gravity and surface tension. We focus on two open sets of dimensionless parameters $(\alpha ,\beta )$, where $\alpha$ and $\beta$ are the inverse square of the Froude number and the Weber number, respectively. For each arbitrary but fixed pair $(\alpha ,\beta )$ in one of these sets, two-dimensional traveling periodic waves bifurcate from the trivial constant flow. In one open set we find a one-parameter family of periodic waves, whereas in the other open set we find two geometrically distinct one-parameter families of periodic waves. Starting from a transverse spatial dynamics formulation of the governing equations, we investigate the transverse linear instability of these periodic waves and the induced dimension-breaking bifurcation. The two results share a common analysis of the purely imaginary spectrum of the linearization at a periodic wave. We apply a simple general criterion for the transverse linear instability problem and a Lyapunov center theorem for the dimension-breaking bifurcation. For parameters $(\alpha ,\beta )$ in the open set where there is only one family of periodic waves, we prove that these waves are linearly transversely unstable. For the other open set, we show that the waves with larger wavenumber are transversely linearly unstable. We also identify an open subset of parameters for which both families of periodic waves are transversely linearly unstable. For each of these transversely linearly unstable periodic waves, a dimension-breaking bifurcation occurs in which three-dimensional doubly periodic waves bifurcate from the two-dimensional periodic wave.

1 Introduction

We consider a three-dimensional inviscid fluid with constant density $\rho$ occupying a region

$$\begin{aligned}D_{\eta }=\{(X, Y, z) \in \mathbb {R}^3 : \, 0< Y < h+\eta (X, z, t)\},\end{aligned}$$

where (X, Y, z) are Cartesian coordinates, h is the mean fluid depth, and $\eta > -h$ is the unknown free surface of the fluid depending on the horizontal spatial variables X, z and the time variable t. The fluid is under the influence of the gravitational force with acceleration constant g and surface tension with coefficient T. We assume that the flow is irrotational and denote by $\phi$ an Eulerian velocity potential. Choosing a coordinate frame moving from left to right along the X-axis with constant velocity $c>0$, the fluid motion is described by Laplace’s equation

$$\begin{aligned} \phi _{XX}+\phi _{YY}+\phi _{zz} = 0 \quad \text {for} \quad 0< Y < 1+\eta , \end{aligned}$$

(1.1)

with boundary conditions

$$\begin{aligned} \begin{aligned}&\phi _Y = 0&\quad&\text {on}\, Y=0,&\\&\phi _Y = \eta _t - \eta _X + \eta _X \phi _X + \eta _z \phi _z&\quad&\text {on}\, Y=1+\eta ,&\\&\phi _t - \phi _X + \frac{1}{2}\left( \phi _X^2+\phi _Y^2+\phi _z^2\right) + \alpha \eta - \beta {\mathcal {K}} = 0{} & {} \text {on}\, Y=1+\eta .&\end{aligned} \end{aligned}$$

(1.2)

Here, we have used dimensionless variables by taking the characteristic length scale h and characteristic time scale h/c. The dimensionless parameters

$$\begin{aligned}\quad \alpha = \frac{gh}{c^2} \quad \text {and} \quad \beta = \frac{T}{\rho hc^2}\end{aligned}$$

are the inverse square of the Froude number and the Weber number, respectively, and the quantity ${\mathcal {K}}$ is twice the mean curvature of the free surface $\eta$, given by

$$\begin{aligned}{\mathcal {K}} = {\left[ \frac{\eta _X}{\sqrt{1+\eta _X^2+\eta _z^2}}\right] }_X + \left[ \frac{\eta _z}{\sqrt{1+\eta _X^2+\eta _z^2}}\right] _z.\end{aligned}$$

The set of Eqs. (1.1)–(1.2) are the Euler equations for gravity–capillary waves on water of finite depth. The case $\beta =0$, that we do not consider in this work, corresponds to gravity water waves.

We are interested in the transverse dynamics of two-dimensional traveling periodic water waves. In the above formulation these are steady solutions which are periodic in X and do not depend on the second horizontal coordinate z and on the time t. Their existence is well-recorded in the literature; e.g., see [14, 19, 20, 30] and the references therein. Many of these results are obtained using methods from bifurcation theory. Bifurcations of two-dimensional periodic waves are determined by the positive roots of the linear dispersion relation

$$\begin{aligned} {\mathcal {D}}(k) {:}{=} (\alpha + \beta k^2) \sinh |k| - |k|\cosh (k)=0, \end{aligned}$$

(1.3)

obtained by looking for nontrivial solutions of the form $(\eta (X),\phi (X,Y))=(\eta _{k},\phi _k(Y)) \exp (\text {i}kX)$ to the system (1.1)–(1.2) linearized at $(\eta _0(X), \phi _0(X,Y))=(0,0)$. Associated to any positive root k of the linear dispersion relation, one finds a one-parameter family of periodic waves $\{(\eta _\varepsilon (X),\phi _\varepsilon (X,Y))\}_{\varepsilon \in (-\varepsilon _0,\varepsilon _0)}$ with wavenumbers close to k, for sufficiently small $\varepsilon _0>0$. These periodic waves bifurcate from the trivial solution $(\eta _0,\phi _0)=(0,0)$.

Depending on the values of the two parameters $\alpha$ and $\beta$, the linear dispersion relation (1.3) possesses positive roots in the following cases:

• 1. one positive simple root $k_*>0$ if $\alpha \in (0, 1)$ and $\beta > 0$; we refer to this set of parameters as Region I;

• 2. two positive simple roots $0<k_{*,1}<k_{*,2}$ if $\alpha > 1$ and $0<\beta <\beta (\alpha )$, where $(\alpha ,\beta (\alpha ))$ belongs to the curve $\Gamma$ with parametric equations

  $$\begin{aligned} \begin{aligned}&\alpha = \frac{s^2}{2 \sinh ^2(s)} + \frac{s}{2 \, \tanh (s)}\\&\beta = -\frac{1}{2\sinh ^2(s)} + \frac{1}{2s \, \tanh (s)}, \end{aligned} s \in (0, \infty ); \end{aligned}$$

  (1.4)

  we refer to this set of parameters as Region II;

• 3. one positive simple root $k_*>0$ if $\alpha =1$ and $\beta <1/3$;

• 4. one positive double root $k_*$ if $(\alpha ,\beta )$ belongs to the curve $\Gamma$ given in (1.4).

The linear dispersion relation being even in k, together with any positive root k we also find the negative root $-k$. We illustrate these properties in the left panel of Fig. 1.

Fig. 1

Left: in the $(\beta ,\alpha )$-plane, sketch of the nonzero roots of the linear dispersion relation (1.3). We use dots to indicate simple roots and crosses to indicate double roots. Right: in Region II, plot of the curves $\Gamma _m$ for $m=2,3,4$ which are excluded from our analysis

Here, we focus on the periodic waves which bifurcate in the two open parameter regions I and II. For simplicity, in Region II we assume that $k_{*,1}$ and $k_{*,2}$ satisfy the non-resonance condition $k_{*,2}/k_{*,1} \notin \mathbb {N}$. This assumption means that $(\alpha ,\beta )$ does not belong to any of the curves $\Gamma _m$ for $m \in \mathbb {N}$, $m \ge 2$, with parametric equations

$$\begin{aligned} \begin{array}{l} \alpha = \displaystyle {-\frac{m^2 s}{(1-m^2)\tanh (s)}+\frac{ms}{(1-m^2)\tanh ( ms)}}\\ \beta = \displaystyle {\frac{1}{(1-m^2)s \tanh (s)}- \frac{m}{(1-m^2)s \tanh (ms)}} \end{array}, \quad s \in (0, \infty );\end{aligned}$$

see the right panel in Fig. 1. Then, for any fixed $(\alpha ,\beta )$ in Region I, there is a one-parameter family of two-dimensional periodic waves $\{(\eta _\varepsilon (X),\phi _\varepsilon (X,Y))\}_{\varepsilon \in (-\varepsilon _0,\varepsilon _0)}$ with wavenumbers close to $k_*$, whereas for $(\alpha ,\beta )$ in Region II, there are two geometrically distinct families of periodic waves

$$\begin{aligned}\{(\eta _{\varepsilon ,1}(X), \phi _{\varepsilon ,1}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}\, \, \text {and} \, \, \{(\eta _{\varepsilon ,2}(X), \phi _{\varepsilon , 2}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}\end{aligned}$$

with wavenumbers close to $k_{*,1}$ and $k_{*,2}$, respectively.

The purpose of our transverse dynamics analysis is twofold: to identify the periodic waves in regions I and II which are transversely linearly unstable and to discuss the induced dimension-breaking bifurcations. Roughly speaking, a two-dimensional wave is transversely linearly unstable if the Euler equations (1.1)–(1.2) linearized at the wave possess solutions which are bounded in the horizontal coordinates (X, z) and exponentially growing in time t. The dimension-breaking bifurcation is the bifurcation of three-dimensional steady solutions emerging from the two-dimensional transversely unstable wave. Typically, these three-dimensional solutions are periodic in the transverse horizontal coordinate z; see Fig. 2 for an illustration in the case of a two-dimensional periodic wave. Though of different type, these two questions share a common spectral analysis of the linear operator at the two-dimensional wave. This is the key, and most challenging, part of our analysis.

Fig. 2

Illustration of a dimension-breaking bifurcation. Left: plot of a two-dimensional periodic wave. Right: plot of a bifurcating three-dimensional doubly periodic wave

The transverse stability of periodic waves was mostly studied for simpler model equations obtained from the Euler equations (1.1)–(1.2) in different parameter regimes: the Kadomtsev–Petviashvili-I equation for the regime of large surface tension ($\alpha \sim 1,\ \beta >1/3$) was considered in [26, 27, 39], the Davey–Stewartson system for the regime of weak surface tension ($(\alpha ,\beta )$ close to the curve $\Gamma$) in [17], and a fifth order KP equation for the regime of critical surface tension ($\alpha \sim 1,\ \beta \sim 1/3$) in [32]; see also the recent review paper [29]. All these results predict that gravity–capillary periodic waves are linearly transversely unstable. We point out that pure gravity periodic, or solitary, water waves ($\beta =0$) are expected to be linearly transversely stable [2, 31].

For the Euler equations, previous works on transverse instability mostly treat the case of solitary waves; see [22, 48, 49] for the large surface regime and the more recent work [25] for the weak surface tension regime close to the curve $\Gamma$. In both regimes, the dimension-breaking bifurcation has been studied in [23] (large surface tension) and [25] (weak surface tension). For periodic waves, the transverse instability predicted in the regime of large surface tension ($\alpha \sim 1,\ \beta >1/3$) has been confirmed in [28]. In addition, the dimension-breaking bifurcation was studied showing the bifurcation of a one-parameter family of three-dimensional doubly periodic waves, as illustrated in Fig. 2. In the present work, we treat these two questions for the periodic waves bifurcating in the open parameter regions I and II.

For completeness, we mention that there are other stability/instability results for these periodic waves. When the perturbations are constant in z, the references [15, 16, 43] through formal expansions have provided a characterization for the Benjamin–Feir instability¹ (see e.g. [6, 7, 9, 38, 47] for rigorous proofs), the work [12] demonstrates numerically that periodic waves are sometimes spectrally unstable even when the Benjamin–Feir instability is not present, references [40, 41, 44,45,46, 50] indicate through both numerical and theoretical investigations that harmonic resonances feature even more intriguing instability phenomena, e.g. nested instabilities or multiple high-frequency instability bubbles. The works [35,36,37] approach these questions through their own proposals of fully dispersive model equations. In particular, [36, 37] find qualitatively the same instability characterization for periodic waves in their models as [15, 43]. Instability under three-dimensional perturbations has been considered numerically, experimentally and using various model equations, such as the Davey–Stewartson equation [11, 15, 33, 34, 52]. In particular, the instability criterion that we arrive at here can be formally obtained by taking $l=0$ in equation (3.9) in [11], and using the formulas for the coefficients for gravity–capillary waves in [1, 15]; see Appendix D. Note that in contrast to the previous studies, we restrict our attention to perturbations which have the same wavelength as the periodic wave in the X-direction.

Our approach to transverse dynamics follows the ideas developed for solitary waves in [22, 23, 25]. The starting point of the analysis is a spatial dynamics formulation of the three-dimensional, time-dependent equations (1.1)–(1.2) in which the horizontal coordinate z, transverse to the direction of propagation, plays the role of time. After flattening the free surface by an appropriate change of variables, the Eqs. (1.1)–(1.2) are written as a dynamical system of the form

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}U}{\mathop {}\!\textrm{d}z} = {\mathcal {F}}(U), \end{aligned}$$

(1.5)

together with nonlinear boundary conditions of the form ${\mathcal {B}}(U)=0$ on $y=0,1$, such that ${\mathcal {F}}(U)$ and ${\mathcal {B}}(U)$ do not contain derivatives with respect to the horizontal coordinate z. Two-dimensional solutions of the Euler equations (1.1)–(1.2) correspond to equilibria of this dynamical system. We refer to [25] and the references therein for a discussion of spatial dynamics approaches in the literature.

For the transverse linear instability problem, we consider the linearization of the dynamical system (1.5) at a two-dimensional steady periodic solution and apply a simple general instability criterion [17] adapted to the Euler equations in [25]. In Region I, we show that the periodic waves $\{(\eta _\varepsilon (X),\phi _\varepsilon (X,Y))\}_{\varepsilon \in (-\varepsilon _0,\varepsilon _0)}$ are transversely linearly unstable, provided $\varepsilon _0$ is sufficiently small. In Region II, we obtain transverse linear instability for the periodic waves $\{(\eta _{\varepsilon ,2},\phi _{\varepsilon , 2})\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ with wavenumbers close to the largest root $k_{*,2}$ of the linear dispersion relation. For the second family of periodic waves, $\{(\eta _{\varepsilon ,1},$ $\phi _{\varepsilon , 1})\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ with wavenumbers close to $k_{*,1}$, we can only conclude on transverse instability for parameter values $(\alpha , \beta )$ situated in the open region between the curves $\Gamma$ and $\Gamma _2$; see the right panel in Fig. 1.

The dimension-breaking bifurcation is studied for the transversely linearly unstable periodic waves. Here, we use the time-independent, but nonlinear, version of the dynamical system above. Applying a Lyapunov center theorem, we prove that from each unstable periodic wave bifurcates a family of doubly periodic waves. These doubly periodic waves are qualitatively similar to the ones found in [28] in the case of large surface tension, but different from other doubly periodic waves in the literature, as for instance the ones constructed in [10, 21] which bifurcate from the trivial solution.

The common part of the proofs of these two results is the analysis of the purely imaginary spectrum of a linear differential operator with variable coefficients determined by the two-dimensional steady periodic solution. This analysis is the major part of our work. Our main result shows that this linear operator possesses precisely one pair of simple nonzero purely imaginary eigenvalues. Though it relies upon standard perturbation arguments for linear operators, the proof is rather long because of the complicated formulas for the linear operator. This spectral result is the key property allowing to apply both the transverse instability criterion and the Lyapunov center theorem.

In the following theorem, we summarize the results obtained for Region I.

Theorem 1.1

(Region I) Fix $(\alpha ,\beta )$ in Region I and let $k_* > 0$ be the unique positive root of the linear dispersion relation (1.3).

• (i) (Existence) There exist $\varepsilon _0 > 0$ and a one-parameter family of two-dimensional steady solutions $\{(\eta _\varepsilon (X), \phi _\varepsilon (X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ to Eqs. (1.1)–(1.2), such that $(\eta _0,\phi _0) =(0, 0)$ and $(\eta _\varepsilon , \phi _\varepsilon )$ are periodic in X with wavenumber $k_\varepsilon = k_* + \mathcal {O}(\varepsilon ^2)$.

• (ii) (Transverse instability) There exists $\varepsilon _1>0$ such that for each $\varepsilon \in (-\varepsilon _1, \varepsilon _1)$ the periodic solution $(\eta _\varepsilon (X), \phi _\varepsilon (X,Y))$ is transversely linearly unstable.

• (iii) (Dimension-breaking bifurcation) There exists $\varepsilon _2>0$, such that for each $\varepsilon \in (-\varepsilon _2, \varepsilon _2)$ there exist $\delta _\varepsilon > 0$, $\ell _\varepsilon ^*>0$, and a one-parameter family of three-dimensional doubly periodic waves $\{(\eta _\varepsilon ^{\delta }(X, z),\phi _\varepsilon ^{\delta }(X,Y,z))\}_{\delta \in (-\delta _\varepsilon , \delta _\varepsilon )}$, with wavenumber $k_\varepsilon$ in X and wavenumber $\ell _\delta = \ell _\varepsilon ^* + \mathcal {O}(\delta ^2)$ in z, bifurcating from the periodic solution $(\eta _\varepsilon (X),$ $\phi _\varepsilon (X, Y))$.

We point out that $\pm {{{\,\textrm{i}\,}}}\ell _\varepsilon ^*$ where $\ell _\varepsilon ^*>0$ is given in Theorem 1.1(iii) are the two nonzero purely imaginary eigenvalues of the linearization at the periodic wave. The results found for Region II are summarized in Theorem 5.1 from Sect. 5.

In our presentation we focus on Region I, the arguments being, up to some computations, the same for Region II. In Sect. 2 we recall the spatial dynamics formulation of the three-dimensional time-dependent Euler equations (1.1)–(1.2) from [25] and the existence result for two-dimensional periodic waves given in Theorem 1.1(i). We also give some explicit expansions of these solutions which are computed in Appendix A. In Sect. 3 we prove the results for the linear operator. Some of the long computations needed here are given in Appendices B and C. In Sect. 4 we present the transverse dynamics results and in Sect. 5 we discuss the results for Region II. Finally, in Appendix D we show how the instability criterion can be derived formally using the Davey–Stewartson approximation.

2 Preliminaries

In this section, we recall the spatial dynamics formulation from [25] and the result on existence of two-dimensional periodic solutions of the system (1.1)–(1.2).

2.1 Spatial Dynamics Formulation

Following [25], we make the change of variables

$$\begin{aligned} Y=y(1+\eta (X, z, t)),\quad \phi (X,Y, z, t) = \Phi (X, y, z, t), \end{aligned}$$

in (1.1)–(1.2) to flatten the free surface. Since we consider periodic solutions, in addition, we set $x=k X$ with k the wavenumber in X. We introduce two new variables,

$$\begin{aligned}\begin{aligned}\omega&= -\int _0^1 \left( \Phi _z - \frac{y \eta _z \Phi _y}{1+\eta }\right) y \Phi _y \mathop {}\!\textrm{d}y + \frac{\beta \eta _z}{(1+k^2\eta _x^2+\eta _z^2)^{1/2}}, \\ \xi&= (1+\eta ) \left( \Phi _z - \frac{y \eta _z \Phi _y}{1+\eta }\right) . \end{aligned}\end{aligned}$$

and set $U = (\eta , \omega , \Phi , \xi )^{\text {T}}$. Then, the Eqs. (1.1)–(1.2) can be written as a dynamical system of the form

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}U}{\mathop {}\!\textrm{d}z} = D U_t + F(U), \end{aligned}$$

(2.1)

with boundary conditions

$$\begin{aligned} \Phi _y = y \eta _t + B(U) \quad \text {on} \quad y=0,1. \end{aligned}$$

(2.2)

Here, D is the linear operator defined by

$$\begin{aligned} D U=(0, \Phi |_{y=1}, 0, 0)^{\text {T}},\end{aligned}$$

F is the nonlinear mapping $F(U)=(F_1(U), F_2(U), F_3(U), F_4(U))^{\text {T}}$ given by

$$\begin{aligned} \begin{aligned} F_1(U)&= W \left( \frac{1+k^2\eta _x^2}{\beta ^2-W^2}\right) ^{1/2},\\ F_2(U)&= \frac{W}{(1+\eta )^2} \left( \frac{1+k^2\eta _x^2}{\beta ^2-W^2}\right) ^{1/2} \int _0^1 y \Phi _y \xi \mathop {}\!\textrm{d}y - k^2\left[ \eta _x \left( \frac{\beta ^2-W^2}{1+k^2\eta _x^2}\right) ^{1/2} \right] _x \\&\quad + \alpha \eta - k\Phi _x|_{y=1} \\&\quad + \int _0^1 \Bigg \{ \frac{\xi ^2 - \Phi ^2_y}{2(1+\eta )^2} + \frac{k^2}{2} \left( \Phi _x - \frac{y \eta _x \Phi _y}{1+\eta }\right) ^2 + k^2\left[ \left( \Phi _x - \frac{y \eta _x \Phi _y}{1+\eta }\right) y \Phi _y \right] _x \\&\quad \qquad \qquad + k^2 \left( \Phi _x - \frac{y \eta _x \Phi _y}{1+\eta }\right) \frac{y \eta _x \Phi _y}{1+\eta } \Bigg \} \mathop {}\!\textrm{d}y, \\ F_3(U)&= \frac{\xi }{1+\eta }+ \frac{y \Phi _y W}{1+\eta } \left( \frac{1+k^2\eta _x^2}{\beta ^2-W^2}\right) ^{1/2}, \end{aligned}\end{aligned}$$

and

$$\begin{aligned}\begin{aligned} F_4(U)&= -\frac{\Phi _{yy}}{1 + \eta } - k^2\left[ (1+\eta )\left( \Phi _x - \frac{y \eta _x \Phi _y}{1+\eta }\right) \right] _x \\&\qquad + k^2\left[ \left( \Phi _x - \frac{y \eta _x \Phi _y}{1+\eta }\right) y \eta _x \right] _y + \frac{(y\xi )_y W}{1+\eta } \left( \frac{1+k^2\eta _x^2}{\beta ^2-W^2}\right) ^{1/2}, \end{aligned} \end{aligned}$$

where

$$\begin{aligned} W = \omega + \frac{1}{1+\eta } \int _0^1 y \Phi _y \xi \mathop {}\!\textrm{d}y, \end{aligned}$$

and B is the nonlinear mapping defined by

$$\begin{aligned} B(U) = - ky\eta _x + k^2y\eta _x \Phi _x + \frac{\eta \Phi _y}{1+\eta }- \frac{k^2y^2\eta _x^2 \Phi _y}{1+\eta } + \frac{y\xi W}{1+\eta } \left( \frac{1+k^2\eta _x^2}{\beta ^2-W^2}\right) ^{1/2}. \end{aligned}$$

The choice of the function spaces is made precise later in Sects. 3 and 4.

The system (2.1)–(2.2) inherits the symmetries of the Euler equations (1.1)–(1.2). As a consequence of the horizontal spatial reflection $z\mapsto -z$, the system (2.1)–(2.2) is reversible with reversibility symmetry R acting by

$$\begin{aligned} R\begin{pmatrix} \eta \\ \omega \\ \Phi \\ \xi \end{pmatrix} (x,y,t)= \begin{pmatrix}\eta \\ -\omega \\ \Phi \\ -\xi \end{pmatrix}(x,y,t), \end{aligned}$$

(2.3)

which anticommutes with D and F and commutes with B. The second horizontal spatial reflection $x\mapsto -x$, implies that the system (2.1)–(2.2) possesses a reflection symmetry

$$\begin{aligned} S \begin{pmatrix}\eta \\ \omega \\ \Phi \\ \xi \end{pmatrix}(x,y,t)= \begin{pmatrix}\eta \\ \omega \\ -\Phi \\ xi \end{pmatrix} (-x,y,t), \end{aligned}$$

(2.4)

which commutes with D, F, and B. There are in addition two continuous symmetries, which are the horizontal spatial translations in x and z.

2.2 Two-Dimensional Periodic Waves

As mentioned in the introduction, the existence of two-dimensional traveling water waves is well recorded in the literature and can be proved by different methods. We give in Appendix A a short existence proof based on an analytic version of the Crandall–Rabinowitz local bifurcation theorem. For $(\alpha ,\beta )$ in Region I, we obtain from Theorem 1.1(i) the branch of solutions $\{(\eta _\varepsilon (X), \phi _\varepsilon (X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ with

$$\begin{aligned} \eta _\varepsilon (X) = \widetilde{\eta }_\varepsilon (x),\quad \phi _\varepsilon (X,Y)=\widetilde{\Phi }_\varepsilon (x,y),\quad X=k_\varepsilon x,\quad Y=y(1+\eta (X)), \end{aligned}$$

where $(\widetilde{\eta }_\varepsilon ,\widetilde{\Phi }_\varepsilon )\in H^{2}_{{{\,\textrm{per}\,}}}({\mathbb {S}})\times H^{2}_{{{\,\textrm{per}\,}}}(\Sigma )$ is analytic in $\varepsilon$, in which

$$\begin{aligned}\begin{aligned}&H^{s}_{{{\,\textrm{per}\,}}}({\mathbb {S}}) = \{u \in H^{s}_{{{\,\textrm{loc}\,}}}(\mathbb {R}) : \, u(x+2\pi ) = u(x), \ x \in \mathbb {R}\},\\&H^{s}_{{{\,\textrm{per}\,}}}(\Sigma ) = \{u \in H^{s}_{{{\,\textrm{loc}\,}}}(\mathbb {R}\times (0,1)) : \, u(x+2\pi , y) = u(x,y), \ y \in (0,1), \ x \in \mathbb {R}\}, \end{aligned}\end{aligned}$$

and $\mathbb {S} = (0, 2\pi )$, $\Sigma = \mathbb {S} \times (0,1)$. In addition, the functions $\widetilde{\eta }_\varepsilon$ and $\widetilde{\Phi }_\varepsilon$ are even and odd in x, respectively, and satisfy

$$\begin{aligned} (\widetilde{\eta }_\varepsilon ,\widetilde{\Phi }_\varepsilon )(x+\pi ) = (\widetilde{\eta }_{-\varepsilon },\widetilde{\Phi }_{-\varepsilon })(x). \end{aligned}$$

As a consequence of the latter property, the wavenumber $k_\varepsilon$ is even in $\varepsilon$.

Besides this existence result, for our purposes we need to compute the first terms of the expansion in $\varepsilon$ of the two-dimensional periodic solutions. We write²

$$\begin{aligned}{} & {} k_\varepsilon = k_* + \varepsilon ^2 k_2 + \mathcal {O}(\varepsilon ^4), \nonumber \\{} & {} \widetilde{\eta }_\varepsilon (x) = \varepsilon \eta _1(x) + \varepsilon ^2 \eta _2(x) + \mathcal {O}(\varepsilon ^3),\nonumber \\{} & {} \widetilde{\Phi }_\varepsilon (x, y) = \varepsilon \Phi _1(x,y) + \varepsilon ^2 \Phi _2(x,y) + \mathcal {O}(\varepsilon ^3), \end{aligned}$$

(2.5)

where $k_*$ is the positive root of the linear dispersion relation (1.3). Substituting these expansions into the Euler equations (1.1)–(1.2), we obtain in Appendix A the following explicit formulas:

$$\begin{aligned} k_2= & {} \frac{k_*^3}{d(k_*)} \Big ( \left( 9\alpha \beta +16\right) k_* - 12\alpha \beta k_*\cosh (2k_*) + 3\alpha \beta k_* \cosh (4k_*)\nonumber \\{} & {} \qquad \qquad - 8\alpha (2c(k_*) -1) \sinh (2k_*)-4\alpha (c(k_*)+2)\sinh (4k_*) \Big ), \end{aligned}$$

(2.6)

and

$$\begin{aligned}{} & {} \eta _1(x)=\sinh (k_*)\cos (x),\quad \Phi _1(x,y) = \cosh (k_*y)\sin (x), \nonumber \\{} & {} \eta _2(x) = \frac{k_*}{4}\left( c(k_*) + 1 \right) \sinh (2k_*)\cos (2x) - \frac{k_*^2}{4\alpha }, \nonumber \\{} & {} \Phi _2(x,y) = \frac{k_*}{4}\left( c(k_*) \cosh (2k_*y)+2\sinh (k_*)y\sinh (k_*y)\right) \sin (2x), \end{aligned}$$

(2.7)

where

$$\begin{aligned} c(k_*)= & {} -1- \frac{k_*(\cosh (2k_*)+2)}{{\mathcal {D}}(2k_*)},\nonumber \\ d(k_*)= & {} 32\alpha \left( 2\beta k_*(\cosh (2k_*)-1) + 2k_* -\sinh (2k_*) \right) , \end{aligned}$$

(2.8)

and ${\mathcal {D}}(k)$ is the linear dispersion relation (1.3).

For each $\varepsilon \in (-\varepsilon _0, \varepsilon _0)$, the solution $({\eta }_\varepsilon ,{\Phi }_\varepsilon )$ of the Euler equations (1.1)–(1.2) provides a solution

$$\begin{aligned} U_\varepsilon (x,y)= (\widetilde{\eta }_\varepsilon (x), 0, \widetilde{\Phi }_\varepsilon (x, y), 0)^{{{\,\mathrm{\text {T}}\,}}} \end{aligned}$$

(2.9)

of the dynamical system (2.1)–(2.2) for $k=k_\varepsilon$, hence satisfying

$$\begin{aligned} {\left\{ \begin{array}{ll} F(U_\varepsilon ) = 0,\\ \widetilde{\Phi }_{\varepsilon y} = B(U_\varepsilon ), &{} \text {on}\, y = 0,1. \end{array}\right. } \end{aligned}$$

(2.10)

In addition, the above parity properties of $\widetilde{\eta }_\varepsilon$ and $\widetilde{\Phi }_\varepsilon$ imply that $SU_\varepsilon =U_\varepsilon$ where S is the reflection symmetry given in (2.4).

3 Analysis of the Linear Operator

For fixed $(\alpha ,\beta )$ in Region I, we denote by $L_\varepsilon$ the linear operator which appears in the linearization of the dynamical sytem (2.1)–(2.2) at the periodic wave $U_\varepsilon$ for $k=k_\varepsilon$. We prove the properties of $L_\varepsilon$ needed for the transverse dynamics analysis in Sect. 4.

For notational simplicity we remove the tilde from (2.10) and write from now on ${\eta }_\varepsilon$ and ${\Phi }_\varepsilon$ instead of $\widetilde{\eta }_\varepsilon$ and $\widetilde{\Phi }_\varepsilon$, respectively.

3.1 The Linear Operator $\varvec{L_\varepsilon }$

A direct computation of the differential of F at the periodic wave $U_\varepsilon$ gives the following explicit formulas for $L_\varepsilon U{:}{=}\mathop {}\!\textrm{d}F[U_\varepsilon ] U$,

$$\begin{aligned} L_\varepsilon U = \begin{pmatrix} \omega /\beta + H_1(\omega , \xi ) \\ \alpha \eta - \beta k_\varepsilon ^2 \eta _{xx} - k_\varepsilon \Phi _x|_{y=1} + H_2(\eta , \Phi )\\ \xi + H_3(\omega , \xi ) \\ -k_\varepsilon ^2 \Phi _{xx} - \Phi _{yy} + H_4(\eta , \Phi )\end{pmatrix},\quad U= \begin{pmatrix} \eta \\ \omega \\ \Phi \\ \xi \end{pmatrix}, \end{aligned}$$

(3.1)

where

$$\begin{aligned}H_1(\omega , \xi )&= \frac{(1 + k^2_\varepsilon \eta _{\varepsilon x}^2)^{1/2}}{\beta } \left( \omega + \frac{1}{1+\eta _\varepsilon } \int _0^1 y \Phi _{\varepsilon y} \xi \mathop {}\!\textrm{d}y \right) - \frac{\omega }{\beta },\nonumber \\ H_2(\eta , \Phi )&= \beta k^2_\varepsilon \eta _{xx} - \beta k^2_\varepsilon \left[ \frac{\eta _x}{(1+k^2_\varepsilon \eta _{\varepsilon x}^2)^{3/2}}\right] _x\nonumber \\&\quad + \int _0^1 \left\{ k^2_\varepsilon \Phi _{\varepsilon x}\Phi _x - \frac{\Phi _{\varepsilon y}\Phi _y}{(1+\eta _\varepsilon )^2} + \frac{\Phi _{\varepsilon y}^2\eta }{(1+\eta _\varepsilon )^3}-k^2_\varepsilon \frac{y^2\eta _{\varepsilon x}^2\Phi _{\varepsilon y}\Phi _y}{(1+\eta _\varepsilon )^2}\right. \nonumber \\&\quad \left. -k^2_\varepsilon \frac{y^2\eta _{\varepsilon x}\Phi _{\varepsilon y}^2 \eta _x}{(1+\eta _\varepsilon )^2}+ k_\varepsilon \frac{y^2 \eta _{\varepsilon x}\Phi _{\varepsilon y}^2 \eta }{(1+\eta _\varepsilon )^3}\right. \nonumber \\&\quad \left. + k^2_\varepsilon \left[ y\Phi _{\varepsilon y}\Phi _x + y\Phi _{\varepsilon x}\Phi _y - \frac{2y^2\eta _{\varepsilon x}\Phi _{\varepsilon y}\Phi _y}{1+\eta _\varepsilon }-\frac{y^2 \Phi _{\varepsilon y}^2 \eta _x}{1+\eta _\varepsilon } + \frac{y^2 \Phi _{\varepsilon y}^2 \eta _{\varepsilon x}\eta }{(1+\eta _\varepsilon )^2}\right] _x\right\} \mathop {}\!\textrm{d}y,\\ \end{aligned}$$

$$\begin{aligned}\begin{aligned}&H_3(\omega , \xi ) = -\frac{\eta _\varepsilon \xi }{1+\eta _\varepsilon } + \frac{(1 + k^2_\varepsilon \eta _{\varepsilon x}^2)^{1/2} y \Phi _{\varepsilon y}}{\beta (1+\eta _\varepsilon )} \left( \omega + \frac{1}{1+\eta _\varepsilon } \int _0^1 y \Phi _{\varepsilon y} \xi \mathop {}\!\textrm{d}y \right) , \\&H_4(\eta , \Phi ) = k^2_\varepsilon \Big [-\eta _\varepsilon \Phi _x - \Phi _{\varepsilon x}\eta + y \Phi _{\varepsilon y}\eta _x + y\eta _{\varepsilon x}\Phi _y\Big ]_x + \Big [k_\varepsilon y\eta _x + B_{l\varepsilon }(\eta , \Phi )\Big ]_y. \end{aligned}\end{aligned}$$

To this expression of $L_\varepsilon U$ we add the linear boundary conditions obtained by taking the differential of B at $U_\varepsilon$,

$$\begin{aligned} \Phi _y= B_{l\varepsilon }(U) {:}{=} \mathop {}\!\textrm{d}B[U_\varepsilon ] U=0,\quad \text {on}\, y = 0,1, \end{aligned}$$

(3.2)

where

$$\begin{aligned}\begin{aligned}B_{l\varepsilon }(U)&= k_\varepsilon y(-\eta _x + k_\varepsilon \eta _{\varepsilon x} \Phi _x + k_\varepsilon \Phi _{\varepsilon x}\eta _x) \\&\quad + \frac{\eta _\varepsilon \Phi _y}{1+\eta _\varepsilon } + \frac{\Phi _{\varepsilon y} \eta }{(1+\eta _\varepsilon )^2} + k^2_\varepsilon \frac{y^2 \eta _{\varepsilon x}^2 \Phi _{\varepsilon y} \eta }{(1+\eta _\varepsilon )^2} - k^2_\varepsilon \frac{y^2\eta _{\varepsilon x}^2 \Phi _y}{1+\eta _\varepsilon } - 2k^2_\varepsilon \frac{y^2 \eta _{\varepsilon x}\Phi _{\varepsilon y} \eta _x}{1+\eta _\varepsilon }. \end{aligned}\end{aligned}$$

Notice that $B_{l\varepsilon }(U)$ only depends on the components $\eta$ and $\Phi$ of U. We will sometimes write $B_{l\varepsilon }(\eta , \Phi )$ for convenience.

For $s\ge 0$, we define the Hilbert space

$$\begin{aligned} {\mathcal {X}}^s = H^{s+1}_{{{\,\textrm{per}\,}}}({\mathbb {S}}) \times H^s_{{{\,\textrm{per}\,}}}({\mathbb {S}}) \times H^{s+1}_{{{\,\textrm{per}\,}}}(\Sigma ) \times H^s_{{{\,\textrm{per}\,}}}(\Sigma ). \end{aligned}$$

(3.3)

The action of the operator $L_\varepsilon$ is taken in ${\mathcal {X}}^0$ with domain of definition

$$\begin{aligned}{\mathcal {Y}}^1_\varepsilon = \{U=(\eta , \omega , \Phi , \xi )^{{{\,\mathrm{\text {T}}\,}}} \in {\mathcal {X}}^1 : \, \Phi _y = B_{l\varepsilon }(\eta , \Phi ) \ \text {on}\, y=0,1\},\end{aligned}$$

chosen to include the boundary conditions. Then $L_\varepsilon$ is well-defined and closed in ${\mathcal {X}}^0$, and its domain ${\mathcal {Y}}^1_\varepsilon$ is compactly embedded in ${\mathcal {X}}^0$. The latter property implies that the operator $L_\varepsilon$ has pure point spectrum consisting of isolated eigenvalues with finite algebraic multiplicity. As a consequence of the reflection symmetry S given in (2.4), which commutes with F and leaves invariant $U_\varepsilon$, the subspaces

$$\begin{aligned} {\mathcal {X}}^0_+=\{U\in {\mathcal {X}}^0:\, SU=U\},\quad {\mathcal {X}}^0_-=\{U\in {\mathcal {X}}^0:\, SU=-U\}, \end{aligned}$$

(3.4)

are invariant under the action of $L_\varepsilon$.

One inconvenience of this functional-analytic setting is that the domain of definition ${\mathcal {Y}}^1_\varepsilon$ of the linear operator $L_\varepsilon$ depends on $\varepsilon$. This difficulty is well-known and can be handled using an appropriate change of variables first introduced for the three-dimensional steady nonlinear Euler equations in [24]. Here, we proceed as in [22] and replace $\Phi$ by $\Upsilon =\Phi + \chi _y$, where $\chi$ is the unique solution of the elliptic problem

$$\begin{aligned} \begin{aligned}-k_\varepsilon ^2\chi _{xx}-\chi _{yy}&=B_{l\varepsilon }(U){} & {} \text {in}\, \Sigma ,\\ \chi&=0{} & {} \text {on}\, y = 0,1, \end{aligned} \end{aligned}$$

so that $\Upsilon$ satisfies the boundary conditions $\Upsilon _y=0$ on $y=0,1$ which do not depend on $\varepsilon$. The linear mapping defined by $G_\varepsilon (\eta ,\omega ,\Phi ,\xi )^{{{\,\mathrm{\text {T}}\,}}} = (\eta ,\omega ,\Upsilon ,\xi )^{{{\,\mathrm{\text {T}}\,}}}$, is a linear isomorphism in both ${\mathcal {X}}^0$ and ${\mathcal {X}}^1$, it depends smoothly on $\varepsilon$ and the same is true for its inverse $G_\varepsilon ^{-1}$. Setting ${\widetilde{L}}_\varepsilon = G_\varepsilon L_\varepsilon G_\varepsilon ^{-1}$ the operator ${\widetilde{L}}_\varepsilon$ acts in ${\mathcal {X}}^0$ with domain of definition

$$\begin{aligned}{\mathcal {Y}}^1 = \{U=(\eta , \omega , \Upsilon , \xi )^{{{\,\mathrm{\text {T}}\,}}} \in {\mathcal {X}}^1 : \,\Upsilon _y = 0 \ \text {on}\, y=0,1\},\end{aligned}$$

which does not depend on $\varepsilon$ anymore. While ${\widetilde{L}}_\varepsilon$ allows us to rigorously apply general results for linear operators, it is more convenient to use $L_\varepsilon$ for explicit computations.

3.2 Spectral Properties of $\varvec{L_0}$

The unperturbed operator $L_0$ obtained for $\varepsilon =0$ is a differential operator with constant coefficients. Therefore, eigenvalues, eigenfunctions, and generalized eigenfunctions can be explicitly computed using Fourier series in the variable x. In particular, for purely imaginary values ${{{\,\textrm{i}\,}}}\ell$ with $\ell \in \mathbb {R}$ the eigenvalue problem $(L_0 - {{{\,\textrm{i}\,}}}\ell \, {\mathbb {I}})U = 0$ possesses nontrivial solutions in the $n^{\text {th}}$ Fourier mode if and only if

$$\begin{aligned} (\alpha + \beta \sigma ^2)\sigma \sinh \sigma -n^2k_*^2 \cosh \sigma = 0 \quad \text {with}\quad \sigma ^2=n^2k_*^2+\ell ^2. \end{aligned}$$

For fixed $(\alpha ,\beta )$ in Region I, this equality holds if and only if $\ell = 0$ and $n=0,1$; see also [18]. Consequently, 0 is the only purely imaginary eigenvalue of $L_0$ and it has geometric multiplicity three. The associated eigenvectors are given by the explicit formulas:

$$\begin{aligned} \begin{aligned}&\zeta _0 = \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}, \quad \zeta _-= \begin{pmatrix} -\sinh (k_*)\sin (x) \\ 0 \\ \cosh (k_*y)\cos (x) \\ 0 \end{pmatrix}, \quad \zeta _+ = \begin{pmatrix} \sinh (k_*)\cos (x) \\ 0 \\ \cosh (k_*y) \sin (x) \\ 0 \end{pmatrix}. \end{aligned} \end{aligned}$$

(3.5)

Associated to each eigenvector there is a Jordan chain of length two, so that the algebraic multiplicity of the eigenvalue 0 is six. The generalized eigenvectors associated to $\zeta _0, \zeta _-$ and $\zeta _+$ are given by, respectively,

$$\begin{aligned} \psi _0 = \begin{pmatrix} 0 \\ 0 \\ 0 \\ 1 \end{pmatrix}, \quad \psi _- = \begin{pmatrix}0\\ -\beta \sinh (k_*) \sin (x) \\ 0 \\ \cosh (k_*y)\cos (x)\end{pmatrix}, \quad \psi _+=\begin{pmatrix} 0 \\ \beta \sinh (k_*)\cos (x) \\ 0 \\ \cosh (k_*y)\sin (x)\end{pmatrix}.\qquad \end{aligned}$$

(3.6)

Notice that the reflection symmetry S given in (2.4) acts on these eigenvectors as follows:

$$\begin{aligned}\begin{aligned}&S\zeta _0 = -\zeta _0,&\quad&S\zeta _-=-\zeta _-,&\quad&S\zeta _+=\zeta _+,&\\&S\psi _0= - \psi _0,&\quad&S\psi _-=-\psi _-,&\quad&S\psi _+ = \psi _+.&\end{aligned}\end{aligned}$$

These formulas are consistent with the ones already found in [18]. The remaining eigenvalues of $L_0$ are bounded away from the imaginary axis.

3.3 Main Result

We summarize in the next theorem the properties of the linear operator $L_\varepsilon$ needed for our transverse dynamics analysis. The same properties hold for the operator ${\widetilde{L}}_\varepsilon$.

Theorem 3.1

(Linear operator) There exist positive constants $\varepsilon _1$, $C_1$, and $\Lambda _1$, such that for each $\varepsilon \in (-\varepsilon _1, \varepsilon _1)$ the following properties hold.

• (i) The linear operator $L_\varepsilon$ acting in ${\mathcal {X}}^0$ with domain ${\mathcal {Y}}^1_\varepsilon$ has an eigenvalue 0 with algebraic multiplicity four, and two simple purely imaginary eigenvalues $\pm {{{\,\textrm{i}\,}}} \ell _\varepsilon$ with $\ell _\varepsilon >0$ and $\ell _0=0$. Any other purely imaginary value ${{{\,\textrm{i}\,}}}\ell \in {{{\,\textrm{i}\,}}}\mathbb {R}{\setminus } \{0, \pm {{{\,\textrm{i}\,}}}\ell _\varepsilon \}$ belongs to the resolvent set of $L_\varepsilon$.

• (ii) The restriction of $L_\varepsilon$ to the invariant subspace ${\mathcal {X}}^0_+$ has the two simple purely imaginary eigenvalues $\pm {{{\,\textrm{i}\,}}} \ell _\varepsilon$ and any other value ${{{\,\textrm{i}\,}}}\ell \in {{{\,\textrm{i}\,}}}\mathbb {R}{\setminus } \{\pm {{{\,\textrm{i}\,}}}\ell _\varepsilon \}$ belongs to the resolvent set.

• (iii) The inequality

  $$\begin{aligned}\left\| (L_\varepsilon -{{{\,\textrm{i}\,}}}\ell \,{\mathbb {I}})^{-1}\right\| _{{\mathcal {L}}(X^0)} \le \frac{C_1}{|\ell |},\end{aligned}$$

  holds for each real number $\ell$ with $|\ell |>\Lambda _1$.

Proof

We rely on the properties of the operator $L_0$ and perturbation arguments for $\varepsilon$ sufficiently small. The operators ${\widetilde{L}}_\varepsilon$ and ${\widetilde{L}}_0$ having the same domain of definition ${\mathcal {Y}}^1$, standard perturbation arguments show that ${\widetilde{L}}_\varepsilon$ is a small relatively bounded perturbation of ${\widetilde{L}}_0$ for $\varepsilon$ sufficiently small. The result in item (iii) is an immediate consequence of this property. Indeed, for $\varepsilon =0$ the inequality from (iii) is given in [18], which implies that a similar inequality holds for ${\widetilde{L}}_0$, with possibly different values $C_1$ and $\Lambda _1$. The operator ${\widetilde{L}}_\varepsilon$ being a relatively bounded perturbation of ${\widetilde{L}}_0$ for sufficiently small $\varepsilon$, from the inequality for ${\widetilde{L}}_0$ we obtain that item (iii) holds for ${\widetilde{L}}_\varepsilon$, and then for $L_\varepsilon$. It remains to prove items (i) and (ii). This is the main part of the proof of the theorem.

Spectral decomposition. The results in Sect. 3.2 show that the spectrum $\sigma (L_0)$ of the linear operator $L_0$ satisfies

$$\begin{aligned} \sigma (L_0)=\{0\}\cup \sigma _1(L_0),\quad \sigma _1(L_0)\subset \{\lambda \in {\mathbb {C}}:\, |{{{\,\textrm{Re}\,}}}\, \lambda |> d_1\}, \end{aligned}$$

for some $d_1>0$, where 0 is an eigenvalue with algebraic multiplicity six and geometric multiplicity three, and the same is true for the linear operator ${\widetilde{L}}_0$. The six-dimensional spectral subspace ${\mathcal {E}}_0$ associated to the eigenvalue 0 of $L_0$ is spanned by the eigenvectors $\zeta _0,\ \zeta _{\pm }$ given in (3.5) and generalized eigenvectors $\psi _0,\ \psi _{\pm }$ given in (3.6). For $\varepsilon \ne 0$ sufficiently small, ${\widetilde{L}}_\varepsilon$ is a small relatively bounded perturbation of ${\widetilde{L}}_0$. Consequently, there exists a neighborhood $V_0\subset {\mathbb {C}}$ of the origin such that

$$\begin{aligned} V_0\subset \{\lambda \in {\mathbb {C}}:\, |{{{\,\textrm{Re}\,}}}\,\lambda |< d_1/4\} \end{aligned}$$

and

$$\begin{aligned} \sigma ({\widetilde{L}}_\varepsilon )=\sigma _{0}({\widetilde{L}}_\varepsilon )\cup \sigma _{1}({\widetilde{L}}_\varepsilon ),\quad \sigma _{0}({\widetilde{L}}_\varepsilon )\subset V_0,\quad \sigma _{1}({\widetilde{L}}_\varepsilon )\subset \{\lambda \in {\mathbb {C}}:\, |{{{\,\textrm{Re}\,}}}\,\lambda |> d_1/2\}, \end{aligned}$$

for sufficiently small $\varepsilon$, where the spectral subspace associated to $\sigma _{0}({\widetilde{L}}_\varepsilon )$ is six-dimensional, and the same is true for $L_\varepsilon$. Moreover, for the operator $L_\varepsilon$, there exists a basis $\{\zeta _0(\varepsilon ), \zeta _{\pm }(\varepsilon ),$ $\psi _0(\varepsilon ), \psi _{\pm }(\varepsilon )\}$ of the six-dimensional spectral subspace ${\mathcal {E}}_\varepsilon$ associated to $\sigma _{0}(L_\varepsilon )$ which is the analytic continuation, for sufficiently small $\varepsilon$, of the basis $\{\zeta _0,\ \zeta _{\pm },\ \psi _0,\ \psi _{\pm }\}$ of the six-dimensional spectral subspace ${\mathcal {E}}_0$ associated to the eigenvalue 0 of $L_0$ [42, Chapter VII, $\S \,$1.3]. The two bases share the symmetry properties,

$$\begin{aligned}\begin{aligned}&S\zeta _0(\varepsilon ) = -\zeta _0(\varepsilon ),&\quad&S\zeta _-(\varepsilon )=-\zeta _-(\varepsilon ),&\quad&S\zeta _+(\varepsilon )=\zeta _+(\varepsilon ),&\\&S\psi _0(\varepsilon )= -\psi _0(\varepsilon ),&\quad&S\psi _-(\varepsilon )=-\psi _-(\varepsilon ),&\quad&S\psi _+(\varepsilon ) = \psi _+(\varepsilon ),&\end{aligned}\end{aligned}$$

which hold for $\varepsilon =0$ and are preserved for $\varepsilon \not =0$ because the subspaces ${\mathcal {X}}^0_\pm$ defined in (3.4) are invariant under the action of $L_\varepsilon$. Thus, we have the decomposition ${\mathcal {E}}_\varepsilon = {\mathcal {E}}_{\varepsilon ,+} \oplus {\mathcal {E}}_{\varepsilon ,-}$ with

$$\begin{aligned}\begin{aligned}&{\mathcal {E}}_{\varepsilon , +}=\{U \in {\mathcal {E}}_{\varepsilon } : \, SU = U\}=\text {span} \{\zeta _+(\varepsilon ), \psi _+(\varepsilon )\},\\&{\mathcal {E}}_{\varepsilon ,-}=\{U \in {\mathcal {E}}_\varepsilon : \, SU = -U\} =\text {span}\{\zeta _0(\varepsilon ), \zeta _-(\varepsilon ), \psi _0(\varepsilon ), \psi _-(\varepsilon )\}. \end{aligned}\end{aligned}$$

These spaces ${\mathcal {E}}_{\varepsilon , \pm }$ are invariant under the action of $L_\varepsilon$.

Purely imaginary eigenvalues of $L_\varepsilon$ necessarily belong to the neighborhood $V_0$ of 0. Therefore, they are determined by the action of $L_\varepsilon$ on the spectral subspace ${\mathcal {E}}_\varepsilon$. This action is represented by a $6\times 6$ matrix. The decomposition ${\mathcal {E}}_\varepsilon = {\mathcal {E}}_{\varepsilon ,+} \oplus {\mathcal {E}}_{\varepsilon ,-}$ above, implies that we can further decompose the action of $L_\varepsilon$ by restricting to the invariant subspaces ${\mathcal {E}}_{\varepsilon , \pm }$. In other words, the $6\times 6$ matrix is a block matrix with a $2 \times 2$ block representing the action of $L_\varepsilon$ on ${\mathcal {E}}_{\varepsilon , +}$ and a $4\times 4$ block representing the action of $L_\varepsilon$ on ${\mathcal {E}}_{\varepsilon , -}$. Our task is to determine the eigenvalues of these two matrices. This will prove the result in part (i) of the theorem. For the restriction of the linear operator $L_\varepsilon$ to the invariant subspace ${\mathcal {X}}^0_+$ in part (ii) of the theorem, it is enough to consider the eigenvalues of the $2 \times 2$ matrix.

Eigenvalues of the $\varvec{4\times 4}$ matrix. It turns out that a basis of the subspace ${\mathcal {E}}_{\varepsilon ,-}$ can be explicitly obtained using the symmetries of the Euler equations. First, the Euler equations (1.1)–(1.2) are invariant under the transformation $\phi \mapsto \phi + C$ for any real constant C. This implies that the dynamical system (2.1)–(2.2) is invariant under the transformation $U\mapsto U+\zeta _0$ where $\zeta _0=(0,0,1,0)^{\text {T}}$. Consequently, $\zeta _0$ belongs to the kernel of $L_\varepsilon$ and since $S\zeta _0=-\zeta _0$ it belongs to ${\mathcal {E}}_-$. We choose $\zeta _0(\varepsilon )=\zeta _0$ and then a direct computation gives the generalized eigenvector

$$\begin{aligned}\psi _0(\varepsilon ) = \begin{pmatrix} 0 \\ -\int _0^1 y \Phi _{\varepsilon y} \mathop {}\!\textrm{d}y \\ 0 \\ 1+\eta _\varepsilon \end{pmatrix},\end{aligned}$$

satisfying $L_\varepsilon \psi _0(\varepsilon )=\zeta _0$ and $S\psi _0(\varepsilon )=-\psi _0(\varepsilon )$.

Next, the invariance of the Euler equations under horizontal spatial translations in x implies that the derivative $U_{\varepsilon x}=(\eta _{\varepsilon x}, 0, \Phi _{\varepsilon x}, 0)^{\text {T}}$ of the periodic wave belongs to the kernel of $L_\varepsilon$. Since $SU_{\varepsilon x}=-U_{\varepsilon x}$, the vector $U_{\varepsilon x}$ belongs to ${\mathcal {E}}_{\varepsilon ,-}$. From the expansions (2.5), we find that $U_{\varepsilon x}=\varepsilon \zeta _-+\mathcal {O}(\varepsilon ^2)$. This gives a second vector $\zeta _-(\varepsilon ) = \varepsilon ^{-1}U_{\varepsilon x}$ which belongs to the kernel of $L_\varepsilon$, and also to the invariant subspace ${\mathcal {E}}_-$, with the property that $\zeta _-(\varepsilon )\rightarrow \zeta _-$ as $\varepsilon \rightarrow 0$. The corresponding generalized eigenvector is given by

$$\begin{aligned}\psi _-(\varepsilon ) = \frac{1}{\varepsilon }\begin{pmatrix} 0 \\ \dfrac{\eta _{\varepsilon x}\beta }{(1+k_\varepsilon ^2 \eta _{\varepsilon x}^2)^{1/2}} - \int _0^1 y \Phi _{\varepsilon y} \left( \Phi _{\varepsilon x}-\dfrac{\eta _{\varepsilon x}y\Phi _{\varepsilon y}}{1+\eta _{\varepsilon }}\right) \mathop {}\!\textrm{d}y \\ 0 \\ (1+\eta _\varepsilon )\left( \Phi _{\varepsilon x}-\dfrac{\eta _{\varepsilon x}y\Phi _{\varepsilon y}}{1+\eta _\varepsilon }\right) \end{pmatrix}.\end{aligned}$$

The above shows that there is a basis $\{\zeta _0(\varepsilon ), \psi _0(\varepsilon ), \zeta _-(\varepsilon ), \psi _-(\varepsilon )\}$ for ${\mathcal {E}}_{\varepsilon ,-}$ satisfying

$$\begin{aligned} L_\varepsilon \zeta _0(\varepsilon ) = 0,\quad L_\varepsilon \psi _0(\varepsilon ) = \zeta _0(\varepsilon ),\quad L_\varepsilon \zeta _-(\varepsilon ) = 0,\quad L_\varepsilon \psi _-(\varepsilon ) = \zeta _-(\varepsilon ). \end{aligned}$$

Thus, 0 is the only eigenvalue of the $4\times 4$ matrix representing the action of $L_\varepsilon$ onto ${\mathcal {E}}_{\varepsilon ,-}$ and it has geometric multiplicity two and algebraic multiplicity four.

Eigenvalues of the $\varvec{2\times 2}$ matrix. We consider a basis $\{\zeta _+(\varepsilon ), \psi _+(\varepsilon )\}$ of the subspace ${\mathcal {E}}_{\varepsilon ,+}$ which is the smooth continuation of the basis $\{\zeta _+,\psi _+\}$ of ${\mathcal {E}}_{0,+}$, and denote by ${\mathcal {M}}(\varepsilon )$ the $2 \times 2$ matrix representing the action of $L_\varepsilon$ on this basis. At $\varepsilon =0$, we have that $L_0\zeta _+=0$ and $L_0\psi _+=\zeta _+$, which implies that

$$\begin{aligned} {\mathcal {M}}(0) = \begin{pmatrix} 0 &{}\qquad 1 \\ 0 &{}\qquad 0 \end{pmatrix}. \end{aligned}$$

For $\varepsilon \ne 0$, we write

$$\begin{aligned} {\mathcal {M}}(\varepsilon ) = \begin{pmatrix} m_{11}(\varepsilon ) &{}\qquad 1+m_{12}(\varepsilon )\\ m_{21}(\varepsilon ) &{}\qquad m_{22}(\varepsilon ) \end{pmatrix}. \end{aligned}$$

The invariance of the Euler equations under horizontal spatial translations in x, implies that the periodic waves translated by a half-period $\pi$ are also periodic solutions. Comparing their expansions in $\varepsilon$ with the ones of $(\eta _\varepsilon , \Phi _\varepsilon )$ we conclude that

$$\begin{aligned} \eta _\varepsilon (x)=\eta _{-\varepsilon }(x+\pi ), \quad \Phi _\varepsilon (x,y)=\Phi _{-\varepsilon }(x+\pi ,y). \end{aligned}$$

Since the $2\times 2$ matrices corresponding to these solutions are the same this implies that ${\mathcal {M}}(\varepsilon ) = {\mathcal {M}}(-\varepsilon )$, and as a consequence, we have the expansion $m_{ij}(\varepsilon )=m_{ij}^{(2)}\varepsilon ^2 + {\mathcal {O}}(\varepsilon ^4)$, for $\varepsilon$ sufficiently small.

Next, the reversibility of $L_\varepsilon$ implies that the spectrum of $L_\varepsilon$ is symmetric with respect to the origin in the complex plane. Moreover, because $L_\varepsilon$ is a real operator, its spectrum is also symmetric with respect to the real line. These observations combined imply that the two eigenvalues of ${\mathcal {M}}(\varepsilon )$ are either both real or both purely imaginary, and their sum is equal to 0. Consequently, $m_{11}(\varepsilon )=-m_{22}(\varepsilon )$. Further, the product of these two eigenvalues is equal to the determinant of ${\mathcal {M}}(\varepsilon )$. Therefore, the eigenvalues are both real if $\det {\mathcal {M}}(\varepsilon ) < 0$ and both purely imaginary if $\det {\mathcal {M}}(\varepsilon )>0$. We have

$$\begin{aligned} \det {\mathcal {M}}(\varepsilon ) = -m_{11}^2(\varepsilon )-m_{21}(\varepsilon )(1+m_{12}(\varepsilon )) = -m_{21}^{(2)}\varepsilon ^2 + {\mathcal {O}}(\varepsilon ^4),\end{aligned}$$

so the result in theorem holds provided $m_{21}^{(2)}<0$.

The final step is the computation of the sign of $m_{21}^{(2)}$. We prove in Appendix B that

$$\begin{aligned} m_{21}^{(2)} = - 4k_* \cdot \frac{4\beta k_* \sinh ^2(k_*)+ 2k_* -\sinh (2k_*) }{4\beta k_* \sinh ^2(k_*)+2k_* + \sinh (2k_*)} \cdot k_2, \end{aligned}$$

(3.7)

where $k_2$ is the coefficient in the expansion of the wavenumber $k_\varepsilon$ given in (2.6). Replacing the formula for $k_2$ we obtain

$$\begin{aligned} m_{21}^{(2)} = \frac{k_*^4}{8\alpha }\cdot \frac{1}{4\beta k_* \sinh ^2(k_*)+2k_* + \sinh (2k_*)}\cdot {\widetilde{m}}_{21}^{(2)}, \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} {\widetilde{m}}_{21}^{(2)} =&-\left( 9\alpha \beta +16\right) k_* + 12\alpha \beta k_*\cosh (2k_*) - 3\alpha \beta k_* \cosh (4k_*)\\ {}&+8\alpha (2c(k_*) -1) \sinh (2k_*)+4\alpha (c(k_*)+2)\sinh (4k_*), \end{aligned} \end{aligned}$$

(3.8)

with $c(k_*)<-1$ given in (2.8). Clearly, $m_{21}^{(2)}$ and ${\widetilde{m}}_{21}^{(2)}$ have the same sign. Proposition C.1 in Appendix C shows that ${\widetilde{m}}_{21}^{(2)}<0$ for $(\alpha ,\beta )$ in Region I. This completes the proof of the theorem. $\square$

4 Transverse Dynamics

We show that the two-dimensional periodic waves in Theorem 1.1(i) are linearly transversely unstable for $\varepsilon$ sufficiently small, and then discuss the induced dimension-breaking bifurcation. These two results prove the parts (ii) and (iii) of Theorem 1.1.

Throughout this section, we consider a two-dimensional periodic wave $U_\varepsilon$ such that the associated linearized of operator $L_\varepsilon$ studied in Sect. 3 possesses two simple purely imaginary eigenvalues $\pm {{{\,\textrm{i}\,}}}\ell _\varepsilon$ as in Theorem 3.1, hence by fixing $\varepsilon \in (-\varepsilon _1,\varepsilon _1)$.

4.1 Transverse Linear Instability

Linearizing the system (2.1)–(2.2) at $U_\varepsilon$ we obtain the linear system

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}U}{\mathop {}\!\textrm{d}z}= DU_t + \mathop {}\!\textrm{d}F[U_\varepsilon ]U, \end{aligned}$$

(4.1)

with boundary conditions

$$\begin{aligned} \Phi _y = y\eta _t + B_{l\varepsilon }(U) \quad \text {on}\, y=0,1. \end{aligned}$$

(4.2)

The periodic wave $U_\varepsilon$ is transversely linearly unstable if the linear equation has a solution of the form $\exp (\sigma t) U_\sigma (z)$ with ${{\,\textrm{Re}\,}}\sigma > 0$ and $U_\sigma \in C^1_{\text {b}}(\mathbb {R}, {\mathcal {X}}^0) \cap C_{\text {b}}(\mathbb {R}, {\mathcal {X}}^1)$. For the construction of such a function, we closely follow the approach developed in [25] where the authors studied the transverse instability of solitary waves for the Euler equations. The only difference is that the functions were localized in $x\in \mathbb {R}$ in [25], whereas here they are periodic in x. We use the following general result from [25], which we have slightly modified; see Remark 4.2.

Theorem 4.1

(Theorem 1.3 [25]) Consider real Banach spaces ${\mathcal {X}}$, ${\mathcal {Z}}$, ${\mathcal {Z}}_i$, $i=1,2$, and a partial differential equation of the form

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}U}{\mathop {}\!\textrm{d}z} = D_1 U_t + D_2 U_{tt}+ LU. \end{aligned}$$

(4.3)

Assume that the following properties hold:

• (i) ${\mathcal {Z}}\subset {\mathcal {Z}}_i \subset {\mathcal {X}}$, $i=1,2$, with continuous and dense embeddings;

• (ii) $L,D_1$, and $D_2$ are closed linear operators in ${\mathcal {X}}$ with domains ${\mathcal {Z}}, {\mathcal {Z}}_1$, and ${\mathcal {Z}}_2$, respectively;

• (iii) the spectrum of L contains a pair of isolated purely imaginary eigenvalues $\pm {{{\,\textrm{i}\,}}} \ell _*$ with odd multiplicity;

• (iv) there exists an involution $R \in {\mathcal {L}}({\mathcal {X}})$ which anticommutes with L and $D_i$, $i=1,2$, i.e., the Eq. (4.3) is reversible.

Then, for each sufficiently small $\sigma >0$, Eq. (4.3) has a solution of the form $\exp (\sigma t) U_\sigma (z)$ with ${{\,\textrm{Re}\,}}\sigma > 0$ and $U_\sigma \in C^1(\mathbb {R}, {\mathcal {X}}) \cap C(\mathbb {R}, {\mathcal {Z}})$ a periodic function.

Remark 4.2

Theorem 1.3 [25] assumes that the linear operators $L, D_1,$ and $D_2$ have the same domain of definition, while we in Theorem 4.1 allow for different domains, just like in Theorem 2.1 of [17]. This is needed since the operators $D_1$ and $D_2$ in our application (and also in [25]) have different domains than L. Note in particular that the hypotheses imply that $D_1$ and $D_2$ are relatively bounded perturbations of L by Remarks 1.4 and 1.5, Chapter 4.1.1 [42].

This general result does not directly apply to the system (4.1)–(4.2) because the boundary condition (4.2) contains the extra term $y\eta _t$ which involves a derivative with respect to t. We proceed as in [25] and eliminate this term by an appropriate change of variables, similar to the one used for $L_\varepsilon$ in Sect. 3.

We replace the variable $\Phi$ in U by a new variable $\Theta = \Phi + {\theta }_{yt}$ where ${\theta }$ is the unique solution of the elliptic boundary value problem

$$\begin{aligned} \begin{aligned} -k_{\varepsilon }^2{\theta }_{xx}-\theta _{yy} + B_{l\varepsilon }(0,{\theta }_y)&= y\eta{} & {} \text {in}\, \Sigma , \\ {\theta }&= 0{} & {} \text {on}\, y = 0,1, \end{aligned} \end{aligned}$$

where we set $B_{l\varepsilon }(\eta ,\Phi )=B_{l\varepsilon }(U)$ because $B_{l\varepsilon }(U)$ only depends on $\eta$ and $\Phi$. In the boundary value problem for $\theta$, we regard t as a parameter and assume analytic dependence on t. Then the mapping defined by ${Q}(\eta , \omega , \Phi , \xi )^{{{\,\mathrm{\text {T}}\,}}}= (\eta , \omega , {\Theta }, \xi )^{{{\,\mathrm{\text {T}}\,}}}$ is a linear isomorphism on both ${{\mathcal {X}}}^0$ and ${{\mathcal {X}}}^1$. The transformed linearized problem (4.1)–(4.2) for $V= (\eta , \omega , {\Theta }, \xi )^{{{\,\mathrm{\text {T}}\,}}}$ is of the form

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}V}{\mathop {}\!\textrm{d}z} = D_1V_t + D_2 V_{tt} + L_\varepsilon V, \end{aligned}$$

(4.4)

with boundary conditions

$$\begin{aligned} {\Theta }_y = B_{l\varepsilon }(V) \quad \text {on}\, y=0,1. \end{aligned}$$

(4.5)

The two linear operators $D_1$ and $D_2$ are bounded in ${{\mathcal {X}}}^0$ and defined by

$$\begin{aligned}D_1 \begin{pmatrix} \eta \\ \omega \\ {\Theta } \\ \xi \end{pmatrix} = \begin{pmatrix} 0 \\ {\Theta }|_{y=1} + k_\varepsilon {\theta }_{xy}|_{y=1} - H_2(0, {\theta }_y)\\ \widehat{\theta }_y \\ -\eta - k_\varepsilon ^2(-\eta _{\varepsilon }\theta _{xy}+y\eta _{\varepsilon x}\theta _{yy})_x \end{pmatrix}, \quad D_2\begin{pmatrix} \eta \\ \omega \\ {\Theta } \\ \xi \end{pmatrix} = \begin{pmatrix} 0 \\ -{\theta }_y|_{y=1} \\ 0 \\ 0 \end{pmatrix},\end{aligned}$$

where $\widehat{\theta }$ is the unique solution of elliptic boundary value problem

$$\begin{aligned}\begin{aligned} -k_\varepsilon ^2 \widehat{\theta }_{xx}-\widehat{\theta }_{yy} + B_{l\varepsilon }(0, \widehat{\theta }_y)&= y \left( \frac{\omega }{\beta }+H_1(\omega , \xi )\right){} & {} \text {in}\, \Sigma , \\ {\hat{\theta }}&= 0{} & {} \text {on}\, y=0,1. \end{aligned}\end{aligned}$$

We use the system (4.4)–(4.5) and the result in Theorem 4.1 to prove the transverse linear instability of the periodic wave $U_\varepsilon$.

Proof of Theorem 1.1(ii)

We apply Theorem 4.1 to the Eq. (4.4) with Hilbert spaces ${\mathcal {X}}={\mathcal {Z}}_i ={{\mathcal {X}}}^0$, ${\mathcal {Z}}={{\mathcal {Y}}}^1_\varepsilon$, $i=1,2$, operators $D_1,D_2$ defined as above, and $L_{\varepsilon }$. Since $D_1$ and $D_2$ are bounded on ${\mathcal {X}}^0$, they are closed operators in ${\mathcal {X}}^0$. The first two hypotheses (i) and (ii) are satisfied. The spectral condition (iii) is verified by Theorem 3.1(i). The reverser is R defined in Sect. 2.1 and its anticommutativity with $D_1,D_2$ and L is preserved by the change of variables Q. Thus, Eq. (4.4) is reversible. Theorem 4.1 now gives the statement of Theorem 1.1(ii).$\square$

4.2 Dimension-Breaking Bifurcation

We look for three-dimensional steady solutions of the system (2.1)–(2.2) which bifurcate from the transversely unstable periodic wave $U_\varepsilon$. Taking

$$\begin{aligned}U(x, y, z) = U_\varepsilon (x, y) + {\widetilde{U}} (x, y, z), \quad {\widetilde{U}} =(\eta ,\omega ,\Phi ,\xi )^{{{\,\mathrm{\text {T}}\,}}},\end{aligned}$$

in (2.1)–(2.2) we obtain the equation

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}{\widetilde{U}} }{\mathop {}\!\textrm{d}z} = F(U_\varepsilon + {\widetilde{U}} ), \end{aligned}$$

(4.6)

together with the boundary conditions

$$\begin{aligned} \Phi _y = B(U_\varepsilon +{\widetilde{U}} )-B(U_\varepsilon ), \quad \text {on}\, y=0,1. \end{aligned}$$

(4.7)

The mappings F and B are defined on an open neighborhood M of $0 \in {\mathcal {X}}^1$ which is contained in the set

$$\begin{aligned} \{(\eta , \omega , \Phi , \xi )^{{{\,\mathrm{\text {T}}\,}}} \in {\mathcal {X}}^1 : \, |W(x)| < \beta , \eta (x) > -1 \ \text {for all}\, x \in \mathbb {R}\}, \end{aligned}$$

and are analytic. The periodic wave $U_\varepsilon$ belongs to M, for sufficiently small $\varepsilon$, and we look for bounded solutions ${\widetilde{U}}$ such that $U_\varepsilon +{\widetilde{U}}(z)\in M$, for all $z\in \mathbb {R}$. Our main tool for constructing such solutions is an infinite-dimensional version of the reversible Lyapunov center theorem (see [13] for the original finite-dimensional version and [3] for an infinite-dimensional extension under more general hypotheses).

Theorem 4.3

Let ${\mathcal {X}}$ and ${\mathcal {Y}}$ be real Banach spaces such that ${\mathcal {Y}}$ is continuously embedded in ${\mathcal {X}}$. Consider the evolutionary equation

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}U}{\mathop {}\!\textrm{d}t} = F(U), \end{aligned}$$

(4.8)

where $F \in {\mathscr {C}}_{\text {b}, \text {u}}^3({\mathcal {U}}, {\mathcal {X}})$ with ${\mathcal {U}} \subset {\mathcal {Y}}$ a neighborhood of $\,0$. Assume that $F(0)=0$ and that the following properties hold:

• (i) there exists an involution $R\in {\mathcal {L}}({\mathcal {X}})\cap {\mathcal {L}}({\mathcal {Y}})$ which anticommutes with F, i.e., the Eq. (4.8) is reversible;

• (ii) the linear operator $L {:}{=}{\mathop {}\!\textrm{d}}F[0]$ possesses a pair of simple eigenvalues $\pm {{{\,\textrm{i}\,}}}\omega _0$ with $\omega _0 > 0$;

• (iii) for each $n \in \mathbb {Z}{\setminus } \{-1, 1\}$, ${{{\,\textrm{i}\,}}}n\omega _0$ belongs to the resolvent set of L;

• (iv) there exists a positive constant C such that

  $$\begin{aligned} \Vert (L-{{{\,\textrm{i}\,}}}n\omega _0 {\mathbb {I}} )^{-1}\Vert _{{\mathcal {L}}({\mathcal {X}}, {\mathcal {X}})} \le \frac{C}{|n|},\quad \Vert (L-{{{\,\textrm{i}\,}}}n \omega _0 {\mathbb {I}})^{-1}\Vert _{{\mathcal {L}}({\mathcal {X}}, {\mathcal {Y}})} \le C, \end{aligned}$$

  (4.9)

  as $n\rightarrow \pm \infty$.

Then, there exists a $\delta _0>0$ and a ${\mathscr {C}}^1$-curve $\{(U(\delta ),\omega (\delta ))\}_{\delta \in (-\delta _0, \delta _0)}$ where $U(\delta )$ is a real periodic solution to (4.8) with period $2\pi /\omega (\delta )$. Furthermore, $(U(0),\omega (0))=(0,\omega _0)$.

Theorem 4.3 cannot be directly applied to (4.6)–(4.7) because the boundary condition is nonlinear. We make a nonlinear change of variables which transforms these nonlinear boundary conditions into linear boundary conditions. Similarly to our previous changes of variables from Sect. 3.1 and Sect. 4.1, we replace $\Phi$ by a new variable $\Theta = \Phi + \theta _y$ where $\theta$ is the unique solution of the elliptic boundary value problem

$$\begin{aligned}\begin{aligned} -k_\varepsilon ^2\theta _{xx}-\theta _{yy}+B_{l\varepsilon }(0,\theta _y)&=B(U){} & {} \text {in}\, \Sigma ,\\ \theta&= 0{} & {} \text {on}\, y=0,1, \end{aligned}\end{aligned}$$

and define $Q(\eta , \omega , \Phi , \xi )^{{{\,\mathrm{\text {T}}\,}}}=(\eta , \omega , \Theta , \xi )^{{{\,\mathrm{\text {T}}\,}}}$. Using the method from [22] (see also [25]) one can show that Q is a near-identity analytic diffeomorphism from a neighborhood $M_1$ of $0 \in {\mathcal {X}}^1$ onto possibly a different neighborhood $M_2$ of $0 \in {\mathcal {X}}^1$ and that for each $U \in M_1$, the linear operator $\mathop {}\!\textrm{d}Q[U]:{\mathcal {X}}^1 \rightarrow {\mathcal {X}}^1$ extends to an isomorphism $\widehat{\mathop {}\!\textrm{d}Q}[U]:{\mathcal {X}}^0 \rightarrow {\mathcal {X}}^0$ which depends analytically on U and the same holds for the inverse $\widehat{\mathop {}\!\textrm{d}Q}[U]^{-1}$. Then the Eq. (4.6) is transformed into

$$\begin{aligned} \frac{\mathop {}\!\textrm{d}V}{\mathop {}\!\textrm{d}z} = L_\varepsilon V + N(V), \end{aligned}$$

(4.10)

where

$$\begin{aligned} N{:}{=}\widetilde{F}-L_\varepsilon ,\quad \widetilde{F}(V) = \widehat{\mathop {}\!\textrm{d}Q}[Q^{-1}(V)](F(U_\varepsilon +Q^{-1}(V))), \end{aligned}$$

and the boundary condition (4.7) becomes linear,

$$\begin{aligned} \Theta _y = B_{l\varepsilon }(V)\quad \text {on}\, y=0,1. \end{aligned}$$

In particular, we recover the linear operator $L_\varepsilon$ studied in Sect. 3, and we can apply the Lyapunov center theorem to conclude.

Proof of Theorem 1.1(iii)

The Eq. (4.10) is a dynamical system in the phase space ${\mathcal {X}}^0$ with vector field defined in a neighborhood of 0 in ${\mathcal {Y}}^1_\varepsilon$. Because the change of variables Q preserves reversibility and reflection symmetries, the vector field in (4.10) anti-commutes with the reverser R and commutes with the reflection S. Consequently, the system (4.10) is reversible with reverser R and the reflection symmetry S implies that the subspace ${\mathcal {X}}_+^0$ given in (3.4) is invariant. Taking ${\mathcal {X}}={\mathcal {X}}^0_+$ and ${\mathcal {Y}}={\mathcal {Y}}^1_\varepsilon \cap {\mathcal {X}}^0_+$ the results in Theorem 3.1 imply that the hypotheses of Theorem 4.3 hold, for $\varepsilon$ sufficiently small. This proves Theorem 1.1(iii). $\square$

5 Parameter Region II

The analysis done for $(\alpha ,\beta )$ in Region I can be easily transferred to the parameter Region II. However, the final result is different because the linear dispersion relation (1.3) possesses two positive roots for $(\alpha ,\beta )$ in this parameter region. We point out the differences and then state the main result for this parameter region.

Denote by $k_{*,1}$ and $k_{*,2}$ the two positive roots of the dispersion relation. Take $k_{*,1}<k_{*,2}$ and assume that $k_{*,2}/k_{*,1} \notin \mathbb {N}$.

First, the existence of two-dimensional periodic waves is proved in the same way, with the difference that we now find two geometrically distinct families of two-dimensional periodic waves $\{(\eta _{\varepsilon , 1}(X), \phi _{\varepsilon , 1}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ and $\{(\eta _{\varepsilon , 2}(X), \phi _{\varepsilon , 2}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)}$ with wavenumbers $k_{\varepsilon ,1} = k_{*,1} + \mathcal {O}(\varepsilon ^2)$ and $k_{\varepsilon ,2}=k_{*,2} + \mathcal {O}(\varepsilon ^2)$, respectively. The expansions (2.5) remain valid with $k_*$ replaced by $k_{*,1}$ for the first family and by $k_{*,2}$ for the second family, and this is also the case for all other symbolic computations.

Next, the analysis of the linear operator $L_\varepsilon$ given in Sect. 3 stays the same until the last step of the proof of Theorem 3.1 which consists in showing that $m_{21}^{(2)}$ is negative. The formula for $m_{21}^{(2)}$ is the same, but the result is different for the first family of periodic waves. The analysis in Appendix C gives the conclusion that $m_{21}^{(2)}$ is negative for the second family of periodic waves, whereas for the first family of periodic waves it is negative only when $2k_{*,1} > k_{*,2}$. This condition is satisfied if and only if $(\alpha , \beta )$ belongs to the open region between $\Gamma _2$ and $\Gamma$ in Fig. 1.

Consequently, the two-dimensional periodic waves $(\eta _{\varepsilon , 2}(X), \phi _{\varepsilon , 2}(X,Y))$ are transversely linearly unstable, whereas the periodic waves $(\eta _{\varepsilon , 1}(X), \phi _{\varepsilon , 1}(X, Y))$ are transversely linearly unstable if $(\alpha , \beta )$ lies between $\Gamma _2$ and $\Gamma$. Notice that our approach does not allow us to conclude on stability because the general criterion in Theorem 4.1 only provides sufficient conditions for instability. Finally, the dimension-breaking result holds for all transversely linearly unstable waves.

We summarize these results in the following theorem.

Theorem 5.1

(Region II) Fix $(\alpha ,\beta )$ in Region II and let $k_{*,1}, k_{*,2}$ be the two positive roots of the dispersion relation (1.3). Assume that $k_{*,1}< k_{*,2}$ and $k_{*,2}/k_{*,1} \notin \mathbb {N}$. Denote by $\Gamma _2$ the $(\alpha ,\beta )$-parameter curve for which $2k_{*,1} = k_{*,2}$.

• (i) (Existence) There exist $\varepsilon _0 > 0$ and two geometrically distinct families of two-dimensional steady periodic solutions

  $$\begin{aligned}\begin{aligned}&\{(\eta _{\varepsilon ,1}(X), \phi _{\varepsilon ,1}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)} \quad \text {and} \quad \{(\eta _{\varepsilon ,2}(X), \phi _{\varepsilon , 2}(X, Y))\}_{\varepsilon \in (-\varepsilon _0, \varepsilon _0)} \end{aligned}\end{aligned}$$

  to the Eqs. (1.1)–(1.2), such that $(\eta _{0,i},\phi _{0,i}) =(0, 0)$ and $(\eta _{\varepsilon , i}, \phi _{\varepsilon , i})$ are periodic in X with wavenumbers $k_{\varepsilon , i} = k_{*,i}+ \mathcal {O}(\varepsilon ^2)$ for $i = 1,2$.

• (ii) (Transverse instability) There exists $\varepsilon _1>0$ such that for each $\varepsilon \in (-\varepsilon _1,\varepsilon _1)$ the periodic solution $(\eta _{\varepsilon , 2}, \phi _{\varepsilon , 2})$ is transversely linearly unstable. The solution $(\eta _{\varepsilon , 1}, \phi _{\varepsilon , 1})$ is transversely linearly unstable if $\,2k_{*,1} > k_{*,2}$, which occurs for $(\alpha , \beta )$ in the open region between the curves $\Gamma _2$ and $\Gamma$.

• (iii) (Dimension-breaking bifurcation) There exists $\varepsilon _2>0$ such that for each transversely linearly unstable wave $(\eta _{\varepsilon , i}, \phi _{\varepsilon ,i})$ with $\varepsilon \in (-\varepsilon _2,\varepsilon _2)$, $i=1,2$, there exist $\delta _\varepsilon >0$, $\ell _{\varepsilon ,i}^*>0$, and a family of three-dimensional doubly periodic waves $\{(\eta _{\varepsilon ,i}^\delta (X, z),$ $\phi _{\varepsilon ,i}^\delta (X, Y, z))\}_{\delta \in (-\delta _{\varepsilon ,i}, \delta _{\varepsilon ,i})}$, with wavenumber $k_{*,i}$ in X and wavenumber $\ell _\delta =\ell _{\varepsilon ,i}^*+\mathcal {O}(\delta ^2)$ in z, bifurcating from the periodic solution $(\eta _{\varepsilon , i}, \phi _{\varepsilon ,i})$.

Appendices

A Existence and Expansion of the Two-Dimensional Periodic Waves

For this discussion it is more convenient to use the original system (1.1)–(1.2) instead of the dynamical system (2.1). Restricting to two-dimensional steady solutions, we make the change of variables

$$\begin{aligned} X=k^{-1} x,\quad Y=y(1+\eta (X)),\quad \eta (X)={\widetilde{\eta }}(x),\quad \phi (X,Y) = {\widetilde{\Phi }}(x, y). \end{aligned}$$

Dropping the tildes we obtain the equations

$$\begin{aligned}{} & {} k^2 \Phi _{xx} + \frac{1}{(1+\eta )^2}\Phi _{yy} - 2k^2 \frac{y \eta _x}{1+\eta } \Phi _{xy} \nonumber \\{} & {} \quad + k^2 \left( \frac{y \eta _x}{1+\eta }\right) ^2 \Phi _{yy} + k^2 \left( \frac{2y\eta _x^2}{(1+\eta )^2}-\frac{y\eta _{xx}}{1+\eta }\right) \Phi _y = 0, \end{aligned}$$

(A.1)

for $0< y < 1$ with boundary conditions

$$\begin{aligned} \begin{aligned} \Phi _y&= 0{} & {} \text {on}\, y=0,&\\ \Phi _y&= (1+\eta )(-k \eta _x+ k^2\eta _x \Phi _x)- k ^2 \eta _x^2 \Phi _y{} & {} \text {on}\, y=1,&\\ \alpha \eta&- k \left( \Phi _x - \frac{\eta _x}{1+\eta }\Phi _y \right) - \beta k^2 \left( \frac{\eta _x}{(1+k^2\eta _x^2)^{1/2}}\right) _x{} & {} {}&\\&+ \frac{1}{2}\left( k^2\left( \Phi _x - \frac{\eta _x}{1+\eta } \Phi _y\right) ^2 + \frac{\Phi _y^2}{(1+\eta )^2}\right) = 0&\quad&\text {on}\, y=1.&\end{aligned} \end{aligned}$$

(A.2)

Our main tool for showing the existence of nontrivial solutions to this system is an analytic version of the classical Crandall–Rabinowitz theorem (see [8, Theorem 8.3.1]).

Theorem A.1

Let ${\mathcal {X}}$ and ${\mathcal {Y}}$ be Banach spaces, ${\mathcal {U}} \subset {\mathcal {X}} \times \mathbb {R}$ an open subset, and $F=F(w, \lambda )$ an analytic map from $\mathcal U$ to $\mathcal Y$ with $F(0,\lambda )=0$ for all $(0, \lambda ) \in {\mathcal {U}}$. Suppose that for some $(0, \lambda _*) \in {\mathcal {U}}$ there exists a $w_* \ne 0 \in {\mathcal {X}}$, such that

$$\begin{aligned}&\mathop {}\!\textrm{d}_w F[0,\lambda _*] \,\, \text {is a Fredholm operator of index zero}, \\&\ker \mathop {}\!\textrm{d}_w F[0,\lambda _*]= {{\,\textrm{span}\,}}\{w_*\}, \\&\mathop {}\!\textrm{d}^2_{\lambda w} F[0,\lambda _*]w_* \not \in {{\,\textrm{ran}\,}}\mathop {}\!\textrm{d}_w F[0, \lambda _*]. \end{aligned}$$

Then there exists a local bifurcation curve $\{(w(\varepsilon ),\lambda (\varepsilon )): |\varepsilon | < \varepsilon _0 \}$, such that $(w(0),\lambda (0))=(0,\lambda _*)$ and

$$\begin{aligned} \{(w,\lambda )\in \mathcal {V}: w\ne 0,\, F(w,\lambda )=0\} =\{(w(\varepsilon ),\lambda (\varepsilon ))\in \mathcal {U}: 0< |\varepsilon | < \varepsilon _0 \} \end{aligned}$$

for some neighbourhood $\mathcal {V}$ of $(0,\lambda _*)$ in ${\mathcal {U}}$. The functions $w(\varepsilon )$ and $\lambda (\varepsilon )$ are analytic and $w'(0)=w^*$.

The existence of the scaled family of periodic waves $({\widetilde{\eta }}_\varepsilon (x),{\widetilde{\Phi }}_\varepsilon (x,y))$ with $k=k_\varepsilon$ follows by applying Theorem A.1 to Eqs. (A.1), (A.2), taking F(w, k) to be the nonlinear operator defined by (A.1) and the boundary conditions at $\{y=1\}$, with³

$$\begin{aligned} {\mathcal {X}}&= \{w=(\Phi , \eta ) \in H^{2}_{{{\,\textrm{per}\,}}, {{\,\textrm{odd}\,}}}(\Sigma )\times H^{2}_{{{\,\textrm{per}\,}}, {{\,\textrm{even}\,}}}({\mathbb {S}}): \Phi _y|_{y=0}=0\},\\ {\mathcal {U}}&=\{ (\Phi , \eta , k) \in \mathcal {X} \times \mathbb {R}: \min \eta>-1, \, k>0\}\quad \hbox {and}\\ {\mathcal {Y}}&=\{(f, g, h)\in L^2_{{{\,\textrm{per}\,}}, {{\,\textrm{odd}\,}}}(\Sigma )\times H^{1/2}_{{{\,\textrm{per}\,}}, {{\,\textrm{odd}\,}}}({\mathbb {S}}) \times L^2_{{{\,\textrm{per}\,}}, {{\,\textrm{even}\,}}}({\mathbb {S}})\}. \end{aligned}$$

In particular, the linearised system $\mathop {}\!\textrm{d}_w F[0, k]w=(f,g,h)$ is given by

$$\begin{aligned} \begin{aligned}&k^2 \Phi _{xx} + \Phi _{yy} = f&\quad&\text {for}\, 0< y < 1,&\\&\Phi _{y}= 0&\quad&\text {on}\, y=0,&\\&\Phi _{y}+ k\eta _{x}=g&\quad&\text {on}\, y=1,&\\&\alpha \eta - k \Phi _{x} - \beta k^2 \eta _{xx} = h&\quad&\text {on}\, y=1.&\end{aligned} \end{aligned}$$

(A.3)

Since $\mathop {}\!\textrm{d}_w F[0, k]$ has compact resolvent, it is Fredholm with index 0. Taking Fourier series in x,

$$\begin{aligned}{} & {} \eta (x) = \sum _{n=0}^\infty \eta _{n} \cos (nx), \quad \Phi (x,y) = \sum _{n=1}^\infty \phi _{n}(y)\sin (nx),\\{} & {} f(x,y)= \sum _{n=1}^\infty f_{n}(y)\sin (nx),\quad g(x)= \sum _{n=1}^\infty g_{n}\sin (nx),\quad h(x)= \sum _{n=0}^\infty h_{n}\cos (nx), \end{aligned}$$

the system decouples into

$$\begin{aligned} \begin{aligned} -(kn)^2\phi _{n}(y) + \phi _{n}''(y)&=f_n&\quad&\text {for}\, 0< y < 1,&\\ \phi _{n}'(0)&=0,{} & {} {}&\\ \phi _{n}'(1)-kn\eta _{n}&=g_n,{} & {} {}&\\ (\alpha + \beta (kn)^2)\eta _{n}-kn\phi _{n}(1)&=h_n,{} & {} &\end{aligned} \end{aligned}$$

(A.4)

and we obtain nontrivial solutions of the homogeneous problem precisely when $\mathcal D(nk)=0$ where $\mathcal D$ is the linear dispersion relation in (1.3). For $(\alpha , \beta )$ in Region I, we find that the kernel is one-dimensional for $k=k_*$, and that it is spanned by the first Fourier mode $\eta _*(x)=\sinh (k_*)\cos (x)$, $\Phi _*(x,y)=\cosh (k_*y)\sin (x)$. The corresponding range space is given by $(f,g,h)\in {\mathcal {Y}}$ such that

$$\begin{aligned} g_1 \cosh (k_*) + h_1\sinh (k_*) = \int _0^1 f_1(y)\cosh (k_*y)\mathop {}\!\textrm{d}y. \end{aligned}$$

(A.5)

Indeed, integrating twice by parts we find

$$\begin{aligned}\begin{aligned}&\int _0^1 f_1(y)\cosh (k_*y)\mathop {}\!\textrm{d}y - h_1 \sinh (k_*)\\&\quad = \int _0^1 \left( k_*^2\phi _{1}(y)-\phi _{1}''(y)\right) \cosh (k_*y) \mathop {}\!\textrm{d}y - h_1\sinh (k_*)\\&\quad = \phi _{1}'(1)\cosh (k_*)-k_*\phi _{1}(1)\sinh (k_*) - h_1\sinh (k_*)\\&\quad = (g_1+k_*\eta _{1})\cosh (k_*) - (\alpha + \beta k_*^2)\eta _{1}\sinh (k_*) \\&\quad = g_1 \cosh (k_*), \end{aligned}\end{aligned}$$

where we have also used the linear dispersion relation ${\mathcal {D}}(k_*)=0$. A straightforward calculation shows that the transversality condition $\mathop {}\!\textrm{d}^2_{kw} F[0, k_*]w_* \not \in {{\,\textrm{ran}\,}}\mathop {}\!\textrm{d}_w F[0, k_*]$ is equivalent to ${\mathcal {D}}'(k_*) \ne 0$, which is known to be true for all positive roots of the dispersion relation in Region I (see the discussion in the introduction). As noted in Sect. 2.2, the wavenumber $k_\varepsilon$ is even in $\varepsilon$, and therefore $k_\varepsilon =k_*+\mathcal {O}(\varepsilon ^2)$. This concludes the proof of Theorem 1.1(i). Theorem 5.1(i) for Region II follows in the same way, replacing $k_*$ by $k_{*,1}$ and $k_{*,2}$ and using the assumption that $k_{*,2}/k_{*,1} \notin \mathbb {N}$. In the computations below we focus on Region I, but the formulas for Region II are simply obtained by replacing $k_*$ by $k_{*,i}$, $i=1,2$.

The power series expansion of the solutions $({\widetilde{\eta }}_\varepsilon (x),{\widetilde{\Phi }}_\varepsilon (x,y))$ can be obtained by inserting the expansions (2.5) into Eqs. (A.1)–(A.2) and equating equal powers of $\varepsilon$. At order $\mathcal {O}(\varepsilon )$ we simply obtain the system (A.3), and thus $(\eta _1, \Phi _1)=(\eta _*, \Phi _*)$ as stated in (2.7).

Next, at order $\mathcal {O}(\varepsilon ^2)$ we obtain the equations for $\eta _2$ and $\Phi _2$,

$$\begin{aligned} \begin{aligned}&k_*^2 \Phi _{2xx} + \Phi _{2yy} =2\eta _1 \Phi _{1yy} + 2k_*^2 y \eta _{1x} \Phi _{1xy} + k^2_* y \eta _{1xx} \Phi _{1y}&\quad&\text {for}\, 0< y < 1,&\\&\Phi _{2y} =0&\quad&\text {on}\, y=0,&\\&\Phi _{2y}+k_*\eta _{2x}= k^2_* \eta _{1x} \Phi _{1x}-k_*\eta _1 \eta _{1x}{} & {} \text {on}\, y=1,&\\&\alpha \eta _2 - \beta k_*^2 \eta _{2xx} - k_* \Phi _{2x} = {-}k_* \eta _{1x} \Phi _{1y} - \tfrac{1}{2} (k_*^2 \Phi _{1x}^2 + \Phi _{1y}^2)&\quad&\text {on}\, y=1.&\end{aligned} \end{aligned}$$

(A.6)

Inserting the explicit formulas for $\eta _1$ and $\Phi _1$ we obtain

$$\begin{aligned} \begin{aligned}&k_*^2 \Phi _{2xx} + \Phi _{2yy} =k_*^2 \sinh (k_*)\sin (2x)\left( \cosh (k_*y)-\tfrac{3}{2}k_* y \sinh (k_*y)\right){} & {} \text {for}\, 0< y < 1,&\\&\Phi _{2y} =0{} & {} \text {on}\, y=0,&\\&\Phi _{2y}+k_*\eta _{2x} = \left( \tfrac{k_*}{2}\sinh ^2(k_*)-\tfrac{k_*^2}{2} \sinh (k_*)\cosh (k_*)\right) \sin (2x){} & {} \text {on}\, y=1,&\\&\alpha \eta _2 - \beta k_*^2 \eta _{2xx} - k_* \Phi _{2x} = {-}\tfrac{k_*^2}{4}\cosh (2k_*)\cos (2x)-\tfrac{k_*^2}{4}{} & {} \text {on}\, y=1.&\end{aligned}\nonumber \\ \end{aligned}$$

(A.7)

Observing that the right-hand side only involves the Fourier modes 0 and 2, we find the formulas for $\eta _2$ and $\Phi _2$ in (2.7).

Finally, the coefficient $k_2$ is determined from the expansion at order ${\mathcal {O}}(\varepsilon ^3)$,

$$\begin{aligned} k_*^2 \Phi _{3xx} + \Phi _{3yy}&=-2k_*k_2\Phi _{1xx}-3\eta _1^2 \Phi _{1yy} + 2\eta _2 \Phi _{1yy} + 2\eta _1 \Phi _{2yy} \\&\qquad +2k_*^2y(\eta _{2x}\Phi _{1xy}+\eta _{1x}\Phi _{2xy}-\eta _{1x}\eta _1\Phi _{1xy}-\eta _{1x}^2\Phi _{1y}) \\&\qquad -k_*^2y^2\eta _{1x}^2 \Phi _{1yy} + yk_*^2(\eta _{1xx}\Phi _{2y}+\eta _{2xx}\Phi _{1y}-\eta _{1xx}\eta _1\Phi _{1y}) \end{aligned}$$

for $0< y < 1$, and

$$\begin{aligned} \begin{aligned}&\Phi _{3y} =0{} & {} \text {on}\, y=0,&\\&\Phi _{3y}+k_*\eta _{3x} = -k_*(\eta _1\eta _{2x}+\eta _{1x}\eta _2) - k_2\eta _{1x}{} & {} {}&\\&\quad \qquad \qquad \quad \qquad + k_*^2(\eta _{1x}\Phi _{2x}+\eta _{2x}\Phi _{1x}+\eta _1\eta _{1x}\Phi _{1x})-k_*^2\eta _{1x}^2\Phi _{1y}{} & {} \text {on}\, y=1,&\\&\alpha \eta _3 - \beta k_*^2 \eta _{3xx} - k_* \Phi _{3x} =k_2\Phi _{1x}-k_*(\eta _{1x}\Phi _{2y}+\eta _{2x}\Phi _{1y}-\eta _{1x}\eta _1 \Phi _{1y}){} & {} {}&\\&\qquad \qquad \quad \qquad \qquad +2\beta k_*k_2 \eta _{1xx} - \frac{3}{2}\beta k_*^4 \eta _{1xx} \eta ^2_{1x}{} & {} {}&\\&\qquad \qquad \qquad \qquad \quad -k_*^2(\Phi _{1x}\Phi _{2x}-\eta _{1x}\Phi _{1x}\Phi _{1y})-\Phi _{1y}\Phi _{2y}+\eta _1\Phi _{1y}^2{} & {} \text {on}\, y=1.&\\ \end{aligned} \end{aligned}$$

The right hand sides involving only Fourier modes 1 and 3, we write

$$\begin{aligned}\eta _3(x)=\eta _{31}\cos (x)+ \eta _{33}\cos (3x), \quad \Phi _{3}(x,y)=\phi _{31}(y)\sin (x)+\phi _{33}(y)\sin (3x),\end{aligned}$$

and the resulting systems for $(\eta _{31},\phi _{31})$ and $(\eta _{33},\phi _{33})$ are decoupled. The coefficient $k_2$ only appears in the terms with Fourier mode 1 so that it is enough to solve the equations for $(\eta _{31},\phi _{31})$. These equations are of the form (A.4) with $n=1$, $(\eta _{1},\phi _{1})$ replaced by $(\eta _{31},\phi _{31})$ and $(f_1, g_1, h_1)$ replaced by $(f_{31}, g_{31}, h_{31})$, where, after computations, we find the explicit formulas

$$\begin{aligned} \begin{aligned} f_{31}(y) =&\left( \frac{k_*^2\sinh ^2(k_*)}{4}-\frac{k_*^4}{2\alpha }-\frac{k_*^3}{4}\left( c(k_*)+1\right) \sinh (2k_*)+2k_*k_2\right) \cosh (k_*y) \\&+\frac{k_*^3\sinh ^2(k_*)}{2}y\sinh (k_*y) + k_*^3 c(k_*)\sinh (k_*)\cosh (2k_*y) \\&+ \frac{3k_*^4c(k_*)\sinh (k_*)}{4}y \sinh (2k_*y), \end{aligned}\end{aligned}$$

and

$$\begin{aligned}\begin{aligned} g_{31} =&-\frac{k_*^2c(k_*)}{16}\cosh (k_*) + \left( -\frac{k_*^3c(k_*)}{4}+\frac{k_*^3}{16}-\frac{k_*^3}{4\alpha }+k_2\right) \sinh (k_*)\\&+ \frac{k_*^2c(k_*)}{16}\cosh (3k_*) -\frac{3}{16}k_*^3 \sinh (3k_*),\\ h_{31} =&\left( \frac{k_*^3}{16}-\frac{k_*^3c(k_*)}{4} +k_2\right) \cosh (k_*) - \left( \frac{9}{32}\beta k_*^4 + 2\beta k_* k_2\right) \sinh (k_*) \\&-\frac{k_*^3}{16}\cosh (3k_*)+\frac{3}{32}\beta k_*^4 \sinh (3k_*). \end{aligned}\end{aligned}$$

Substituting these formulas into the solvability condition (A.5) and solving for $k_2$ we obtain the formula (2.6).

B Computation of the Coefficient $\varvec{m_{21}^{(2)}}$

We prove the equality (3.7) which connects the coefficient ${m_{21}^{(2)}}$ with the coefficient $k_2$ in the expansion of the wavenumber $k_\varepsilon$ of the periodic wave.

We emphasize the dependence on k of the vector field F in (2.1) by writing F(U, k) and similarly for B in the boundary conditions (2.2) we write B(U, k). Setting

$$\begin{aligned} {\widetilde{B}}(U,k)=B(U,k) - \Phi _{y}, \end{aligned}$$

the two-dimensional periodic wave $U_\varepsilon$ given in (2.10) satisfies

$$\begin{aligned} {\left\{ \begin{array}{ll}F(U_\varepsilon , k_\varepsilon ) = 0, \quad y\in (0,1) \\ \widetilde{B}(U_\varepsilon ,k_\varepsilon ) = 0, \quad y = 0,1. \end{array}\right. } \end{aligned}$$

(B.1)

The linear operator $L_\varepsilon$ is equal to ${{{\,\textrm{D}\,}}}_U F[U_\varepsilon , k_\varepsilon ]$ with boundary conditions ${{{\,\textrm{D}\,}}}_U \widetilde{B}[U_\varepsilon , k_\varepsilon ]= 0.$ To determine $m_{21}^{(2)}$, we study the problem

$$\begin{aligned} L_\varepsilon \zeta _+(\varepsilon ) = m_{11}(\varepsilon )\zeta _+(\varepsilon )+m_{21}(\varepsilon )\psi _+(\varepsilon ),\end{aligned}$$

(B.2)

with the boundary conditions

$$\begin{aligned} {{{\,\textrm{D}\,}}}_U\widetilde{B}(U_\varepsilon , k_\varepsilon )\zeta _+(\varepsilon ) = 0.\end{aligned}$$

(B.3)

We will make a connection between (B.1) and (B.2)–(B.3) using the expansions in $\varepsilon$ of $F, {\widetilde{B}},$ $U_\varepsilon$, $k_\varepsilon$, $\zeta _+(\varepsilon )$, $\psi _+(\varepsilon )$, $m_{11}(\varepsilon )$ and $m_{21}(\varepsilon )$.

For the system (B.1), we write

$$\begin{aligned} \begin{aligned}&U_\varepsilon = \varepsilon U_1 + \varepsilon ^2 U_2 + \varepsilon ^3U_3 + {\mathcal {O}}(\varepsilon ^4),\\&k_\varepsilon = k_* + \varepsilon ^2 k_2 + {\mathcal {O}}(\varepsilon ^4),\\ \end{aligned} \end{aligned}$$

and take the Taylor expansion of F(U, k) at $(0,k_*)$,

$$\begin{aligned} \begin{aligned} F(U, k)&= {{{\,\textrm{D}\,}}}_U F[0, k_*] U + \frac{1}{2}{{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U,U) \\&\quad + {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U, k-k_*) + \frac{1}{6}{{{\,\textrm{D}\,}}}_{UUU}^3F[0,k_*](U,U,U) \\&\quad + {\mathcal {O}}\left( |k-k_*|^2\Vert U\Vert _{{\mathcal {X}}^1}+|k-k_*|\Vert U\Vert _{{\mathcal {X}}^1}^2 + \Vert U\Vert _{{\mathcal {X}}^1}^4\right) , \end{aligned} \end{aligned}$$

where all derivatives ${{{\,\textrm{D}\,}}}^{(q)}_kF[0,k_*] = 0$ because $F(0,k)=0$ for all k. A similar Taylor expansion can be written for the nonlinear boundary condition ${\widetilde{B}}$. In particular $L_0 = {{{\,\textrm{D}\,}}}_UF[0,k_*]$ with boundary conditions $\widetilde{B}_0 U = {{{\,\textrm{D}\,}}}_U\widetilde{B}[0,k_*]U = 0$ on $y=0,1$. Inserting these expansions into (B.1), at order $\mathcal {O}(\varepsilon )$, we find

$$\begin{aligned} {\left\{ \begin{array}{ll} L_0U_1 =0, \quad y\in (0,1),\\ \widetilde{B}_0U_1 =0, \quad y=0,1. \end{array}\right. }\end{aligned}$$

(B.4)

Thus $U_1$ belongs to the kernel of $L_0$ and we choose $U_1=\zeta _+$ which is in agreement with the expansion of $U_\varepsilon$ given by (2.5). At order $\mathcal {O}(\varepsilon ^2)$, we obtain the system

$$\begin{aligned} {\left\{ \begin{array}{ll}L_0 U_2 = - \frac{1}{2}{{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U_1,U_1), \quad y\in (0,1),\\ \widetilde{B}_0U_2 = -\frac{1}{2}{{{\,\textrm{D}\,}}}_{UU}^2\widetilde{B}[0,k_*](U_1,U_1), \quad y = 0,1, \end{array}\right. }\end{aligned}$$

(B.5)

and at order $\mathcal {O}(\varepsilon ^3)$ we find

$$\begin{aligned} {\left\{ \begin{array}{ll} L_0U_3 = -{{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U_1,U_2)-{{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1,k_2) \\ \qquad \qquad \qquad - \frac{1}{6}{{{\,\textrm{D}\,}}}_{UUU}^3F[0,k_*](U_1,U_1,U_1), \qquad \qquad \quad y\in (0,1),\\ \widetilde{B}_0U_3 = -{{{\,\textrm{D}\,}}}_{UU}^2\widetilde{B}[0,k_*](U_1,U_2)-{{{\,\textrm{D}\,}}}_{Uk}^2\widetilde{B}[0,k_*](U_1,k_2) \\ \qquad \qquad \qquad - \frac{1}{6}{{{\,\textrm{D}\,}}}_{UUU}^3\widetilde{B}[0,k_*](U_1,U_1,U_1), \qquad \qquad \quad y=0,1. \end{array}\right. } \end{aligned}$$

(B.6)

The nonhomogeneous systems (B.5) and (B.6) have each a unique solution $U_2$ and $U_3$, respectively, up to an element in the kernel of $L_0$. Furthermore, $U_2=(\eta _2,0,\Phi _2,0)^{\text {T}}$ in agreement with the expansion of $U_\varepsilon$ given by (2.5).

Similarly, for the spectral problem (B.2)–(B.3) we take

$$\begin{aligned} \begin{aligned}&\zeta _+(\varepsilon )\, = \zeta _+ \, \,+ \varepsilon \zeta _{+,1} \, + \varepsilon ^2 \zeta _{+,2} \, + \mathcal {O}(\varepsilon ^3),\\&\psi _+(\varepsilon ) = \psi _+ + \varepsilon \psi _{+,1} + \varepsilon ^2 \psi _{+,2} + \mathcal {O}(\varepsilon ^3),\\&m_{11}(\varepsilon ) = \varepsilon ^2 m_{11}^{(2)} + \mathcal {O}(\varepsilon ^4),\\&m_{21}(\varepsilon ) = \varepsilon ^2 m_{21}^{(2)} + \mathcal {O}(\varepsilon ^4), \end{aligned} \end{aligned}$$

the Taylor expansion

$$\begin{aligned} \begin{aligned} L_\varepsilon&= {{{\,\textrm{D}\,}}}_UF[U_\varepsilon , k_\varepsilon ](\, \cdot \,) \\&= {{{\,\textrm{D}\,}}}_U F[0,k_*](\, \cdot \,) + {{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U_\varepsilon , \, \cdot \,) + {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](\, \cdot , k_\varepsilon -k_*) \\&\quad + \frac{1}{2}{{{\,\textrm{D}\,}}}_{UUU}^3F[0,k_*](U_\varepsilon , U_\varepsilon , \, \cdot \,) + {\mathcal {O}}\Big (|k_\varepsilon -k_*|^2+|k_\varepsilon -k_*|\Vert U_\varepsilon \Vert _{{\mathcal {X}}^1} + \Vert U_\varepsilon \Vert _{{\mathcal {X}}^1}^3\Big ), \end{aligned} \end{aligned}$$

and a similar Taylor expansion can be written for ${{{\,\textrm{D}\,}}}_U\widetilde{B}[U_\varepsilon ,k_\varepsilon ]$. Inserting these expansions in Eqs. (B.2)–(B.3), we find at order $\mathcal {O}(1)$ the system

$$\begin{aligned}{\left\{ \begin{array}{ll} L_0\zeta _+ = 0, \quad y \in (0,1),\\ \widetilde{B}_0\zeta _+ = 0, \quad y = 0,1. \end{array}\right. }\end{aligned}$$

This is precisely the system (B.4) and we may choose $U_1=\zeta _+$. Next, at order $\mathcal {O}(\varepsilon )$, we find

$$\begin{aligned} {\left\{ \begin{array}{ll} L_0 \zeta _{+,1} = -{{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U_1, U_1), \quad y \in (0,1),\\ \widetilde{B}_0 \zeta _{+,1} = -{{{\,\textrm{D}\,}}}_{UU}^2\widetilde{B}[0,k_*](U_1, U_1), \quad y = 0,1. \end{array}\right. } \end{aligned}$$

Comparing to (B.5), we choose $\zeta _{+,1} = 2U_2$. Finally, at order $\mathcal {O}(\varepsilon ^2)$, we have the system

$$\begin{aligned} {\left\{ \begin{array}{ll} L_0 \zeta _{+,2} = -3{{{\,\textrm{D}\,}}}_{UU}^2F[0,k_*](U_1, U_2) - {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1, k_2)\\ \qquad \qquad \qquad - \frac{1}{2}{{{\,\textrm{D}\,}}}_{UUU}^3F[0,k_*](U_1,U_1,U_1) + m_{11}^{(2)}\zeta _+ + m_{21}^{(2)}\psi _+, \quad y \in (0,1),\\ \widetilde{B}_0 \zeta _{+,2} = -3{{{\,\textrm{D}\,}}}_{UU}^2\widetilde{B}[0,k_*](U_1, U_2) - {{{\,\textrm{D}\,}}}_{Uk}^2\widetilde{B}[0,k_*](U_1, k_2)\\ \qquad \qquad \qquad -\frac{1}{2}{{{\,\textrm{D}\,}}}_{UUU}^3\widetilde{B}[0,k_*](U_1,U_1,U_1), y = 0,1. \end{array}\right. } \end{aligned}$$

Subtracting the system (B.6) from the one above we obtain the system

$$\begin{aligned} {\left\{ \begin{array}{ll} L_0 (\zeta _{+,2}-3U_3) = 2 {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1, k_2)+ m_{11}^{(2)}\zeta _+ + m_{21}^{(2)}\psi _+, &{} y \in (0,1),~\\ \widetilde{B}_0 (\zeta _{+,2}-3U_3) = 2 {{{\,\textrm{D}\,}}}_{Uk}^2\widetilde{B}[0,k_*](U_1, k_2), &{} y = 0,1, \end{array}\right. }\qquad \end{aligned}$$

(B.7)

in which $k_2$ and $m_{21}^{(2)}$ appear in the right hand sides of the two equalities.

The connection between $k_2$ and $m_{21}^{(2)}$ is obtained from the solvability condition for the system (B.7). A convenient way to find this solvability condition is by taking the scalar product of the first equation in (B.7) with the dual vector

$$\begin{aligned} \zeta _+^*=\begin{pmatrix} 0 \\ \sinh (k_*)\cos (x) \\ 0 \\ \cosh (k_*y)\sin (x)\end{pmatrix}, \end{aligned}$$

(B.8)

which belongs to the kernel of the $L^2$-adjoint $L_0^*$ of the linear operator $L_0$. This leads to the equality

$$\begin{aligned} m_{21}^{(2)} = \frac{\langle L_0(\zeta _{+,2}-3U_3),\zeta _+^*\rangle }{\langle \psi _+, \zeta _+^*\rangle }-2\frac{\langle {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1,k_2), \zeta _+^*\rangle }{\langle \psi _+,\zeta _+^*\rangle }. \end{aligned}$$

(B.9)

It remains to explicitly compute the right hand side of this equality.

First, we compute the denominator

$$\begin{aligned}\begin{aligned}\langle \psi _+, \zeta _+^* \rangle&= \int _0^{2\pi } \int _0^1 \cosh ^2(k_*y)\sin ^2(x)\mathop {}\!\textrm{d}y \mathop {}\!\textrm{d}x + \int _0^{2\pi } \beta \sinh ^2(k_*)\cos ^2(x) \mathop {}\!\textrm{d}x \\&= \pi \left( \frac{1}{2} + \frac{\cosh (k_*)\sinh (k_*)}{2k_*} + \beta \sinh ^2(k_*)\right) . \end{aligned}\end{aligned}$$

Next, observe that $\zeta _{+,2}=(\eta _{+,2},0, \Phi _{+,2},0)^{{{\,\mathrm{\text {T}}\,}}}$, and similarly the second and fourth components of $U_3$ vanish. Setting

$$\begin{aligned} \zeta _{+,2}-3U_3 = (\eta , 0, \Phi , 0)^{{{\,\mathrm{\text {T}}\,}}}, \end{aligned}$$

and taking into account the boundary conditions from (B.7), we find

$$\begin{aligned}\begin{aligned} \langle L_0(\zeta _{+,2}-3U_3), \zeta _+^* \rangle&= \int _0^1 \int _0^{2\pi } \left( -k_*^2\Phi _{xx}-\Phi _{yy}\right) \cosh (k_*y)\sin (x) \mathop {}\!\textrm{d}x \mathop {}\!\textrm{d}y \\&\quad + \int _0^{2\pi } \left( \alpha \eta -\beta k_*^2 \eta _{xx} - k_*\Phi _x\big |_{y=1}\right) \sinh (k_*)\cos (x) \mathop {}\!\textrm{d}x\\&= -2\int _0^{2\pi } k_2 \eta _{1x} \cosh (k_*)\sin (x)\mathop {}\!\textrm{d}x \\&= 2k_2\int _0^{2\pi } \sinh (k_*)\cosh (k_*)\sin ^2(x)\mathop {}\!\textrm{d}x\\&=\pi k_2 \sinh (2k_*), \end{aligned}\end{aligned}$$

where we have integrated twice by parts and used the linear dispersion relation ${\mathcal {D}}(k_*)=0$. Finally,

$$\begin{aligned}{{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1, k_2) = k_2 \begin{pmatrix} 0 \\ -2\beta k_* \eta _{1xx} - \Phi _{1x}|_{y=1} \\ 0 \\ -2k_*\Phi _{1xx}\end{pmatrix},\end{aligned}$$

which gives

$$\begin{aligned}\langle {{{\,\textrm{D}\,}}}_{Uk}^2F[0,k_*](U_1, k_2), \zeta _+^*\rangle = \pi k_2 \left( 2\beta k_* \sinh ^2(k_*)+ k_*\right) .\end{aligned}$$

Replacing these explicit formulas into (B.9) gives the equality (3.7).

C Sign of the Coefficient $\varvec{m_{21}^{(2)}}$

Consider $\widetilde{m}_{21}^{(2)}$ given by (3.8) and assume $k_*>0$ is such that ${\mathcal {D}}(k_*)=0$. Let

$$\begin{aligned}\sigma = \tanh (k_*) \quad \text {and} \quad \widetilde{T} = \frac{\beta }{\alpha } k_*^2.\end{aligned}$$

Note that $\sigma \in (0, 1)$ and $\widetilde{T} > 0$. A straightforward but lengthy calculation shows that

$$\begin{aligned} \widetilde{m}_{21}^{(2)}= & {} \frac{8k_*}{(1+\widetilde{T})(1-\sigma ^2)^2} \Bigg \{\frac{(1-\sigma ^2)(9-\sigma ^2)+\widetilde{T}(3-\sigma ^2)(7-\sigma ^2)}{\sigma ^2-\widetilde{T}(3-\sigma ^2)}\nonumber \\{} & {} \qquad \qquad \qquad \qquad \qquad \quad +\,8\sigma ^2 - 2(1-\sigma ^2)^2(1+\widetilde{T})-\frac{3\sigma ^2 \widetilde{T}}{1+\widetilde{T}}\Bigg \}\nonumber \\= & {} \frac{8k_*}{(1+\widetilde{T})^2 (1-\sigma ^2)^2} \cdot \frac{1}{\sigma ^2-\widetilde{T}(3-\sigma ^2)} \cdot \left( a_3 \widetilde{T}^3 + a_2\widetilde{T}^2 + a_1\widetilde{T} + a_0\right) ,\nonumber \\ \end{aligned}$$

(C.1)

in which

$$\begin{aligned}\begin{aligned} a_3&= -2\sigma ^6+10\sigma ^4-14\sigma ^2+6, \quad&a_2&=-6\sigma ^6+30\sigma ^4-55\sigma ^2+33,\\ a_1&=-6\sigma ^6+33\sigma ^4-62\sigma ^2+36, \quad&a_0&= -2\sigma ^6+13\sigma ^4-12\sigma ^2+9. \end{aligned}\end{aligned}$$

The coefficients $a_i$ are all positive for $\sigma \in (0, 1)$ and $i=0,1,2,3$. It is now clear that all factors in the above formula for $\widetilde{m}_{21}^{(2)}$ are positive, except for $(\sigma ^2-\widetilde{T}(3-\sigma ^2))^{-1}$, and the sign of $\widetilde{m}_{21}^{(2)}$ is precisely the sign of $\sigma ^2 - \widetilde{T}(3-\sigma ^2)$. Because

$$\begin{aligned} {\mathcal {D}}(2k_{*}) = -\frac{2\alpha \sigma \cosh (2k_*)}{1+\sigma ^2} (\sigma ^2-\widetilde{T}(3-\sigma ^2), \end{aligned}$$

where we have used that ${\mathcal {D}}(k_*)=0$, we find that

$$\begin{aligned}\text {sign}(\sigma ^2-\widetilde{T}(3-\sigma ^2))=-\text {sign}({\mathcal {D}}(2k_*)).\end{aligned}$$

In Region I, the linear dispersion relation ${\mathcal {D}}(k)=0$ has a unique positive simple root $k=k_*$. It follows that the smooth function $k\mapsto {\mathcal {D}}(2k)$ changes signs exactly once at $k=k_*/2$. Since ${\mathcal {D}}(0) =0$ and ${\mathcal {D}}'(0) < 0$, we conclude that ${\mathcal {D}}(2k)$ is negative for $k \in (0, k_*/2)$, and positive for $k \in (k_*/2, \infty )$. In particular, we have ${\mathcal {D}}(2k_*) > 0$. In Region II, the linear dispersion relation ${\mathcal {D}}(k)=0$ has two positive simple roots $k=k_{*,1}$ and $k=k_{*,2}$, $0< k_{*,1} < k_{*,2}$. This means that the smooth function ${\mathcal {D}}(2k)$ changes signs exactly twice, first at $k=k_{*,1}/2$ and then at $k=k_{*,2}/2$. Since ${\mathcal {D}}(0) = 0$ and ${\mathcal {D}}'(0)>0$, we conclude that ${\mathcal {D}}(2k)$ is positive on $(0, k_{*,1}/2)$, negative on $(k_{*,1}/2, k_{*,2}/2)$ and positive on $(k_{*,2}/2, \infty )$. This implies that ${\mathcal {D}}(2k_{*,2})>0$, whereas

$$\begin{aligned}\begin{aligned}&{\mathcal {D}}(2k_{*,1})> 0 \quad \text {if} \quad k_{*,1}>k_{*,2}/2,\\&{\mathcal {D}}(2k_{*,1})<0 \quad \text {if} \quad k_{*,1}<k_{*,2}/2. \end{aligned}\end{aligned}$$

The parameter region in which we have ${\mathcal {D}}(2k_{*,1}) > 0$ is precisely the open region between the curve $\Gamma$ and $\Gamma _2$. We summarize our findings below.

Proposition C.1

If $(\alpha ,\beta )$ belongs to Region I, the coefficient $m_{21}^{(2)}$ is negative. If $(\alpha ,\beta )$ belongs to Region II, then

• (i) $m_{21}^{(2)}$ is negative for both wavenumbers $k_{*,1}$ and $k_{*,2}$ if $(\alpha , \beta )$ lies in the open region between $\Gamma$ and $\Gamma _2$,

• (ii) $m_{21}^{(2)}$ is positive for $k_{*,1}$ and negative for $k_{*,2}$ if $(\alpha , \beta )$ lies to the left of $\,\Gamma _2$.

D Formal Derivation of Instability Criterion

We demonstrate how our instability criterion can formally be derived based on the Davey–Stewartson model. This is most easily expressed in the original physical frame of reference and dimensional variables.⁴ Making a multiple-scales Ansatz

$$\begin{aligned} \phi&=\varepsilon \frac{\sqrt{gk}}{k^2} A(\varepsilon kz, \varepsilon ^2 \sqrt{gk} t)\frac{\cosh (k(y+h))}{\cosh (kh)} e^{{{\,\textrm{i}\,}}(k x-\omega t)} + \text {c.c.} + \cdots ,\\ \eta&= \varepsilon \frac{{{\,\textrm{i}\,}}\omega \sqrt{gk}}{k^2(g+k^2T/\rho )} A(\varepsilon kz, \varepsilon ^2 \sqrt{gk} t)e^{{{\,\textrm{i}\,}}(kx-\omega t)} + \text {c.c.} + \cdots , \end{aligned}$$

one obtains at cubic order in $\varepsilon$ an evolution equation of NLS type in the stretched transverse direction $\zeta =\varepsilon kz$,

$$\begin{aligned} {{\,\textrm{i}\,}}\frac{\partial A}{\partial \tau }+\mu \frac{\partial ^2 A}{\partial \zeta ^2}=\chi |A|^2 A. \end{aligned}$$

In this approximation, the families of two-dimensional periodic waves which we study can be represented by the $\zeta$-independent solution $A=\exp (-{{\,\textrm{i}\,}}\chi \tau )$, and a classical result says that it is unstable precisely when $\mu \chi <0$. According to Ablowitz & Segur [1] (see also Djordjevic & Redekopp [15]) $\mu$ is always positive, while $\chi$ is given by⁵

$$\begin{aligned} \chi= & {} \frac{1}{4\sqrt{\sigma (1+{\tilde{T}})}} \Bigg (\frac{(1-\sigma ^2)(9-\sigma ^2)+{\tilde{T}}(3-\sigma ^2)(7-\sigma ^2)}{\sigma ^2-{\tilde{T}}(3-\sigma ^2)}\\{} & {} \qquad \qquad \qquad \qquad \qquad \qquad +\,8\sigma ^2-2(1-\sigma ^2)^2(1+{\tilde{T}})-\frac{3\sigma ^2 {\tilde{T}}}{1+{\tilde{T}}} \Bigg ), \end{aligned}$$

in which ${\tilde{T}}=k^2 T/(g\rho )$ and $\sigma =\tanh (kh)$. Comparing with (C.1), we find that

$$\begin{aligned} {\widetilde{m}}_{21}^{(2)}=\frac{32 k_* h \sqrt{\sigma (1+{\tilde{T}})}}{(1+{\tilde{T}})(1-\sigma ^2)^2} \chi . \end{aligned}$$

Hence ${\widetilde{m}}_{21}^{(2)}$ and $\chi$ have the same sign and give rise to the same instability criterion. The fact that the sign changes precisely along the second harmonic resonance curve $\Gamma _2$, where $k_{*,2}=2k_{*, 1}$, agrees with Figure 1 of [1].