Lattice paths with a first return decomposition constrained by the maximal height of a pattern

We consider the system of equations $A_k(x)=p(x)A_{k-1}(x)(q(x)+\sum_{i=0}^k A_i(x))$ for $k\geq r+1$ where $A_i(x)$, $0\leq i \leq r$, are some given functions and show how to obtain a close form for $A(x)=\sum_{k\geq 0}A_k(x)$. We apply this general result to the enumeration of certain subsets of Dyck, Motzkin, skew Dyck, and skew Motzkin paths, defined recursively according to the first return decomposition with a monotonically non-increasing condition relative to the maximal ordinate reached by an occurrence of a given pattern $\pi$.


Introduction and notations
Let A be a combinatorial class, that is a collection of similar objects (lattices, trees, permutations, words) endowed with a size function ℓ whose values are non-negative integers, so that the number |ℓ −1 (n)| of objects of a given size n is finite. With A we associate the generating function which appears to be the main source of interest for researchers working in enumerative combinatorics.
Given a function s : A → N, called statistic, and an integer k 0, we define the combinatorial class A k consisting of objects a ∈ A such that s(a) = k, and associate the following generating function with this class: In three recent articles [1,2,6], the authors enumerate particular subclasses of lattice paths (Dyck, Motzkin, 2-Motzkin and Schröder paths) defined recursively according to the first return decomposition with a condition on the path height. More precisely, in the first paper, they focus on the Dyck path set D h, consisting of the union of the empty path with all Dyck paths P having a first return decomposition P = U αDβ satisfying the conditions: where h is the height statistic, i.e. the maximal ordinate reached by a path. In these studies, the authors need to consider a system of equations involving the generating functions D k (x), k 0, for the subset of paths P ∈ D h, satisfying h(P ) = k: anchored with initial condition D 0 (x) = 1. They deduce algebraically a close form of the generating function D(x) = k 0 D k (x) for D h, by solving System (2), and prove that D h, is enumerated by the Motzkin sequence, that is A001006 in the On-Line Encyclopedia of Integer Sequences (OEIS) [13]. The second paper is a similar study in the context of Motzkin and Schröder paths.
Inspired by these two works, the motivation of the paper is to extend such studies by investigating the solutions of a more general system defined as follows. For an integer r 0 and two functions p(x) and q(x), we consider the system of equations: anchored with A i (x) = u i (x) for 0 i r where u i (x), 0 i r, are some given functions. For the sake of brevity, we set u = u r (x), v = r i=0 u i (x) − u, p = p(x) and q = q(x). Before considering how Equation (3) is related to particular sets of lattice paths defined in the same way as the set D h, , let us recall some definitions and notations. A skew Motzkin path (see [7]) of length n 0 is a lattice path in the quarter plane starting at point (0, 0), ending on the x-axis (but never going below), consisting of n steps of the four following types: up steps U = (1, 1), down steps D = (1, −1), flat steps F = (1, 0) and left steps L = (−1, −1), such that left and up steps never overlap. Let SM be the set of all skew Motzkin paths of any length. A Motzkin path is a skew Motzkin path with no left steps. Let M be the set of all Motzkin paths. A Dyck path is a Motzkin path with no flat steps. Let D be the set of all Dyck paths. A skew Dyck path is a skew Motzkin path with no flat steps. Let SD be the set of all skew Dyck paths. Obviously, we have D ⊂ SD ⊂ SM and D ⊂ M ⊂ SM.
A pattern of length n 1 in a lattice path P consists of n consecutive steps. The height of an occurrence of a pattern in a given path is the maximal ordinate reached by its points. For a pattern π, we define the statistic h π on lattice paths so that h π (P ) is the maximal height reached by the occurrences of π in P . The amplitude r π of a pattern π is the height of the path π considered as a path touching the x-axis in the quarter plane. For instance, the path P = U U DU U DU DDD contains the pattern U U D and h UUD (P ) = 3, while the amplitude of U U D is r UUD = 2.
The paper is organized as follows. In Section 2, we show how we can obtain a close form for A(x) defined by the system (3). In Section 3, we provide several concrete applications in the context of lattices paths (Dyck, Motzkin, skew Dyck, and skew Motzkin paths) for the statistic h π giving the maximal ordinate reached by the occurrence of π in a path. Finally in Section 4, we provide some possible future research directions.

General result
, and thus we have A(x) = lim k→∞ B k (x). The next theorem provides a close form of A(x) whenever Equation (3) admits a solution, that is when 1 − A k (x)p(x) = 0 for all k r. where Proof. Using B r (x) = u + v, we solve the following linear system for a, b, c, d and obtain (modulo a multiplicative factor) the claimed values. However, it remains to prove that for any k r + 1. We proceed by induction on k. Obviously, for k = r + 1 it is true. So, we assume that the formula is true for k n and we prove that this is also true for k = n + 1. More precisely, we will prove that R = (c + dB n (x))B n+1 (x) − (a + bB n (x)) equals zero.
. The recurrence hypothesis provides B n (x) = a+bBn−1(x) c+dBn−1(x) , and we have A n (x) = B n (x) − B n−1 (x). Replacing respectively B n+1 (x), B n (x) and A n (x) in R, and multiplying it by 1 − p · A n (x) = 0, we obtain where S = B n−1 (x). A simplification of this expression (whatever the value of S) implies that R = 0, which completes the induction. Finally, taking the limit when k → ∞ of both sides of B k (x) = a+bB k−1 (x) c+dB k−1 (x) , and using the fact that Notice that an alternative proof can be obtained by using Chebyshev polynomials of the second kind (see Proposition D.5 p. 508 in [10]).

Some applications
In this section, we apply Theorem 2.1 to study special kinds of classes of lattices paths: Dyck, Motzkin, skew Dyck, and skew Motzkin paths. We extend the definition (1) to these paths using the maximal height reached by the occurrences of π, the statistic h π instead of statistic h. Notice that the statistic h equals the statistic h U on all considered paths, which allows us to focus only on h π . We will deal with patterns π of length at most three.
Given a pattern π, we define the set A hπ , as the union of the empty path with paths P satisfying a recurrence condition (R), which depends on the first return decomposition of a path. The four cases are described in the following table with α, β, γ ∈ A hπ , .
Paths First return decomposition with recurrence condition (R) with h π (U αD) h π (β) and h π (F ) h π (γ) and α = ǫ for the last two cases We keep the notations used in the previous sections. The generating functions A k (x), k 0, correspond to the sets A k of Dyck paths P ∈ A hπ , so that h π (P ) = k. We set In any case, the set A 0 of paths P ∈ A hπ, such that h π (P ) = 0 is the set of paths having no occurrence of π at height at least one. In the case where the amplitude of π is at least one, A 0 consists of the paths avoiding π. So, we can easily obtain A 0 (x) using the first return decomposition of such paths. For Dyck and Motzkin paths, these generating functions (for short g.f.) are already known (see [3,4,8,9,11,12]). For skew Dyck and Motzkin paths, we can obtain them by similar methods, which are classic and present no difficulty. So, we do not give here all the details to get A 0 (x).
Note that the maximal height of an occurrence of pattern π in P is necessarily greater or equal than its amplitude r π . This means that the generating functions for the sets A k , 1 k < r π satisfy A k (x) = 0. Finally, if r π 1 (which is equivalent to π = F k for all k 1) then A rπ , is the set of paths P ∈ A hπ, having at least one occurrence of π and such that h π (P ) = r π , i.e. all occurrences of π touch the x-axis in P . Similar methods as for A 0 (x) can be used in order to obtain A rπ (x). Theorems 3.1 and 3.2 focus on two examples of these methods in the context of Dyck paths.

Dyck and Motzkin paths
First we consider Dyck paths. For a given pattern π, Condition (R) implies that for k r π + 1 we have which is a particular case of Equation (3) From Theorem 2.1, we can easily deduce the following.
In order to convince the reader that all generating functions A(x) can be obtained easily by calculating algebraically u and v, we give here the methods for the two patterns U U D and DU U .
Proof. As already mentioned above, the set A 0 is the set of Dyck paths avoiding the pattern π. So, a nonempty path P ∈ A 0 is of the form P = U Dα where α ∈ A 0 , which implies that A 0 (x) = 1 1−x . On the other hand, the set A 1 is the set of Dyck paths P ∈ A hπ, such that h UUD (P ) = 1, which cannot be possible because the amplitude r π = 2; so, A 1 (x) = 0. Finally, the set A 2 is the set of Dyck paths P ∈ A hπ , such that h UUD (P ) = 2, which means that P can be written U αDβ with α ∈ A 0 \ǫ and β ∈ A 2 ∪ A 1 ∪ A 0 . So, we have the functional equation which gives the expected result.
Proof. The set A 0 is the set of Dyck paths avoiding the pattern DU U . So, a nonempty path P ∈ A 0 is of the form P = U αDβ where α ∈ A 0 and β = (U D) k for some k 0, which implies that A 0 (x) = 1 + xA0(x) 1−x , and thus A 0 (x) = x−1 2x−1 . On the other hand, the set A 1 is the set of Dyck paths P ∈ A hπ , such that h DUU (P ) = 1, which cannot be possible because the amplitude r π = 2; so, A 1 (x) = 0. Finally, the set A 2 is the set of Dyck paths P ∈ A hπ, such that h DUU (P ) = 2, which means that P can be written U αDU βD with α ∈ A 0 and β ∈ A 0 \{ǫ}.
So, we have the functional equation , which gives the expected result.
So, we present in the following table the first values of the sequences corresponding to the cardinality of A hπ, for patterns π of length at most three.
Statistic h π a n , 1 n 9 OEIS h, U , D, U D, U U , DD, U DD, U U D Now, we examine the set of Motzkin paths. For a given pattern π, Condition (R) implies that for k r π + 1 we have which is a particular case of Equation (3) for r = r π , p(x) = x 2 , q(x) = 0, u = A rπ (x), and v = A 0 (x). Then, the generating function A(x) is the generating function found above in the context of Dyck paths (see (6)) applied on the variable x 2 (x is not replaced with x 2 in u and v). Looking at the two tables above, one observes that some patterns have same sequences, for example DU U and DDU , U F and F D. We actually can generalize this as follows. Let π = π 1 π 2 . . . π k be a pattern of length k and C(π) be the reversed complement of π, that is C(π) = π k π k−1 · · · π 1 where For instance we have C(U F DD) = U U F D. Theorem 3.3. If π A(x) is the generating function for the set A hπ , , then we have Proof. For short we set σ = C(π). Since the reversed complement operation preserves the amplitude, we have r π = r σ . Now, we will exhibit a bijection φ from A hπ, to A hσ , . Suppose π = F k , k 1, if a path P ∈ A hπ, does not contain π, we set φ(P ) = C(P ). It avoids σ and belongs to A hσ , . So, we have π A 0 (x) = σ A 0 (x). If π = F k for some k 1, we have r π = 0 and we also set φ(P ) = C(P ). We easily deduce again that π A 0 (x) = σ A 0 (x). Now, let us consider the case where r π 1 and prove that g.f. π A rπ for pattern π is actually equals σ A rσ for pattern σ. To do this we consider a path P ∈ A hπ, having at least one occurrence of π and such that h π (P ) = r π . We can write P either as (i) P = F α or (ii) P = U αDβ where α, β ∈ A hπ , .
Case (i). We assume P = F α and h π (F ) h π (α). Since the pattern π is not F k (because r π 1), we necessarily have h π (F ) = 0 = h π (α) which means that α avoids π. Therefore, the pattern π occurs once in P and it straddles F and α. We deduce that C(P ) belongs to A hσ , and it has at least one occurrence of σ such that h σ (C(P )) = r σ . So in this case, we define the function φ with φ(P ) = C(P ).
Case (ii). We assume P = U αDβ and h π (U αD) h π (β). If h π (U αD) = 0 then we have h π (β) = 0 and using the same argument as previous, C(P ) belongs to A hσ , and it has exactly one occurrence of σ straddling U αD and β, and such that h σ (C(P )) = r σ . In this subcase, we set φ(P ) = C(P ). If h π (U αD) = r π then the occurrence of π in U αD is at the beginning or at the end (or touches both extremities) of U αD since it must touch the x-axis, but the occurrence does not straddle U αD and β. This means that α avoids π and β either avoids π (i.e. h π (β) = 0, as π = F k ) or contains π with h π (β) = r π . In this subcase, we recursively define φ as φ(P ) = U C(α)Dφ(β). It belongs to A hσ, and has at least one occurrence of σ such that h σ (φ(P )) = r σ . So, φ is a one-to-one correspondence between paths in A hπ, having at least one occurrence of π such that h π (φ(P )) = r π and paths A hσ , having at least one occurrence of σ such that h σ (φ(P )) = r σ . So, we have π A rπ (x) = σ A rσ (x).

Skew Dyck and Motzkin paths
Let us discuss the skew Dyck paths first. For a given pattern π, Condition (R) implies that for k r π + 1 we have which is a particular case of Equation (3) for r = r π , p(x) = x, q(x) = 1, u = A rπ (x), and v = A 0 (x). Then, we deduce the following.
with ∆ = u 2 v 2 x 4 + 2 uv 3 x 4 + v 4 x 4 + 2 u 2 vx 4 + 6 uv 2 The following table gives the first values of generating functions A(x) obtained for all patterns of length at most two. Notice that none sequences appear in [13].
Statistic h π a n , 1 n 9 OEIS h, U , D, U U , U D In the context of skew Motzin paths we have which is a particular case of Equation (3) for r = r π , p(x) = x 2 , q(x) = x · A 0 (x) + 1, u = A rπ (x), which means that v = 0 when r π = 0 and v = A 0 (x) otherwise. Then, the generating function A(x) is the generating function for skew Dyck paths (see Equation (7)) applied on the variable x 2 (x is not replaced with x 2 in u and v), and we obtain the following results for patterns of length at most one. The sequences do not appear in [13].
Statistic h π a n , 1 n 9 OEIS

Conclusion and research directions
Except Dyck and Motzkin cases, where we obtain two sequences appearing in [13] for the pattern U , all the others have never been studied in the literature. So, this work provides a new catalog of sequences which is a possibly fertile ground in number theory, and more precisely in the analyse of their modular congruences (see [5] for instance). Another research direction could be the investigation of the distribution of a given pattern in the set A hπ, . For instance, if π = U U in the context of Dyck paths, then the paths in A hπ, avoiding the pattern U DU are enumerated by the generalized Catalan numbers (see A004148 in [13]), which also count peak-less Motzkin paths. It would be interesting to find a one-to-one correspondence between these objects. In this paper, we focus on the generating A(x) of A hπ , , but we do not deal with the generating functions A k (x) for paths P ∈ A hπ, such that h π (P ) = k since they can be easily obtained. Observing these g.f. for small k, we have identified some of them that are already known in OEIS [13]. For instance, the number of n-length Dyck paths P ∈ A hUU , such that h UU (P ) = 2 is the classical number of Fibonacci minus one (see A000071 in [13]). Such a result suggests us to look for new bijections with known classes counted by this sequence.
Finally, it would be interesting to extend this work to other classes of objects counted by generating functions satisfying System (3).