Parabolic double cosets in Coxeter groups

Parabolic subgroups $W_I$ of Coxeter systems $(W,S)$, as well as their ordinary and double quotients $W / W_I$ and $W_I \backslash W / W_J$, appear in many contexts in combinatorics and Lie theory, including the geometry and topology of generalized flag varieties and the symmetry groups of regular polytopes. The set of ordinary cosets $w W_I$, for $I \subseteq S$, forms the Coxeter complex of $W$, and is well-studied. In this article we look at a less studied object: the set of all double cosets $W_I w W_J$ for $I, J \subseteq S$. Double coset are not uniquely presented by triples $(I,w,J)$. We describe what we call the lex-minimal presentation, and prove that there exists a unique such object for each double coset. Lex-minimal presentations are then used to enumerate double cosets via a finite automaton depending on the Coxeter graph for $(W,S)$. As an example, we present a formula for the number of parabolic double cosets with a fixed minimal element when $W$ is the symmetric group $S_n$ (in this case, parabolic subgroups are also known as Young subgroups). Our formula is almost always linear time computable in $n$, and we show how it can be generalized to any Coxeter group with little additional work. We spell out formulas for all finite and affine Weyl groups in the case that $w$ is the identity element.


Introduction
Let G be a group with subgroups H and K. The group G is partitioned by the collection of double cosets H\G/K = {HgK | g ∈ G}. The double cosets are usually more complicated than the one-sided cosets. For example, different double cosets HgK and Hg K can have different sizes. In this work, we investigate the parabolic double cosets of a finitely generated Coxeter group. That is, given a Coxeter system (W, S) of finite rank |S|, we consider cosets W I wW J where I and J are subsets of the generating set S, and W I = s : s ∈ I denotes the standard parabolic subgroup of W generated by the subset I. Such double cosets are well-studied, e.g., they play a prominent role in the paper of Solomon that first defines the descent algebra of a Coxeter group [11]. For finite Coxeter groups, these double cosets are intervals in Bruhat order, and these intervals have a rank-symmetric generating function with respect to length [7]. Such intervals correspond to the cell decomposition of certain rationally smooth Richardson varieties.
If we fix I and J, then the double quotient W I \W/W J is also well-studied. For example, Stanley [12] shows the Bruhat order on such a double quotient is strongly Sperner (for finite W ), and Stembridge [14] has characterized when the natural root coordinates corresponding to elements in the quotient give an order embedding of the Bruhat order (for any finitely generated W ). The number of elements in the quotient is a product of characters [13,Ex 7.77a], where 1 W J denotes the trivial character on W J . In this paper, we are interested in a basic problem about parabolic double cosets that appears to have been unexamined until now; namely, how many double cosets W I wW J does W have as I and J range across subsets of S? Part of the motivation for this question comes from the analogous problem for ordinary cosets, where the set of such wW I as I ranges across subsets of S is equal to the set of cells of the Coxeter complex. When W is the symmetric group S n , the number of such cells is the nth ordered Bell number, see [8, A000670]. One fact that makes the ordinary case substantially simpler than the double version is that each ordinary parabolic coset is of the form wW I for a unique subset I ⊆ S. If we take w to be the minimal element in the coset, then the choice of w is also unique. While double cosets do have unique minimal elements, different sets I and J often give the same double cosets W I wW J . Thus our question cannot be answered by simply summing equation (1) over all I and J.
To make our problem more tractable, we restrict to the set of double cosets with a fixed minimal element w. Since W has finite rank this set is always finite. Our main contribution is a formula for the number c w of double cosets with minimal element w when W = S n . This formula is efficient in most cases. The key to the formula is a condition on pairs of sets (I, J) that guarantees that each double coset W I wW J arises exactly once. In other words, we get a unique presentation W I wW J for each double coset with minimal element w. This criterion holds for any Coxeter group, and thus in principle our formula for c w can also be extended to any Coxeter group. Specific enumerative consequences for other Coxeter groups are deferred to the full version of this paper.
When W = S n , we can use our formula for c w to calculate the number of double cosets in W . Although this requires summing up n! terms, the approach seems to be a significant improvement over what was previously known. The terms p n for 1 ≤ n ≤ 10 are  [4], [5,Section 5]). It is also the number of cells in a two-sided analogue of the Coxeter complex recently studied by the third author [9].
In the rest of this abstract, we first describe our enumeration formula for the symmetric group in some detail (Theorem 1, Section 2), review some basic properties of double cosets (Section 3), and then explain how to choose a unique presentation for each double coset (Section 4). The proof of the enumeration formula is sketched in Section 5.

The marine model and the formula for c w
Let W = S n , a Coxeter group with generating set S = {s 1 , . . . , s n−1 }, where s i denotes the ith adjacent transposition. Write a permutation w = w 1 · · · w n ∈ S n in one-line notation. Call j a right ascent of w if w j < w j+1 , and i a left ascent of w if the value i appears to the left of i + 1 in w 1 · · · w n (equivalently, if i is a right ascent of w −1 ). An index in {1, 2, . . . , n − 1} that is not an ascent is a descent. Let Asc(w) denote the right ascent set of w. By symmetry, Asc(w −1 ) is the left ascent set. The right descent set of w is the complement of Asc(w) in S.
In terms of simple generators, j is a right ascent (resp., i is a left ascent) if and only if (ws j ) > (w) (resp., (s i w) > (w)), where is the length function. The element w is the minimal element of a double coset C if and only if C has the form W I wW J , where I and J are subsets of the left and right ascent sets of w respectively.
In addition to distinguishing between left and right ascents, we want to highlight ascents that are "small." A small right ascent of w is an index j such that w j+1 = w j + 1. If j is a small right ascent, then ws j w −1 = s i , where i = w j , and we say that i is a small left ascent of w. Any ascent that is not a small ascent is a large ascent. For example, w = 245613 has Asc(w) = {1, 2, 3, 5} and Asc(w −1 ) = {2, 4, 5}. Its small right ascents are 2 and 3, while its small left ascents are 4 and 5. Both 1 and 5 are large right ascents, and 2 is a large left ascent.
Our formula for c w involves structures and terminology that we will refer to as the marine model. As part of this model, we make the following definitions.
. . , j are all small right ascents, while i − 1 and j + 1 are not.

2.
A (right) float of w is a large right ascent that is not adjacent to any rafts. In other words, an index h is a right float of w if w(h) < w(h + 1) = w(h) + 1, and both w(h) = w(h − 1) + 1 and w(h + 1) + 1 = w(h + 2).

A (right)
rope of w is a large right ascent that is adjacent to exactly one raft. 4. A (right) tether of w is a large right ascent that is adjacent to two rafts.
The sets of (right) rafts, floats, ropes, and tethers of w will be denoted by Rafts(w), Floats(w), Ropes(w), and Tethers(w) respectively. An explanation for this "maritime" terminology is provided in Section 5.
Observe that every right ascent of w is either a tether, a rope, a float, or part of a raft. Also, because rafts are in some sense maximal, distinct rafts cannot be adjacent. For example, if w = id ∈ S n then w has one raft R = [1, n − 1], and no floats, ropes, or tethers. The rafts of 123567849 are [1,2] and [4,6], the only right float is at 8, and the only right tether is at 3.
Theorem 1 There is a finite family of sequences b I m , such that for any permutation w, the total number of parabolic double cosets with minimal element w is equal to The sequences b I m satisfy a linear recurrence, and thus can be easily computed in time linear in m. The definition of these sequences requires a fair bit of notation, and we defer it to Section 5. The set Tethers(w) is typically small (on the order of 1/n), leading to an efficient formula for c w . Unfortunately, there are some permutations for which | Tethers(w)| + | Tethers(w −1 )| is quite large. For example, the permutation w = (1, 2, 17, 18, 3, 4, 19, 20, . . . , 15, 16, 31, 32) has 8 tethers, and its inverse has 14, leading to a sum with 2 22 terms (see Figure 2). In this case, the value c w = 632371867544102 can be determined on a computer within a few minutes.

Parabolic double cosets
As in the introduction, let (W, S) be a Coxeter system of finite rank |S| < ∞. The generating set S is the set of simple reflections of W . The simple reflections and their relations are encoded in an edge-labeled graph on S called the Coxeter graph. Two generators s, s ∈ S are adjacent if the corresponding vertices are adjacent in the Coxeter graph. The set of reflections is the set of conjugates of S, denoted by T = {wsw −1 : s ∈ S, w ∈ W }. Every element w ∈ W can be written as a product of elements of S, and the length (w) of w is the minimal number of simple reflections in such a product. The Bruhat order on W is defined by taking the transitive closure of the relations w < wt, where t ∈ T and (w) < (wt).
As in the case of symmetric group, we say that a right ascent of w is a simple reflection s ∈ S such that (ws) > (w). Similarly, if (sw) > (w), then s is a left ascent. We denote the set of right ascents by Asc(w). The set of left ascents is equal to Asc(w −1 ). The descent set of w is the complement of Asc(w) in S, i.e., it records the simple reflections s such that (ws) < (w). We refer the reader to [1,6] for further background on Coxeter groups.
As in the introduction, the Coxeter group generated by I ⊆ S is a standard parabolic subgroup of W , denoted by W I . We will identify I with the induced subgraph on vertices I of the Coxeter graph for W . For example, this I is connected if the corresponding induced subgraph is connected. The left cosets in W/W I each have a unique minimal length element, and thus W/W I can be identified with the set W I of all minimal length left W I -coset representatives. An element w ∈ W belongs to W I if and only if I contains no right descents of w; that is, if I ⊆ Asc(w).
Every element w ∈ W can be written uniquely as w = w I v, where w I ∈ W I and v ∈ W I . This is the parabolic decomposition of w [6, Sect. 5.12]. The product w = w I v is a reduced factorization, meaning that (w) = (w I ) + (v). As a poset under Bruhat order, every coset wW I is isomorphic to W I . Consequently, if W I is finite, then every coset wW I is also finite, and in addition has a unique maximal element, implying that wW I is a Bruhat interval. Analogous statements can be made for the right cosets W I w. Write I W for the set of minimal length right coset representatives for W I \W .
A parabolic double coset is a subset C ⊆ W of the form C = W I wW J for some w ∈ W and I, J ⊆ S. Parabolic double cosets inherit some of the nice properties of ordinary cosets mentioned previously, including the following:  In this case, each C ∈ W I \W/W J has a unique maximal length element which is also the unique maximal element with respect to Bruhat order. In particular, if C is finite then it is a Bruhat interval.

Canonical presentations of double cosets
Fix a parabolic double coset C. A presentation of C is a choice of I, J and w such that C = W I wW J . Presentations are not unique; for instance, W = W S eW ∅ = W ∅ eW S = W S wW S for any w ∈ W .
To solve the problem that presentations are not unique, we define and characterize three types of presentations: maximal, minimal, and lex-minimal. We show that every parabolic double coset has unique maximal and lex-minimal presentations. The lex-minimal presentations are those that we have found most suitable for enumeration. The presentation W M L (C) wW M R (C) is the maximal presentation for C. We now look at presentations that are as small as possible.
If a connected component I 0 of I is contained in wJw −1 ∩ S for w ∈ I W J , then W I wW J = W I\I0 wW J . In other words, every presentation can be reduced to a minimal presentation. In Proposition 9, we will show that our nomenclature is appropriate, in that minimal presentations have minimum size. That M L (C) contains the set on the right hand side is clear. The other inclusion takes only slightly more work. The equality for M R (C) is analogous.
Corollary 8 Suppose that C = W I wW J is a minimal presentation of C. If T is any connected subset of M L (C), then either T ⊆ I, or T is disjoint and non-adjacent to I and T ⊆ (wJw −1 ) ∩ S.
Given subsets X, Y, Z ⊆ S, write X = Y ∼ Z to mean that X is the disjoint union of Y and Z, and Y and Z are non-adjacent. (In other words, the vertex-induced subgraph of X is isomorphic to the disjoint union of the vertex-induced subgraphs of Y and Z.) The following proposition is, roughly speaking, obtained by repeated application of Corollary 8.

Proposition 9
Let w ∈ I W J . Then a presentation C = W I wW J is minimal if and only if |I| + |J| ≤ |I | + |J | for all other presentations C = W I wW J of C. Furthermore, if C = W I wW J and C = W I wW J are both minimal presentations, then there is a sequence of connected components I 1 , . . . , I m of I and J 1 , . . . , J n of J such that (Note that the order of operations in these two identities is significant, in that wJ j w −1 is disjoint and non-adjacent to I i for all i, j.) We now define lex-minimal presentations and give our desired characterization as a corollary to Proposition 9.
Definition 10 A presentation C = W I wW J of a parabolic double coset C is lex-minimal if w is the minimal element of C, and (|I|, |J|) is lexicographically minimal among all presentations of C.
In other words, if C = W I wW J is lex-minimal and C = W I w W J is another presentation of C, then w ≤ w , and either |I| < |I |, or |I| = |I | and |J| ≤ |J |. Furthermore, every parabolic double coset has a unique lex-minimal presentation.
When w is the identity, Theorem 11 implies that lex-minimal presentations are two-level staircase diagrams in the sense of [10]. Note that while [10] addresses the enumeration of staircase diagrams, two-level staircase diagrams are not considered.

The Marine Model for S n
We now return to the case W = S n , and fix a permutation w = w 1 · · · w n ∈ S n . Given that sets I and J are contained in the left and right ascent set of w, i.e., I ⊆ Asc(w) and J ⊆ Asc(w −1 ), we want to determine when W I wW J is lex-minimal. The Coxeter graph G of S n is a path with vertices labelled {1, 2, . . . , n − 1}. To the permutation w, we associate a diagram called the marine model, as follows. First, take two isomorphic copies G L and G R of G, and let B w be the graph obtained from G L and G R , along with edges connecting the small left ascents of w to the associated small right ascents of w: vertex i in G L and vertex j in G R are connected whenever ws j w −1 = s i . The marine model is the graph A w obtained by deleting from B w all edges of G L and G R that are not incident to any small ascents.
As in the introduction, it is useful to have some terminology to refer to the parts of A w . We use the following terminology: 1. Plank -an edge in A w connecting a small left ascent to a right small ascent.
2. Raft -a maximal connected component of adjacent planks, where two planks are adjacent if the induced subgraph of A w containing both of them is connected. As shown below, rafts are drawn as a series of parallel line segments of the same length connected across the top and bottom, hence the name. As in the introduction, we can identify the raft by the corresponding interval [i, j] of small right ascents of w.
3. Float -a large left or right ascent of w not adjacent to any rafts.
4. Rope -a large left or right ascent of w adjacent to exactly one raft.
5. Tether -a large left or right ascent of w connected to two rafts.
It is helpful to draw A w as follows. Draw two rows of n − 1 dots representing the possible right (top row) and left (bottom row) ascents of of w. Denote small ascents and large ascents by small and large dots respectively. For each raft [i, j] of w, draw a line from dot k on the top to dot w k on the bottom for k = i, . . . , j, and connect the dots i, . . . , j on the top and w i , . . . , w j on the bottom. For example, for w = (1,3,4,5,7,8,2,6,14,15,16,9,10,11,12,13), we have rafts [2,3], [5,5], [9,10], and [12,15]. Thus we draw nine lines from upper dots to lower dots, three lines on the top (2 to 3, 9 to 10, and 12 to 15), and three lines on the bottom (3 to 4, 9 to 12, and 14 to 15). If i is a rope or tether, draw an edge or edges horizontally from the larger dot in position i to the adjacent small ascent or ascents. In our example, the tethers are the edges incident to 4 on the top and 8 on the bottom. The ropes are the edges incident to 1 and 8 on the top and 5 on the bottom. We also circle any dot corresponding to a float, so in our example we circle dot 7 on the top and dot 1 on the bottom. The isolated dots that are not circled are the descents. This example is shown in Figure 1.
The enumeration formula in Theorem 1 counts the number of lex-minimal presentations of double cosets with fixed w ∈ S n as the minimal element. This is equivalent to the number of ways of choosing subsets I and J such that the conditions in Theorem 12 are satisfied. If the marine model of w consists of a single raft, then pairs (I, J) satisfying the conditions can be recognized by a finite state automaton, pictured in Figure 3, that scans from left to right. The complete description of this automaton is deferred to the full paper. As a result, the transfer-matrix machinery can be used to find a recurrence for the number of choices for I and J. A raft with ropes attached can be handled similarly, using different combinations of starting and ending states in the automaton. To make this specific, let (a k m ) m∈N , k ∈ {0, 1, 2, 2 , 2 , 3, 4}, be the family of sequences defined by the recurrence with initial conditions a k 0 , . . . , a k 4 given by the following table. Remark The characteristic polynomial corresponding to the recurrence, 1−6t+13t 2 −16t 3 +11t 4 −4t 5 , factors as (1 − t + t 2 )(1 − 5t + 7t 2 − 4t 3 ), and the sequences (a k m ) m∈N for k = 0, 1, 2, 3, 4 actually satisfy a recurrence of order 3: However, a single recurrence for all sequences makes the definition more concise.
: i 1 + i 2 + i 3 + i 4 = 2 and i 1 = i 3 2 : The tuple (i 1 , i 2 , i 3 , i 4 ) represents the lower-left, upper-left, lower-right, and upper-right outer corners of a raft: i2 i4 i1 i3 . The indicators are 1 (selected) or 0 (not selected) depending on whether or not a rope or tether attached at that point is selected for I or J. By symmetry, all four configurations with exactly one indicator equal to 1 are equivalent, as are the four configurations with exactly one indicator equal to 0. There are three configurations up to symmetry with exactly two indicators equal to 1, and these correspond to the states k = 2, 2 , and 2 . Then a k(i1,i2,i3,i4) m is the number of choices of I and J for a raft of length m. The automaton we use to recognize pairs (I, J) has 8 states (see Figure 3), and thus the generating functions for the sequences a k m can be computed as a sum of terms (−1) i+j det(I−tA) ji det(I−tA) , For a general permutation w, we have to handle floats, rafts, ropes, and tethers. Floats can be chosen or rejected completely independently of the other elements of I and J, leading to the term 2 | Floats(w)|+| Floats(w −1 )| .
Ropes and tethers can also be chosen independently, provided that this choice is incorporated into the calculation for the attached rafts as above. Once these choices are fixed, the calculation can be made independently for each raft. Because ropes are attached to exactly one raft, we gain some efficiency by incorporating the choice of ropes into the calculation for each raft. However, tethers must be chosen in advance, because they appear in two rafts. Thus the formula for c w in Theorem 1 involves a sum across choices S and T of tethers on the bottom and the top respectively, after which we take a product of sequences b I(R,S,T ) |R| corresponding to the rafts. To make this specific, we define a family of sequences b I m indexed by tuples I = (I 1 , I 2 , I 3 , I 4 ), where I i is a non-empty subset of {0, 1} indicating attached ropes or tethers in the same position as used for the sequences a k(i1,i2,i3,i4) n . Here I i = {0} means that no rope or tether can be selected, {1} means that an attached tether must be selected, and {0, 1} means that an attached rope is available for selection. With this convention, where the sum is across (i 1 , i 2 , i 3 , i 4 ) ∈ I 1 × I 2 × I 3 × I 4 . To describe which sequence b I m to use for each raft, suppose that R = [i, j] ∈ Rafts(w), S ⊆ Tethers(w −1 ), and T ⊆ Tethers(w). Then I(R, S, T ) is the 4-tuple (I 1 , I 2 , I 3 , I 4 ), where: Example 13 Let us compute the number of parabolic double cosets whose minimal element is our running example w = (1,3,4,5,7,8,2,6,14,15,16,9,10,11,12,13). There are two floats, so the first term is 2 2 . We have tether 4 on top and tether 8 on the bottom, so S ⊆ {8}, T ⊆ {4}, and there are four terms in the sum. For S = T = ∅, the raft [2,3] is of size 2 and has two ropes on opposite sides (the tether on top right was not selected), and contributes b