Diagrams and essential sets for signed permutations

We introduce diagrams and essential sets for signed permutations, extending the analogous notions for ordinary permutations. In particular, we show that the essential set provides a minimal list of rank conditions defining the Schubert variety or degeneracy locus corresponding to a signed permutation. Our essential set is in bijection with the poset-theoretic version defined by Reiner, Woo, and Yong, and thus gives an explicit, diagrammatic method for computing the latter.


Introduction
Representing a permutation in S n as a matrix of dots in an n×n array of boxes, its (Rothe) diagram is the subset of boxes that remain after striking out the boxes (weakly) south or east of each dot. The boxes of the diagram correspond naturally to the inversions of the permutation, so the number of them is equal to the length of the permutation. In fact, the diagram is a convenient way of encoding a great deal of information about a permutation. The goal of this article is to develop an analogous tool for signed permutations.
A permutation v ∈ S n determines n 2 rank conditions on the set of all n × n matrices, by requiring that each lower-left submatrix have rank at least that of the corresponding part of the permutation matrix for v; more generally, v determines a degeneracy locus for a flagged vector bundle on a variety, by imposing this condition on each fiber. These rank conditions are highly redundant, however. A minimal list of non-redundant rank conditions is determined by the essential set of v, introduced in [Fu]. It can be read easily from the diagram of v: the definition of the essential set identifies it with the set of southeast corners of the diagram. Seeking formulas for degeneracy loci of other classical types [AF0,AF1], we were motivated to find minimal lists of rank conditions corresponding to signed permutations, since these index degeneracy loci in other types. For example, in the type B setting of an odd-rank vector bundle equipped with a nondegenerate quadratic form, a degeneracy locus can be represented (locally) by a (2n + 1) × n matrix whose columns are required to be isotropic and mutually orthogonal. A signed permutation w specifies 2n 2 + n rank conditions on such a matrix, most of which are redundant; we shall find a minimal essential set of rank conditions which suffice, and can be read easily from the diagram of w, which are introduced here.
All the statements and most of the arguments in the body of this paper will be combinatorial, but it will be useful to explain some of this geometric motivation in more detail first. Consider an odd-rank vector bundle V on a variety X, equipped with a nondegenerate quadratic form and flags of subbundles V ⊃ E 1 ⊃ E 2 ⊃ · · · and V ⊃ F 1 ⊃ F 2 ⊃ · · · , with E 1 and F 1 maximal isotropic. A signed permutation w defines a degeneracy locus Ω w ⊆ X by imposing certain rank conditions dim(E p ∩ F q ) ≥ k. Here k = r w (p, q) is the value of a rank function associated with w.
To describe the rank conditions, it is enough to consider the special case where X is the odd orthogonal flag variety, and Ω w is a Schubert variety of codimension equal to the length of w. 1 Let us assume the flag F • comes from a standard basis e n , . . . , e 1 , e 0 , e 1 , . . . , e n (using q to denote −q). That is, extending to a complete flag by setting F 0 = F ⊥ 1 , F 1 = F ⊥ 2 , etc. (so F q = F ⊥ q+1 ), the space F q is spanned by {e i | i ≥ q} for all q. An isotropic flag E • can be represented by a (2n + 1) × n matrix whose rows correspond to the basis vectors e i : labelling the columns n, . . . , 1, the space E p is the span of the columns whose labels are at most p. One can always scale the bottom-most nonzero entry in each column to 1; the remaining entries are either free or determined from the other entries by the isotropicity condition. If the 1's are in positions (w(i), ı) for i = 1, . . . , n, then dim(E p ∩ F q ) = #{i ≤ p | w(i) ≥ q}, and this defines the rank function r w (p, q). The Schubert variety Ω w is defined as the set of all E • such that dim(E p ∩ F q ) ≥ r w (p, q) for 1 ≤ p ≤ n and n ≤ q ≤ n (or, more precisely, by taking the closure of the locus where equality holds). An example is shown in Figure 1, where the entries determined by isotropicity are represented by an "×"; cf. [FP,§6.1]. The diagram of a signed permutation w is a combinatorial abstraction of the matrix representing E • (see Figures 1 and 2). Its definition, given in detail in §1.2, is similar to the diagram of an ordinary permutation, with the additional feature of markings "×"; the white boxes not containing an × form the diagram, while all the white boxes form the extended diagram.
In terms of the matrix, the rank r w (p, q) is equal to the number of dots in the region weakly southwest of the box (q, p). The essential set Ess(w) consists of certain basic triples (k, p, q), where k = r w (p, q) and the box (q − 1, p) is a southeast corner of the (extended) diagram of w. In contrast with ordinary permutations, however, not all southeast corners give essential conditions for a signed permutation. There are two exceptions, stemming from two simple linear-algebraic facts about isotropic subspaces: first, the conditions dim(E p ∩ F q ) ≥ k and dim(E ⊥ p ∩ F ⊥ q ) ≥ k + p + q − 1 are equivalent; and second, for k, p, q > 0 the condition dim(E p ∩ F q ) ≥ k is implied by dim(E p ∩ F ⊥ q ) ≥ k + q − 1. An example is shown in Figure 2. See Definition 1.2 for details.
The key geometric fact about essential sets is the following: Theorem A (Corollary 2.6). The degeneracy locus corresponding to a signed permutation w is defined by the conditions dim(E p ∩ F q ) ≥ k, as (k, p, q) ranges over Ess(w).
Taking X to be the orthogonal flag variety, E • the tautological isotropic flag, and F • a flag of trivial isotropic bundles, Theorem A says the Schubert variety corresponding to w is determined by rank conditions from the essential set. This particular case suffices to prove the general degeneracy locus statement.
The theorem can be interpreted in terms of rank conditions on matrices, as follows. Take a standard basis e n , . . . , e 1 , e 0 , e 1 , . . . , e n for a 6 5 4 3 2 1 0 1 2 3 4 5 6 6 5 4 3 2 1 Figure 2. Diagram and essential rank conditions. In the diagram, each essential position is labelled with the number of dots strictly south and weakly west of it.
vector space of dimension 2n + 1, and fix the symmetric bilinear form defined by e i , e  = δ i,j . Let X be the set of (2n + 1) × n full-rank matrices with isotropic and pairwise orthogonal columns, with rows indexed n, . . . , 0, . . . , n (top to bottom) and columns indexed n, . . . , 1 (left to right).
Corollary. Given a signed permutation w, with rank function r w (p, q) as defined above, let Ω w ⊆ X be the subset where A ≤p, <q is the (upper-left) submatrix on columns n, . . . , p and rows n, . . . , q − 1. Then the conditions nullity(A ≤p, <q ) ≥ k for (k, p, q) ∈ Ess(w) suffice to determine Ω w , and they form a minimal set with this property.
To deduce this from Theorem A, observe that the the flag variety SO 2n+1 /B can be represented as a quotient of the set of matrices X, and the composed map E p ֒→ V ։ V /F q is represented by the submatrix A ≤p, <q . The set Ω w is the preimage of the corresponding Schubert variety under the quotient map. An example is shown in Figure 2.
With Theorem A in mind, we define the basic signed permutation w(k, p, q) associated to a basic triple (k, p, q) so that the Schubert variety Ω w(k,p,q) is defined by a single rank condition (see §2.1 for the combinatorial definition). The geometric statement of Theorem A is a direct consequence of a combinatorial statement about Bruhat order-a signed permutation w is the supremum of the basic signed permutations corresponding to the elements of Ess(w) (Theorem 2.3). This, in turn, relies on our main result about essential sets: Theorem B (Theorem 2.2). The essential set of a signed permutation w corresponds to the set of basic signed permutations which are maximal among all those below w in Bruhat order.
Theorem B also suggests a link with a general poset-theoretic definition of essential set introduced by Reiner, Woo, and Yong in the course of finding descriptions for the cohomology rings of Schubert varieties [RWY]. We establish a bijective correspondence in Proposition 2.10. Along the way, we also show that the set of basic signed permutations is in bijection with the base of the group of signed permutations, which was studied by Lascoux and Schützenberger [LS] and Geck and Kim [GK] in the context of characterizing Bruhat order of Weyl groups.
From an alternative point of view, our diagrammatic definition together with Theorem B may be taken as an efficient method for computing the essential set as defined in [RWY], and hence finding generators for the ideal of the cohomology ring of a Schubert variety. From the perspective of [LS] and [GK], this also leads to an efficient means of comparing elements in Bruhat order on signed permutations.
It would be interesting to see signed analogues of other properties of diagrams for (ordinary) permutations. For instance, "balanced fillings" of the diagram of a permutation are in bijection with reduced words, and this leads to a formula for the corresponding Stanley symmetric function [KLR, FGRS]. Is there a similar story for diagrams of signed permutations? (In [H], a different notion of diagram is proposed, as well as a set of balanced fillings in bijection with reduced words; the tradeoff is that one must label all the boxes of the diagram of a permutation in S 2n , rather than the smaller set of boxes we consider here.) In a different direction, Eriksson and Linusson characterized the sets that can arise as essential sets of permutations, and used this to give an efficient algorithm for reconstructing a permutation from its essential set; in fact, only a subset of the essential set is required [EL]. A result of this kind for signed permutations would be useful, as well.
Much of our discussion of diagrams and essential sets for signed permutations has a "type B" flavor, but this is primarily for notational convenience. The group W n of signed permutations is also the Weyl group of type C, and indeed, one can recast all the results in a form more adapted to the symplectic group; an indication is given at the end of the article. Type D, however, is another matter-the Weyl group is a subgroup of W n , and it does not satisfy the "dissective" condition used in §2.3. It would be interesting to work out the analogous story for type D, but this case seems sufficiently distinct to be treated separately.
Acknowledgements. I am grateful to Sara Billey, Zach Hamaker, and Alexander Woo for helpful comments and suggestions.
The initial impetus for this paper came from the study of degeneracy loci and vexillary signed permutations in an ongoing joint project with William Fulton. I thank him for detailed comments on the manuscript, as well as for an enjoyable and fruitful collaboration.
Notation. Our conventions and notation for signed permutations are very close to those described in [BB,§8.1]. We review them here.
We will consider permutations of (positive and negative) integers . . . , n, . . . , 2, 1, 0, 1, 2, . . . , n, . . . , using the bar to denote a negative sign, and we take the natural order on them, as above. All permutations are finite, in the sense that v(m) = m whenever |m| is sufficiently large. We generally write permutations in S 2n+1 using one-line notation, by listing the values v(n) v(n − 1) · · · v(n).
A signed permutation is a permutation w of these symbols with the property that for each i, w(ı) = w(i). A signed permutation is in W n if w(m) = m for all m > n; this is a group isomorphic to the hyperoctahedral group, the Weyl group of types B n and C n . When writing signed permutations in one-line notation, we only list the values on positive integers: w ∈ W n is represented as w(1) w(2) · · · w(n). For example, w = 2 1 3 is a signed permutation in W 3 , and w(3) = 3 since w(3) = 3. The length of a signed permutation can be computed as • , is 1 2 · · · n, and has length n 2 .
The definition of W n presents it as embedded in the symmetric group S 2n+1 , considering the latter as the group of all permutations of the integers n, . . . , 0, . . . , n. We will write ι : W n ֒→ S 2n+1 for emphasis when a signed permutation is considered as a full permutation. Specifically, ι sends w = w(1) w(2) · · · w(n) to the permutation w(n) · · · w(2) w(1) 0 w(1) w(2) · · · w(n) in S 2n+1 . (There is a similar embedding ι ′ : W n ֒→ S 2n , defined by omitting the value "w(0) = 0".) Using the natural inclusions W n ⊂ W n+1 ⊂ · · · , we have the infinite Weyl group W ∞ = W n , and we will sometimes refer to elements w ∈ W ∞ , when n is understood or irrelevant. The embeddings are compatible with the corresponding inclusions S 2n+1 ⊂ S 2n+3 ⊂ · · · .
A permutation v has a descent at position i if v(i) > v(i + 1); here i may be any integer. The same definition applies to signed permutations w, but we only consider descents at positions i ≥ 0, following the convention of recording the values of w only on positive integers. A descent at 0 simply means that w(1) is negative. For example, w = 2 1 3 has descents at 0 and 2, while ι(w) = 3 1 2 0 2 1 3 has descents at −3, −1, 0, and 2.

Defining the essential set
Our definitions of diagram and essential set for signed permutations are modeled on the analogous notions for permutations, so we briefly review that case first. The conventions we use here are adapted to make the transition to signed permutations simpler, so they differ slightly from those used in [Fu] and elsewhere.
1.1. Essential sets for S 2n+1 . We will consider arrays of boxes with rows and columns indexed by integers {n, . . . , 1, 0, 1, . . . , n}. The permutation matrix associated to v ∈ S 2n+1 has dots in positions (v(i), i), for n ≤ i ≤ n, and is empty otherwise. The diagram of v is the collection of boxes that remain after striking out those which are (weakly) south or east of a dot in the permutation matrix. The number of boxes in the diagram is equal to the length of the permutation.
The rank function of a permutation is defined by for n ≤ p, q ≤ n. This is also the number of dots strictly south and weakly west of the box (q − 1, p) in the permutation matrix of v. Adapting terminology from [Fu] and [RWY], the essential positions of v are the pairs (p, q) such that the box (q − 1, p) is a southeast (SE) corner of the diagram of v. (This apparently awkward notation is easier to understand in terms of the geometry explained in the next paragraph, together with the diagrams.) The essential set of v is the set Ess(v) of (k, p, q) such that (p, q) is an essential position, and k = r v (p, q); it is obviously in bijection with the set of essential positions of v, but it will be useful to preserve the extra information of the rank function.
It is often helpful to think of flags of subspaces E • and F • in a (2n+1)dimensional vector space V , with basis e n , . . . , e 1 , 0, e 1 , . . . , e n . The n × n array corresponds to a matrix with respect to this basis. The space E p is spanned by the columns labelled p, p + 1, . . .-i.e., from p to the left-and the space F q is spanned by the basis vectors e q , e q+1 , . . .i.e., from q down. The number k = r v (p, q) records the dimension of E p ∩ F q ; it is also the corank of the map E p → V /F q , and the box (q − 1, p) is the SE corner of submatrix corresponding to this map. Figure 3 illustrates these notions, for v = 1 3 2 0 2 3 1, with the numbers r v (p, q) placed in the boxes (q − 1, p) corresponding to the essential positions. The essential set is An equivalent, numerical description of essential positions is useful: This is also equivalent to requiring the conditions In other words, v has a descent at b, with a lying in the interval of the jump, and v −1 has a descent at a, with b lying in the interval of the jump. The essential positions are recovered as those (p, q) such that (q − 1, p) satisfies (2), and the essential set is recovered by including k = r v (p, q).
A basic triple (of type A) is a triple of integers (k, p, q) such that k > max{0, 1 − p − q}. It is easy to see that the elements of Ess(v) are basic triples.
1.2. Essential sets for W n . By convention, a signed permutation w is written in terms of its values on positive integers, but since w(ı) = w(i), it is also determined by its values on negative integers. Given w ∈ W n , its corresponding matrix is the (2n + 1) × n array of boxes, with rows indexed by {n, . . . , n} and columns indexed by {n, . . . , 1}, with dots in the boxes (w(i), i) for n ≤ i ≤ 1. For each dot, we also place an "×" in the same column and opposite row, as well as in the boxes to the right of this ×. (To be precise, an × is placed in those boxes (a, b) such that a = w(i) for some i ≤ b. This is similar to the matrix representation of Schubert cells described in [FP,§6.1].) The extended diagram D + (w) of a signed permutation w is the collection of boxes in the (2n + 1) × n rectangle that remain after striking out those which are south or east of a dot. The diagram D(w) ⊆ D + (w) is the subset of boxes of the extended diagram that are not marked with an ×. As for permutations, the number of boxes of D(w) is equal to the length of w. In fact, the inversions of w are in bijection with the boxes of D(w): writing the positive simple roots for B n as ǫ 1 , ǫ 2 − ǫ 1 , . . . , ǫ n − ǫ n−1 , and using the convention The matrix and extended diagram come from the corresponding notions for ordinary permutations, via the embeddeding ι : W n ֒→ S 2n+1 : the matrix and extended diagram of w ∈ W n correspond to the first n columns of the matrix and diagram for ι(w). See Figure 4, where the unshaded boxes are those of D + (w).
The rank function of a signed permutation is defined by for 1 ≤ p ≤ n and n ≤ q ≤ n. Using the symmetry w(ı) = w(i), this is also equal to #{i ≤ p | w(i) ≥ q}, so the rank functions r w and r ι(w) agree where both are defined. Integers (k, p, q) form a basic triple (of type B) if the same inequality as for type A is satisfied, i.e., k > max{0, 1 − p − q}, together with three additional conditions: p > 0, q = 0, and if p = 1 then q > 0.
Before discussing essential sets for signed permutations, we record a simple observation. When v = ι(w) lies in the image of ι : W n ֒→ S 2n+1 , its essential set is symmetric: the lower-right corners of the boxes corresponding to essential positions are reflected about the center. (The example of w = 2 3 1 is shown in Figure 3, where these corners are circled.) To describe this symmetry more precisely, given a basic triple (k, p, q) of type A, let (k, p, q) ⊥ = (k + p + q − 1, p + 1, q + 1) be its reflection; this defines an involution on basic triples. Figure 4. Matrix, diagram, and essential set for a signed permutation. (As a visual aid, the center row is indicated with dashed lines.) Proof. First, we show that the SE corners of the diagram are centrally symmetric.
These are the conditions for (a + 1, b + 1) to be a SE corner corresponding to an essential position.
This shows that (k, p, q) is in Ess(v) if and only if (k ′ , p + 1, q + 1) is in Ess(v), where k ′ = r v (p + 1, q + 1). It remains to check that this rank is equal to k + p + q − 1, as claimed. This is easily done by examining a permutation matrix and using The essential set of a signed permutation w is a subset of the essential set of the corresponding permutation ι(w).
Definition 1.2. Let w be a signed permutation. The essential set of w is the set Ess(w) of triples of integers (k, p, q), with k = r w (p, q), such that (q −1, p) is a SE corner of the extended diagram D + (w), with two exceptions: a SE corner does not contribute to Ess(w) if (i) it is in the rightmost column and (strictly) above the center row; or (ii) it is not in the rightmost column and there is another SE corner above it, in the same column and "opposite" row-that is, in box (q, p)-and the two rank conditions differ by exactly q − 1. The essential positions of w are those pairs (p, q) such that (k, p, q) ∈ Ess(w), for k = r w (p, q). (As before, Ess(w) is clearly in bijection with the set of essential positions.) The two exceptions can be phrased more formally as follows: (i) if p = 1 and q < 0, then (k, p, q) is not in Ess(w); (ii) if p > 1 and q > 0, and both (q −1, p) and (q, p) are SE corners, with k = r w (p, q) = r w (p, q + 1) − q + 1, then (k, p, q) is not in Ess(w). The first exception is easily understood: w(0) = 0 prevents the diagram of ι(w) from having SE corners in this region, so such (k, p, q) are not included in Ess(ι(w)). (Compare Figures 3 and 4.) Noting this, it is easy to see that Ess(w) ⊆ Ess(ι(w)), and consequently, each (k, p, q) ∈ Ess(w) is a basic triple of type B. (In fact, this also explains why those (k, p, q) with p = 1 and q < 0 are excluded in the definition of basic triples of type B.) The second exception has a geometric explanation. For any k, p, q > 0, the condition dim(E p ∩ F q ) ≥ k is implied by the condition dim(E p ∩ F ⊥ q ) ≥ k + q − 1, because an isotropic subspace of F ⊥ q /F q must have dimension no greater than q − 1. The former condition is therefore redundant, and should be excluded from the essential set.

Rank conditions and Bruhat order
The main results of this section show that essential sets determine the corresponding signed permutations, and can be used to make comparisons in Bruhat order. We also establish the connection with the general poset-theoretic essential sets of [RWY]. The statements are primarily combinatorial, but since some of the terminology and motivation is geometric, we begin by briefly reviewing the context.
Let V be an odd-dimensional vector space, say dim V = 2n + 1, and fix a complete flag of subspaces with dim F i = n + 1 − i. Each permutation v ∈ S 2n+1 determines a Schubert variety in the complete flag variety, defined by The rank conditions in this definition are highly redundant. Up to a change of notation, the results of [Fu,§3] show that it is sufficient to restrict to those (p, q) which are essential positions for v. Now equip V with a nondegenerate quadratic form, and assume F • is isotropic. A signed permutation w ∈ W n defines a Schubert variety in the orthogonal flag variety, by a similar prescription: We will see that the rank conditions dim(E p ∩ F q ) ≥ k for (k, p, q) ∈ Ess(w) suffice to define Ω w . (As mentioned in the introduction, the same discussion applies to degeneracy loci, with the flag varieties replaced by an arbitrary base variety.) 2.1. Basic permutations and signed permutations. A basic triple (k, p, q) of type A defines a basic permutation v(k, p, q), which is the minimal element in Bruhat order such that r v (p, q) ≥ k. It can be written down as follows: start at position p, and, proceeding right to left, place k entries in descending order, ending with q; then fill in the remaining entries in increasing order from left to right. For example, v(3, 1, 2) = 4 3 2 2 3 4 1 0 1. (The three entries placed in the first step are at positions 1, 0, and 1, shown in bold.) Recall that a permutation v is grassmannian if it has exactly one descent, and v is bigrassmannian if both v and v −1 are grassmannian. From the construction, v(k, p, q) is grassmannian, having a single descent at p. Moreover, v −1 is equal to v(k, q + 1, p + 1), so v is bigrassmannian. In fact, all bigrassmannian permutations arise this way: the basic permutations are exactly the bigrassmannian ones (see, e.g., [RWY,Lemma 4.1]).
It follows from results of Kobayashi [Ko] and Reiner, Woo, and Yong [RWY] that for a permutation v, the essential set Ess(v) coincides with the set of those basic triples (k, p, q) such that v(k, p, q) is maximal among all basic permutations below v in Bruhat order on the symmetric group.
We will provide an analogue of this fact for signed permutations. Parallel to the type A situation, the basic signed permutation corresponding to a basic triple of type B is the element w = w(k, p, q) which is minimal (in Bruhat order), such that r w (p, q) = k. It is not immediately obvious that such a minimum is unique, though it is not hard to prove (see Lemma 2.5). To write w(k, p, q), start at position p, and place k consecutive entries, in increasing order, ending with q; then fill in the unused positive entries in increasing order. If 0 < q < k, then "q − 1, 1" counts as "consecutive". For example, w(2, 2, 3) = 1 4 3 2, and w(3, 2, 2) = 4 3 1 2.
The assignment of a basic signed permutation to a basic triple is one-to-one. The smallest n such that w(k, p, q) ∈ W n is n(k, p, q) = max{p + k − 1, q + k − 1}. This makes it easy to enumerate the basic elements of W n : we have (5) #{w(k, p, q) ∈ W n } = 2n 3 + n 3 .
We now come to the poset-theoretic characterization of the essential set. For v ∈ S ∞ , the main theorem of [Ko] identifies Ess(v) with the set of type A basic triples (k, p, q) such that v(k, p, q) is maximal among all basic permutations below v in Bruhat order. We have an analogue of this for signed permutations: Theorem 2.2. Let w be a signed permutation. The essential set Ess(w) is equal to the set of basic triples (k, p, q) such that w(k, p, q) is maximal among all basic signed permutations below w in Bruhat order.
In light of the theorem, when no confusion seems likely, we will identify Ess(w) with the corresponding set of basic signed permutations w(k, p, q).
The proof of Theorem 2.2 is postponed to §3. In the rest of this section, we elaborate on some consequences and related facts.
An important property of the essential set is that Ess(w) determines w. To state this more precisely, we use the notion of supremum in a poset. Given a poset P and a subset Y ⊆ P , an element x ∈ P is the supremum of Y if x ≥ y for all y ∈ Y , and if x ′ ≥ y for all y ∈ Y , then x ′ ≥ x. (See [GK,§2.4].) A supremum is clearly unique if it exists; when it does exist, write x = sup(Y ).
The following is an analogue of [Fu,Lemmas 3.10 and 3.14].
Theorem 2.3. Let w be a signed permutation.
(i) w is the supremum of its essential set.
(ii) The essential set Ess(w) is minimal with the property that w = sup(Ess(w)); in other words, w = sup(Y ) for any Y Ess(w). More precisely, choose any (k 0 , p 0 , q 0 ) ∈ Ess(w). Then one can find an element w ′ ∈ W n such that r w ′ (q 0 , p 0 ) < k 0 , and (iii) If Y is any set of basic signed permutations such that sup(Y ) = w, then Y ⊇ Ess(w).
In particular, Ess(w) is the unique minimial set of basic elements whose supremum is w.
In the course of proving this, we will establish a connection with the base of the Coxeter group W n , as well as the notion of essential set considered in [RWY]. This is done in §2.3. First, we briefly digress to describe the connection with geometry.
2.2. Rank conditions. We will use the notation C(k, p, q) to denote the rank condition dim(E p ∩ F q ) ≥ k. The basic triples (k, p, q) of type A (defined in §1.1) correspond to rank conditions C(k, p, q) which are nontrivial and for which equality is feasible; similarly, the basic triples of type B ( §1.2) correspond to rank conditions which are nontrivial and feasible for isotropic subspaces.
Using the correspondence between Bruhat order and containment of Schubert varieties, the fact that v(k, p, q) is the minimal permutation in S ∞ satisfying r v (p, q) ≥ k says the Schubert variety Ω v(k,p,q) is defined by the single rank condition C(k, p, q) inside the complete flag variety. A little more generally, we have the following standard fact (cf. [RWY,Lemma 4

.1 and Proposition 4.6]):
Lemma 2.4. Let (k, p, q) and (k ′ , p ′ , q ′ ) be basic triples (of type A). The rank condition C(k, p, q) implies C(k ′ , p ′ , q ′ ) if and only if v(k, p, q) ≥ v(k ′ , p ′ , q ′ ). Furthermore, for a general permutation v, the condition C(k, p, q) holds on the Schubert variety Ω v if and only if v ≥ v(k, p, q).
The analogous statement for signed permutations and rank conditions on isotropic flags is a consequence of the following lemma.
Lemma 2.5. For a basic triple (k, p, q) of type B, the corresponding basic signed permutation w(k, p, q) is the unique minimum among elements w ∈ W n such that r w (p, q) ≥ k.
Indeed, Lemma 2.5 is equivalent to the statement: the rank condition C(k, p, q) holds on Ω w if and only if w ≥ w(k, p, q). We give a proof using a geometric construction.
Proof. We claim that the subvariety of the orthogonal flag variety defined by the rank condition C(k, p, q) is irreducible and has codimension equal to ℓ(w(k, p, q)). It is therefore equal to the corresponding Schubert variety Ω w(k,p,q) , so the statement about Bruhat order follows.
To prove the claim, it is enough to show the corresponding locus in the orthogonal Grassmannian is irreducible of the correct codimension. This is an exercise in resolving singularities. Let Z ⊆ X = OG(n + 1 − p, 2n+1) be the (reduced) subset defined by dim(E p ∩F q ) ≥ k, where F • is a fixed isotropic flag. This is the closure of the locus Z • where equality holds. If q > 0, then F q is isotropic. In this case, let Z ⊂ X × Gr(k, F q ) be pairs (E p , L) such that L ⊆ E p and L is a k-dimensional subspace of F q . The first projection has image Z, and it is an isomorphism over the open subset Z • . The second projection is surjective onto Gr(k, F q ), with orthogonal Grassmannians OG(n+ 1 −p −k, 2n−2k + 1) as fibers; in particular, Z is smooth and irreducible. Comparing the dimensions of Z and X, we find that Z has codimension (p + q − 1)k + k 2 , which is equal to ℓ(w(k, p, q)) as claimed. The cases k > q > 0 and q ≥ k are similar.
Lemma 2.5 and Theorem 2.3 imply a shorter description of Schubert varieties in the orthogonal flag variety.
Corollary 2.6. The Schubert variety Ω w is defined by the rank conditions C(k, p, q) for basic triples (k, p, q) in Ess(w), and this is a minimal list of rank conditions. [LS], the base of a finite Coxeter group W (or any finite poset) is the set of nonidentity elements x which cannot be written as the supremum of any subset Y ⊆ W not containing x. 2 This notion was developed further by Geck and Kim [GK], who identify the base for each finite Coxeter group, and also by Reading [Re].

Bases. As defined by Lascoux and Schützenberger
In type A, the base of the symmetric group is the set of all bigrassmannian permutations ( [LS,Théorème 4.4]). Below we will prove that the basic elements w(k, p, q) form the base of W n , giving a new description of the base (cf. [GK,Theorem 4.6] for another description).
Remark 2.7. In the first statement of [LS,Théorème 7.4], it is claimed that the base of W n can also be described as those w such that ι(w) is the supremum of elements v(k, p, q) and v(k, p, q) ⊥ in S 2n+1 , for some type A basic triple (k, p, q). 3 However, the example of w(2, 2, 1) = 3 2 1 shows that this is not the case. The gap lies in [LS,Lemme 7.3], which is clarified below in the discussion following Lemma 3.1.
A finite Coxeter group (or poset) is dissective if for every element x of the base, the complement W {x ′ | x ′ ≥ x} has a unique maximal element u; in other words, there is a disjoint decomposition If such a u exists, we call it the dissecting element associated to x. The groups S n and W n (of types A and B) are dissective, but the Weyl group of type D is not [LS, GK].
Given a basic triple (k, p, q), and an integer n ≥ n(k, p, q) (so that w(k, p, q) ∈ W n ), set u(k, p, q, n) = w(n + 2 − p − k, p, q + 1) · w (n) • , replacing "q + 1" with "1" when q = 1. This is the maximal element of W n with r u (q, p) = k − 1. Note that although the basic signed permutation w(k, p, q) does not depend on n, the element u(k, p, q, n) necessarily does. We will soon see that these are the dissecting elements. First, we give another characterization of these signed permutations.
Lemma 2.8. Let (k, p, q) be a basic triple of type B, and fix an integer n with n ≥ n(k, p, q) := max{p + k − 1, q + k − 1}. The element u(k, p, q, n) is the unique maximum among elements w ∈ W n such that r w (p, q) < k.
Equivalently, C(k, p, q) fails on Ω w if and only if w ≤ u(k, p, q, n). This is an easy consequence of Lemma 2.5, using the involution w (n) • . An alternative characterization of the base is given in [GK,§2.4]: it is the set of elements w which are minimal in the complement of an interval {u ′ | u ′ ≤ u}, for some u. In general, this u is not required to be uniquely determined by w, but for dissective posets, there is a unique maximal such u for each w.
Proposition 2.9. The basic signed permutations in W n form the base of W n .
Proof. The number of basic signed permutations is equal to the cardinality of the base (see [LS,p. 24,Remarque] or [GK,p. 300]), so we only need to establish one inclusion. We will show that every basic signed permutation lies in the base.
Since every element w ∈ W n has either r w (p, q) ≥ k or r w (p, q) < k, Lemmas 2.5 and 2.8 show that In particular, w(k, p, q) is the (unique) minimal element in the complement of the interval {w | w ≤ u(k, p, q, n)}, so it lies in the base.
The proof also shows that W n is dissective: since every element of the base is of the form w(k, p, q), the fact that u(k, p, q, n) is the unique maximum in the complement of {w | w ≥ w(k, p, q)} shows that u(k, p, q, n) is the dissecting element corresponding to w(k, p, q).
We now turn to Theorem 2.3.
Proof of Theorem 2.3. In general, an element w of a finite poset is equal to the supremum of the set of base elements lying below w [GK,§2.4]. It is sufficient to include the maximal such elements, and by Theorem 2.2 together with Proposition 2.9, for w ∈ W n , these constitute Ess(w); therefore w = sup(Ess(w)). For the second statement in (ii), concerning minimality, observe that w(k 0 , p 0 , q 0 ) ≤ w(k, p, q) for any (k, p, q) ∈ Ess(w) {(k 0 , p 0 , q 0 )}, because by Theorem 2.2, elements of the essential set are incomparable. By the dissective property, it follows that w(k, p, q) ≤ u(k 0 , p 0 , q 0 , n) for all such (k, p, q), and therefore the intersection is nonempty. Any element of this intersection can be taken as the desired w ′ ; for instance, w ′ = u(k 0 , p 0 , q 0 , n) works.

A bijection.
Finally, we make the relationship with the essential set of [RWY] precise, for both permutations and signed permutations.
For the purposes of this subsection only, let us consider permutations in S n as acting on {1, . . . , n} in the usual way; the definitions and conventions for signed permutations in W n remain as before. This results in some changes of notation for type A, but no further complications. The rank function for v ∈ S n is redefined as r v (p, q) = #{i ≤ p | v(i) > q}, and basic triples of type A have k, p, q > 0 and p ≥ k > p − q. In one-line notation, the corresponding basic permutation v(k, p, q) is v(k, p, q) = 1, · · · , p − k, q + 1, · · · q + k, p − k + 1, · · · , q.
As mentioned before, basic permutations are bigrassmannian, using the fact that the inverse of v(k, p, q) is equal to v(p + q − k, q, p), and they account for all bigrassmannians in S n .
The assignment of v(k, p, q) to a basic triple is one-to-one. It is easy to describe the inverse map: Given a bigrassmannian permutation, record the position of the unique descent as p; starting at this position, collect all the consecutive entries to the left of it-let k be the number of them, and let q +1 be the smallest one. Furthermore, the smallest n such that v(k, p, q) lies in S n is q + k. These two facts make it easy to enumerate the basic permutations in S n (or equivalently, the bigrassmannians)there are 1 6 (n 3 − n) = n+1 3 of them. We will need the dissecting elements for S n , which are defined similarly to the elements u(k, p, q, n) for W n . Given a basic triple (of type A) (k, p, q) and n ≥ q + k, so v(k, p, q) ∈ S n , set t(k, p, q, n) = v(n + 1 − q − k, n − p, q) · w (n) • . (This is the maximal element in S n having r t (q, p) < k.) The diagram of a permutation in S n is defined as usual, via the permutation matrix with dots in positions (v(i), i). The essential set of v ∈ S n is the set of basic triples (k, p, q) such that (q, p) is a SE corner of the diagram of v and k = r v (p, q).
Given a permutation v ∈ S n , let E RWY (v) denote the set of all permutations t ∈ S n which are minimal (in Bruhat order) among those not below v. For w ∈ W n , define E RWY (w) analogously. We write w (n) • for the longest element of S n or W n , depending on context. Proposition 2.10. Fix n.
(i) For any v ∈ S n , the map sending (k, p, q) ∈ Ess(vw (n) (ii) For any w ∈ W n , the map sending (k, p, q) ∈ Ess(ww (n) The proof is a translation of the argument in [RWY,Proposition 4.6]. The factor of w • arises from the fact that their Schubert varieties X v correspond to our Ω vw• ; furthermore, our E p is their (C n /V n−p ) * . (Combining these two identifications explains why X v appears to be identified with Ω w•v in [RWY].) Similarly, their X w is our Ω ww• in the orthogonal flag variety.
3. Proof of Theorem 2.2 Some temporary notation will be useful. Let B be the set of all type B basic triples. For any w ∈ W ∞ , let M (w) be the set of type B basic triples (k, p, q) such that w(k, p, q) is maximal among all basic signed permutations below w in Bruhat order. Thus we must show M (w) = Ess(w).
Defining M (v) analogously for v ∈ S ∞ , we have M (ι(w)) = Ess(ι(w)) by [Ko] (or [RWY,Proposition 4.6]). From the definitions, Ess(w) is contained in Ess(ι(w)) ∩ B. To prove the theorem, we will show that M (w) is also contained in Ess(ι(w))∩B, and then show that the exceptions are the same as the ones specified in the definition of Ess(w) (Definition 1.2). The key ingredient is a comparison of the Bruhat orders on S ∞ and W ∞ , so we begin with two lemmas concerning this. Throughout, we implicitly use the fact that w ≤ w ′ in W ∞ iff ι(w) ≤ ι(w ′ ) in S ∞ -see, e.g., [BB,Corollary 8.1.9]. (The "if" statement fails for the type D Weyl group, giving another hint that this case is more complicated.) Suppose (k, p, q) and (k ′ , p ′ , q ′ ) are two basic triples of type B, hence also of type A. It is easy to see that if v(k ′ , p ′ , q ′ ) ≥ v(k, p, q) in type S ∞ , then w(k ′ , p ′ , q ′ ) ≥ w(k, p, q) in W ∞ . The converse is not true, but the exceptions are easily classified.
To prove the lemma, therefore, we need to compare the rank functions of v(k ′ , p ′ , q ′ ) and w(k ′ , p ′ , q ′ ). The hypothesis is that When q ′ ≤ −k ′ , the negative columns of the permutation matrices for w(k ′ , p ′ , q ′ ) and v(k ′ , p ′ , q ′ ) are the same, and so the corresponding rank functions agree; the inequality (6) cannot hold in this case.
Consider −k ′ < q ′ < 0. In this case the rank functions for w = w(k ′ , p ′ , q ′ ) and v = v(k ′ , p ′ , q ′ ) agree outside the region where 0 < p, q ≤ k ′ + q ′ + 1 and p + q ≤ p ′ + k ′ . Inside this region, r w (p, q) > r v (p, q), so one can choose k so that (6) holds. However, in this region r w (p, q) = r v(k ′ +q ′ ,p ′ ,q ′ +1) (p, q), so the lemma is true in this case. (See Figure 6 for an illustration.) Finally, assume q ′ > 0. As in the previous case, the rank functions for w = w(k ′ , p ′ , q ′ ) and v = v(k ′ , p ′ , q ′ ) agree outside the region where 0 < p < q ′ and −k ′ − q ′ + 1 < q ≤ −p. Inside this region, r w (p, q) > r v (p, q), but r w (p, q) = r v(k ′ ,p ′ ,q ′ ) ⊥ (p, q), so the lemma is true in this case as well. The above lemma and its proof show that w(k, p, q) (This clarifies the claim before [LS,Lemme 7.3], where the first case is missing.) Lemma 3.2. Given a signed permutation w, suppose (k, p, q) and (k ′ , p ′ , q ′ ) are basic triples of type B, and are in the (type A) essential set Ess(ι(w)). Assume that w(k ′ , p ′ , q ′ ) ≥ w(k, p, q) in W n . Then (i) p = p ′ ; (ii) q ′ < 0 and q = q ′ + 1; and (iii) k = k ′ + q ′ .
Proof. By assumption, w(k ′ , p ′ , q ′ ) ≥ w(k, p, q) in W ∞ , but v(k ′ , p ′ , q ′ ) ≥ v(k, p, q) in S ∞ , since elements of an essential set are incomparable. Therefore Lemma 3.1 applies. In each case of that lemma, a (type A) basic triple is asserted to be greater than (k, p, q); again because distinct elements of the essential set are incomparable, the asserted inequality must actually be an equality. In the first case, this yields the above conclusion. The second case of Lemma 3.1 cannot hold, because (k ′ , p ′ , q ′ ) ⊥ = (p ′ +q ′ +k ′ −1, p ′ +1, q ′ +1) is not a type B basic triple.

A variation for type C
The group W n of signed permutations is isomorphic to the (type C) Weyl group of Sp 2n , as well as to the (type B) Weyl group of SO 2n+1 . The main results above are intrinsic to W n , and so do not depend on whether one regards it as type B or type C, but it is sometimes useful to have diagrams adapted to the type C context. Here we briefly sketch the modifications; all the results and proofs can easily be reformulated.
We will use the embdedding of Weyl groups ι ′ : W n ֒→ S 2n , which omits the value w(0) = 0. For a signed permutation w ∈ W n , one forms a matrix by placing dots in a 2n × n array of boxes as before: labelling the rows {n, . . . , 1, 1, . . . , n} and the columns {n, . . . , 1}, dots are placed in the boxes (w(i), i) for n ≤ i ≤ 1 (or equivalently, (w(i), ı) for 1 ≤ i ≤ n). An "×" is placed in boxes (a, b) such that a = w(i) for some i < b; that is, for each dot, place an × in the columns strictly to the right, and rows opposite the dot. This should be compared with the parametrization of Schubert cells in the type C flag variety Sp 2n /B given in [FP,§6.1].
The type C extended diagram is the collection of boxes D + C (w) in the 2n × n rectangle which remain after striking out those to the right or below a dot; the type C diagram is the subset D C (w) ⊆ D + C (w) of boxes not marked by an ×. As before, the number of boxes in D C (w) is equal to ℓ(w)-the boxes (i, j) of D C (w) correspond to inversions ǫ i − ǫ w(j) of w, using 2ǫ 1 , ǫ 2 − ǫ 1 , . . . , ǫ n − ǫ n−1 as the simple roots for type C n . The essential set is determined exactly as in Definition 1.2, except that when q = 1, we understand q − 1 to mean 1. (Thus if D + C (w) has a SE corner in position (1, 2), it determines an essential position (p, q) = (2, 1).) Examples are shown in Figure 8. Just as for type B, the essential set gives a short list of rank conditions defining a symplectic degeneracy locus (or Schubert variety in Sp 2n /B). To set up notation, let e n , . . . , e 1 , e 1 , . . . , e n be a basis for a vector space V , and define a symplectic form by setting e i , e j = 0 and e ı , e j = δ i,j , for i, j > 0. The standard isotropic flag F • is defined by taking F q to be the span of e i , for i ≥ q. As before, the Schubert variety Ω w is defined as The analogue of Corollary 2.6 is this: Theorem 4.1. The type C Schubert variety Ω w ⊆ Sp 2n /B is defined by the minimal list of rank conditions dim(E p ∩ F q ) ≥ k, as (k, p, q) ranges over Ess(w).
The statement for degeneracy loci is as follows. Let V be an evenrank vector bundle on a variety X, equipped with a symplectic form and two isotropic flags V ⊃ E 1 ⊃ E 2 ⊃ · · · and V ⊃ F 1 ⊃ F 2 ⊃ · · · ; set F q = F ⊥ q+1 . The degeneracy locus Ω w ⊆ X associated to a signed permutation is defined by the same rank conditions as the Schubert variety.