On symmetries in phylogenetic trees

Billey et al. [arXiv:1507.04976] have recently discovered a surprisingly simple formula for the number $a_n(\sigma)$ of leaf-labelled rooted non-embedded binary trees (also known as phylogenetic trees) with $n\geq 1$ leaves, fixed (for the relabelling action) by a given permutation $\sigma\in\frak{S}_n$. Denoting by $\lambda\vdash n$ the integer partition giving the sizes of the cycles of $\sigma$ in non-increasing order, they show by a guessing/checking approach that if $\lambda$ is a binary partition (it is known that $a_n(\sigma)=0$ otherwise), then $$ a_n(\sigma)=\prod_{i=2}^{\ell(\lambda)}(2(\lambda_i+\cdots+\lambda_{\ell(\lambda)})-1), $$ and they derive from it a formula and random generation procedure for tanglegrams (and more generally for tangled chains). Our main result is a combinatorial proof of the formula, which yields a simplification of the random sampler for tangled chains.


Introduction
For A a finite set of cardinality n ≥ 1, we denote by B[A] the set of rooted binary trees that are non-embedded (i.e., the order of the two children of each node does not matter) and have n leaves with distinct labels from A. Such trees are known as phylogenetic trees, where typically A is the set of represented species.Note that such a tree has n − 1 nodes and 2n − 1 edges (we take here the convention of having an additional root-edge above the root-node, connected to a 'fake-vertex' that does not count as a node, see Figure 1).The tree γ = σ • γ, with σ = (1, 4, 3)(5) (2,6).Since γ = γ, γ is not fixed by σ (on the other hand γ is fixed by (2, 3) (1,4,6,5)).
The group S(A) of permutations of A acts on B[A]: for γ ∈ B[A] and σ ∈ S(A), σ • γ is obtained from γ after replacing the label i of every leaf by σ(i), see ) and e is an edge of γ (among the 2n − 1 edges).Define the cycle-type of σ as the integer partition λ n giving the sizes of the cycles of σ (in non-increasing order).For λ n an integer partition, the cardinality of B σ [A] is the same for all permutations σ with cycle-type λ, and this common cardinality is denoted by r λ .It is known (e.g. using cycle index sums [1,3]) that r λ = 0 unless λ is a binary partition (i.e., an integer partition whose parts are powers of 2).Billey et al. [2] have recently found the following remarkable formula, valid for any binary partition λ: (1) They prove the formula by a guessing/checking approach.Our main result here is a combinatorial proof of (1), which yields a simplification (see Section 3) of the random sampler for tanglegrams (and more generally tangled chains) given in [2].
Theorem 1.For A a finite set and σ a permutation on A whose cycle-type is a binary partition: • If σ has more than one cycle, let c be a largest cycle of σ; denote by A the set A without the elements of c, and denote by σ the permutation σ restricted to A .Then we have the combinatorial isomorphism As we will see, the isomorphism (2) can be seen as an adaptation of Rémy's method [7] to the setting of (non-embedded rooted) binary trees fixed by a given permutation.Note that Theorem 1 implies that the coefficients r λ satisfy r λ = 1 if λ is a binary partition with one part and r λ = (2|λ\λ 1 | − 1) • r λ\λ1 if λ is a binary partition with more than one part, from which we recover (1).

Proof of Theorem 1
2.1.Case where the permutation σ has one cycle.The fact that |B σ [A]| = 1 if σ has one cycle of size 2 k (for some k ≥ 0) is well known from the structure of automorphisms in trees [6], for the sake of completeness we give a short justification.Since the case k = 0 is trivial we can assume that k ≥ 1.Let c 1 , c 2 be the two cycles of σ 2 (each of size 2 k−1 ), with the convention that c 1 contains the minimal element of A; denote by A 1 , A 2 the induced bi-partition of A, and by which implies |B σ [A]| = 1 by induction on k (note that, also by induction on k, the underlying unlabelled tree is the complete binary tree of height k).The two cases for removing a 2-cycle of leaves (depending whether the two leaves have the same parent or not).The vertices depicted gray are allowed to be the fake vertex above the root-node.

2.2.
Case where the permutation σ has more than one cycle.Let k ≥ 0 be the integer such that the largest cycle of σ has size 2 k .A first useful remark is that σ induces a permutation of the edges (resp. of the nodes) of γ, and each σ-cycle of edges (resp. of nodes) has size 2 i for some i ∈ [0..k].We present the proof of (2) progressively, treating first the case k = 0, then k = 1, then general k.
Case k = 0.This case corresponds to σ being the identity, so that hence we just have to justify that B[A] E[A\{i}] for each fixed i ∈ A. This is easy to see using Rémy's argument [7] 1 , used here in the non-embedded leaf-labelled context: every γ ∈ B[A] is uniquely obtained from some (γ , e) ∈ E[A\{i}] upon inserting a new pending edge from the middle of e to a new leaf that is given label i, see Figure 2(a).
Case k = 1.Let c = (a 1 , a 2 ) be the selected cycle of σ, with a 1 < a 2 .Two cases can arise (in each case we obtain from γ a pair (γ , e) with γ ∈ B σ [A ] and e an edge of γ ): • if a 1 and a 2 have the same parent v, we obtain a reduced tree γ ∈ B σ [A ] by erasing the 3 edges incident to v (and the endpoints of these edges, which are a 1 , a 2 , v and the parent of v), and we mark the edge e of γ whose middle was the parent of v, see the first case of Figure 2(b) • if a 1 and a 2 have distinct parents, we can apply the operation of Figure 2(a) to each of a 1 and a 2 , which yields a reduced tree γ ∈ B σ [A ].We then mark the edge e of γ whose middle was the parent of a 1 , see the second case of Figure 2(b).Conversely, starting from (γ , e) ∈ E[A ], the σ -cycle of edges that contains e has either size 1 or 2: • if it has size 1 (i.e., e is fixed by σ ), we insert a pending edge from the middle of e and leading to "cherry" with labels (a 1 , a 2 ), • if it has size 2, let e = σ (e); then we attach at the middle of e (resp.e ) a new pending edge leading to a new leaf of label a 1 (resp.a 2 ).
The general case k ≥ 0. Recall that the marked cycle of σ is denoted by c.A node or leaf of the tree is generically called a vertex of the tree.We define a c-vertex as a vertex v of γ such that: • if v is a node then all leaves that are descendant of v are in c.
A c-vertex is called maximal if it is not the descendant of any other c-vertex; define a c-tree as a subtree formed by a maximal c-vertex v and its hanging subtree (if v is a leaf then the corresponding c-tree is reduced to v).Note that the maximal c-vertices are permuted by σ.Moreover since the leaves of c are permuted cyclically, the maximal c-vertices actually have to form a σ-cycle of vertices, of size 2 i for some i ≤ k; and in each c-tree, σ 2 i permutes the 2 k−i leaves of the c-tree cyclically.Let be the leaf of minimal label in c, and let w be the maximal c-vertex such that the c-tree at w contains .We obtain a reduced tree γ ∈ B σ [A ] by erasing all c-trees and erasing the parent-edges and parent-vertices of all maximal c-vertices; and then we mark the edge e of γ whose middle was the parent of w, see Figure 3.
Conversely, starting from (γ , e) ∈ E σ [A ], let i ∈ [0..k] be such that the σ -cycle of edges that contains e has cardinality 2 i ; write this cycle as e 0 , . . ., e 2 i −1 , with e 0 = e.Starting from the element of c of minimal label, let (s 0 , . . ., s 2 i −1 ) be the 2 i (successive) first elements of c.And for r ∈ [0..2 i − 1] let c r be the cycle of σ 2 i that contains s r , and let A r be the set of elements in c r (note that A 0 , . . ., A 2 i −1 each have size 2 k−i and partition the set of elements in c).Let T r be the unique (by Section 2.1) tree in B[A r ] fixed by the cyclic permutation c r .We obtain a tree γ ∈ B σ [A] as follows: for each r ∈ [0..2 i − 1] we create a new edge that connects the middle of e r to a new copy of T r .
To conclude we have described a mapping from B σ [A] to E σ [A ] and a mapping from E σ [A ] to B σ [A] that are readily seen to be inverse of each other, therefore n .Then it follows from Burnside's lemma (see [2] for a proof using double and [3] for a proof using the formalism of species) that ( 4)

Application to the random generation of tangled chains
where z λ = 1 m1 m 1 !• • • r mr m r !if λ has m 1 parts of size 1,...,m r parts of size r (recall that n!/z λ is the number of permutations with cycle-type λ).At the level of combinatorial classes, Burnside's lemma gives and thus the following procedure is a uniform random sampler for T (see [2] for details): (1) Choose a random binary partition λ n under the distribution ) Let σ be a permutation with cycle-type λ.For each r ∈ [1..k] draw (independently) a tree T r ∈ B σ [n] uniformly at random.(3) Return the tangled chain corresponding to (T 1 , . . ., T k ).A recursive procedure (using (1)) is given in [2] to sample uniformly at random from B σ [n].From Theorem 1 we obtain a simpler random sampler for B σ [n].We order the cycles of σ as c 1 , . . ., c (λ) such that the cycle-sizes are in non-decreasing order.Then, with A 1 the set of labels in c 1 , we start from the unique tree (by Section 2.1) in B c1 [A 1 ] (where c 1 is to be seen as a cyclic permutation on A 1 ).Then, for i from 2 to (λ) we mark an edge chosen uniformly at random from the already obtained tree, and then we insert the leaves that have labels in c i using the isomorphism (2).
The complexity of the sampler for B σ [n] is clearly linear in n and needs no precomputation of coefficients.However step (1) of the random generator requires a table of p(n) coefficients, where p(n) is the number of binary partitions of n, which is slightly superpolynomial [4], p(n) = n Θ(log(n)) .It is however possible to do step (1) in polynomial time.For this, we consider, for i ≥ 0 and n, j ≥ 1 the coefficient S (i,j) n defined as the sum of r λ k /z λ over all binary partitions of n where the largest part is 2 i and has multiplicity j; note that S (i,j) n = 0 unless j • 2 i ≤ n, we denote by E n the set of such pairs (i, j).Since r λ = 1 and z λ = (|λ| − 1)! if λ has one part, we have the initial condition S (i,j) n = 1/(n − 1)! for j = 1 and 2 i = n.

Figure 1 (
Figure 1(b).We denote by B σ [A] the set of trees fixed by the action of σ, i.e., B σ [A] := {γ ∈ B[A] such that σ • γ = γ}.We also define E σ [A] (resp.E[A]) as the set of pairs (γ, e) where γ ∈ B σ [A] (resp.γ ∈ B[A]) and e is an edge of γ (among the 2n − 1 edges).Define the cycle-type of σ as the integer partition λ n giving the sizes of the cycles of σ (in non-increasing order).For λ n an integer partition, the cardinality of B σ [A] is the same for all permutations σ with cycle-type λ, and this common cardinality is denoted by r λ .It is known (e.g. using cycle index sums[1,3]) that r λ = 0 unless λ is a binary partition (i.e., an integer partition whose parts are powers of 2).Billey et al.[2] have recently found the following remarkable formula, valid for any binary partition λ:

Figure 2 .
Figure 2. (a) Rémy's leaf-removal operation.(b) The two cases for removing a 2-cycle of leaves (depending whether the two leaves have the same parent or not).The vertices depicted gray are allowed to be the fake vertex above the root-node.

For n ≥ 1 ,
denote by n the set {1, . . ., n}.A tanglegram of size n is an orbit of B[n] × B[n] under the relabelling action of S n (see Figure 4 for an example).More generally, for k ≥ 1, a tangled chain of length k and size n is an orbit of B[n] k under the relabelling action of S n , see [5, 2, 3].Let T (k) n be the set of tangled chains of length k and size n, and let t (k) n be the cardinality of T (k)