Estimation of a discrete monotone distribution

We study and compare three estimators of a discrete monotone distribution: (a) the (raw) empirical estimator; (b) the"method of rearrangements"estimator; and (c) the maximum likelihood estimator. We show that the maximum likelihood estimator strictly dominates both the rearrangement and empirical estimators in cases when the distribution has intervals of constancy. For example, when the distribution is uniform on $\{0, ..., y \}$, the asymptotic risk of the method of rearrangements estimator (in squared $\ell_2$ norm) is $y/(y+1)$, while the asymptotic risk of the MLE is of order $(\log y)/(y+1)$. For strictly decreasing distributions, the estimators are asymptotically equivalent.


Introduction
This paper is motivated in large part by the recent surge of acitivity concerning "method of rearrangement" estimators for nonparametric estimation of monotone functions: see, for example, Fougères (1997), , , Chernozhukov et al. (2009) and Anevski and Fougères (2007). Most of these authors study continuous settings and often start with a kernel type estimator of the density, which involves choices of a kernel and of a bandwidth. Our goal here is to investigate method of rearrangement estimators and compare them to natural alternatives (including the maximum likelihood estimators with and without the assumption of monotonicity) in a setting in which there is less ambiguity in the choice of an initial or "basic" estimator, namely the setting of estimation of a monotone decreasing mass function on the non-negative integers N = {0, 1, 2, . . .}.
Suppose that p = {p x } x∈N is a probability mass function; i.e. p x ≥ 0 for all x ∈ N and x∈N p x = 1. Our primary interest here is in the situation in which p is monotone decreasing: p x ≥ p x+1 for all x ∈ N. The three estimators of p we study are: (a). the (raw) empirical estimator, (b). the method of rearrangement estimator, (c). the maximum likelihood estimator.
Notice that the empirical estimator is also the maximum likelihood estimator when no shape assumption is made on the true probability mass function.
Much as in the continuous case our considerations here carry over to the case of estimation of unimodal mass functions with a known (fixed) mode; see e.g. Fougères (1997), Birgé (1987), and Alamatsaz (1993). For two recent papers discussing connections and tradeoffs between discrete and continuous models in a related problem involving nonparametric estimation of a monotone function, see Banerjee et al. (2009) and Maathuis and Hudgens (2009).
Distributions from the monotone decreasing family satisfy ∆p x ≡ p x+1 − p x ≤ 0 for all x ∈ N, and may be written as mixtures of uniform mass functions p x = y≥0 1 y + 1 1 {0,...,y} (x) q y . (1.1) Here, the mixing distribution q may be recovered via for any x ∈ N.
Remark 1.1. From the form of the mass function, it follows that p x ≤ 1/(x + 1) for all x ≥ 0.
Suppose then that we observe X 1 , X 2 , . . . , X n i.i.d. random variables with values in N and with a monotone decreasing mass function p. For x ∈ N, let p n,x ≡ n −1 n i=1 1 {x} (X i ) denote the (unconstrained) empirical estimator of the probabilities p x . Clearly, there is no guarantee that this estimator will also be monotone decreasing, especially for small sample size. We next consider two estimators which do satisfy this property: the rearrangement estimator and the maximum likelihood estimator (MLE).
For a vector w = {w 0 , . . . , w k }, let rear(w) denote the reverse-ordered vector such that w ′ = rear(w) satisfies w ′ 0 ≥ w ′ 1 ≥ . . . ≥ w ′ k . The rearrangement estimator is then simply defined as p R n = rear( p n ). We can also write p R n,x = sup{u : Q n (u) ≤ x}, where Q n (u) ≡ #{x : p n,x ≥ u}. To define the MLE we again need some additional notation. For a vector w = {w 0 , . . . , w k }, let gren(w) be the operator which returns the vector of the k + 1 slopes of the least concave majorant of the points j, j j=0 w i : j = −1, 0, . . . , k .
Here, we assume that −1 j=0 w j = 0. The MLE, also known as the Grenander estimator, is then defined as p G n = gren( p n ).
Thus, p G n,x is the left derivative at x of the least concave majorant (LCM) of the empirical distribution function F n (x) = n −1 n i=1 1 [0,x] (X i ) (where we include the point (−1, 0) to find the left derivative at x = 0). Therefore, by definition, the MLE is a vector of local averages over a partition of {0, . . . , max{X 1 , . . . , X n }}. This partition is chosen by the touchpoints of the LCM with F n . It is easily checked that p G n corresponds to the isotonic estimator for multinomial data as described in Robertson et al. (1988), pages 7-8 and 38-39.
We begin our discussion with two examples: in the first, p is the uniform distribution, and in the second p is strictly monotone decreasing. To compare the three estimators, we consider several metrics: the ℓ k norm for 1 ≤ k ≤ ∞ and the Hellinger distance. Recall that the Hellinger distance between two mass functions is given by while the ℓ k metrics are defined as In the examples, we compare the Hellinger norm and the ℓ 1 and ℓ 2 metrics, as the behaviour of these differs the most.
Example 1. Suppose that p is the uniform distribution on {0, . . . , 5}. For n = 100 independent draws from this distribution we observe p n = (0.20, 0.14, 0.11, 0.22, 0.15, 0.18). Then p R n = (0. 22, 0.20, 0.18, 0.15, 0.14, 0.11), and the MLE may be calculated as p G n = (0. 20, 0.16, 0.16, 0.16, 0.16, 0.16). The estimators are illustrated in Figure 1 (left). The distances of the estimators from the true mass function p are given in Table 1 (left). The maximum likelihood estimator p G n is superior in all three metrics shown. To explore this relationship further, we repeated the estimation procedure for 1000 Monte Carlo samples of size n = 100 from the uniform distribution. Figure 2 (left) shows boxplots of the metrics for the three estimators. The figure shows that here the rearrangement and empirical estimators have the same behaviour; a relationship which we establish rigorously in Theorem 2.1.
Example 2. Suppose that p is the geometric distribution with p x = (1 − θ)θ x for x ∈ N and with θ = 0.75. For n = 100 draws from this distribution we observe p n , p R n and p G n as shown in Figure 1 (right). The distances of the estimators from the true mass function p are given in Table 1 (right). Here, p n is outperformed by p G n and p R n in all the metrics, with p R n performing better in the ℓ 1 and ℓ 2 metrics, but not in the Hellinger distance. These relationships appear to hold true in general, see Figure 2 (left) for boxplots of the metrics obtained through Monte Carlo simulation.  The above examples illustrate our main conclusion: the MLE preforms better when the true distribution p has intervals of constancy, while the MLE and rearrangement estimators are competitive when p is strictly monotone. Asymptotically, it turns out that the MLE is superior if p has any periods of constancy, while the empirical and rearrangement estimators are equivalent. However, if p is strictly monotone, then all three estimators have the same asymptotic behaviour.
Both the MLE and monotone rearrangement estimators have been considered in the literature for the decreasing probability density function. The MLE, or Grenander estimator, has been studied extensively, and much is known about its behaviour. In particular, if the true density is locally strictly decreasing, then the estimator converges at a rate of n 1/3 , and if the true density is locally flat, then the estimator converges at a rate of n 1/2 , cf. Prakasa Rao (1969); Carolan and Dykstra (1999), and the references therein for a further history of the problem. In both cases the limiting distribution is characterized via the LCM of a Gaussian process.
The monotone rearrangement estimator for the continuous density was introduced by  Figure 2: Monte Carlo comparison of the estimators: boxplots of m = 1000 distances of the estimators p n (white), p R n (light grey) and p G n (dark grey) from the truth for a sample size of n = 100. Left: the true distribution is the discrete uniform; and right: the true distribution is the geometric distribution with θ = 0.75. Fougères (1997) (see also ). It is found by calculating the monotone rearrangement of a kernel density estimator (see e.g. Lieb and Loss (1997)). Fougères (1997) shows that this estimator also converges at the n 1/3 rate if the true density is locally strictly decreasing, and it is shown through Monte Carlo simulations that it has better behaviour than the MLE for small sample size. The latter is done by comparing the L 1 metrics for different, strictly decreasing, densities. Unlike our Example 2, the Hellinger distance is not considered.
The outline of this paper is as follows. In Section 2 we show that all three estimators are consistent. We also establish some small sample size relationships between the estimators. Section 3 is dedicated to the limiting distributions of the estimators, where we show that the rate of convergence is n 1/2 for all three estimators. Unlike the continuous case, the local behaviour of the MLE is equivalent to that of the empirical estimator when the true mass function is strictly decreasing. In Section 4 we consider the limiting behaviour of the ℓ p and Hellinger distances of the estimators. In Section 5, we consider the estimation of the mixing distribution q. Proofs and some technical results are given in Section 6. R code to calculate the maximum likelihood estimator (i.e. gren( p G n )) is available from the website of the first author: (will be provided).

Some inequalities and consistency results
We begin by establishing several relationships between the three different estimators.
Let P denote the collection of all decreasing mass functions on N. For any estimator p n of p ∈ P and k ≥ 1 let the loss function L k be defined by L k (p, (2.5) Corollary 2.2. When k = 2, and for any sample size n, it holds that Based on these results, we now make the following remarks.
1. It is always better to use a monotone estimator (either p R n or p G n ) to estimate a monotone mass function. 2. If the true distribution is uniform, then clearly the MLE is the better choice. 3. If the true mass function is strictly monotone, then the estimators p R n and p G n should be asymptotically equivalent. We make this statement more precise in Sections 3 and 4. Figure 2 (right) shows that in this case p R n and p G n have about the same performance for n = 100. 4. When only the monotonicity constraint is known about the true p, then, by Corollary 2.2, p G n is a better choice of estimator than p R n . Remark 2.3. In continuous density estimation one of the most popular measures of distance is the L 1 norm, which corresponds to the ℓ 1 norm on mass functions. However, for discrete mass functions, it is more natural to consider the ℓ 2 norm. One of the reasons is made clear in the following sections (cf. Theorem 3.8, Corollaries 4.1 and 4.2, and Remark 4.4). The ℓ 2 space is the smallest space in which we obtain convergence results, without additional assumptions on the true distribution p.   Figure 3: Comparison of the estimators p n (white), p R n (light grey) and p G n (dark grey).
To examine more closely the case when the true distribution p is neither uniform nor strictly monotone we turn to Monte Carlo simulations. Let p U (y) denote the uniform mass function on {0, . . . , y}. Figure 3 shows boxplots of m = 1000 samples of the estimators for three distributions: On the left we have a small sample size of n = 20, while on the right n = 100. For each distribution and sample size, we calculate the three estimators (the estimators p n , p R n and p G n are shown in white, light grey and dark grey, respectively) and compute their distance functions from the truth (Hellinger, ℓ 1 , and ℓ 2 ). Note that the MLE outperforms the other estimators in all three metrics, even for small sample sizes. It appears also that the more regions of constancy the true mass function has, the better the relative performance of the MLE, even for small sample size (see also Figure 2). By considering the asymptotic behaviour of the estimators, we are able to make this statement more precise in Section 4.
All three estimators are consistent estimators of the true distribution, regardless of their relative performance.
Theorem 2.4. Suppose that p is monotone decreasing. Then all three estimators p n , p G n and p R n are consistent estimators of p in the sense that ρ(p n , p) → 0 almost surely as n → ∞ forp n = p n , p G n and p R n , whenever ρ(p, p) = H(p, p) or ρ(p, p) = ||p − p|| k , 1 ≤ k ≤ ∞.
As a corollary, we obtain the following Glivenko-Cantelli type result.
almost surely.

Limiting distributions
Next, we consider the large sample behaviour of p n , p R n and p G n . To do this, define the fluctuation processes Y n , Y R n , and Y G n as Regardless of the shape of p, the limiting distribution of Y n is well-known. In what follows we use the notation Y n,x → d Y n,x to denote weak convergence of random variables in R (we also use this notation for R d ), and Y n ⇒ Y to denote that the process Y n converges weakly to the process Y . Let Y = {Y x } x∈N be a Gaussian process on the Hilbert space ℓ 2 with mean zero and covariance operator S such that S e (x) , e ( denotes a sequence which is one at location x, and zero everywhere else. The process is well-defined, since For background on Gaussian processes on Hilbert spaces we refer to Parthasarathy (1967).
Theorem 3.1. For any mass function p, the process Y n satisfies Y n ⇒ Y in ℓ 2 .
Remark 3.2. We assume that Y is defined only on the support of the mass function p.

Local Behaviour
At a fixed point x there are only two possibilities for the true mass function p: either x belongs to a flat region for p (i.e. p r = . . . = p x = . . . = p s for some r ≤ x ≤ s), or p is strictly decreasing at x: p x−1 > p x > p x+1 . In the first case the three estimators exhibit different limiting behaviour, while in the latter all three have the same limiting distribution.
In some sense, this result is not surprising. Suppose that x is such that p Then asymptotically this will hold also for p n : p n,x−k > p n,x > p n,x+k for k ≥ 1 and for sufficiently large n. Therefore, in the rearrangement of p n the values at x will always stay the same, i.e. p R n,x = p n,x . Similarly, the empirical distribution function F n will also be locally concave at x, and therefore both x, x − 1 will be touchpoints of F n with its LCM. This implies that p G n,x = p n,x . On the other hand, suppose that x is such that p x−1 = p x = p x+1 . Then asymptotically the empirical density will have random order near x, and therefore both re-orderings (either via rearrangement or via the LCM) will be necessary to obtain p R n,x and p G n,x .
3.1.1 When p is flat at x. We begin with some notation. Let q = {q x } x∈N be a sequence, and let r ≤ s be positive integers. We define q (r,s) = {q r , q r+1 , . . . , q s−1 , q s } to be the r through s elements of q.
Proposition 3.3. Suppose that for some r, s ∈ N with s − r ≥ 1 the probability mass function p satisfies p r−1 > p r = · · · = p s > p s+1 . Then The last statement of the above theorem is the discrete version of the same result in the continuous case due to Carolan and Dykstra (1999) for a density with locally flat regions. Thus, both the discrete and continuous settings have similar behaviour in this situation. Figure 4 shows the exact and limiting cumulative distribution functions when p = 0.2p U (3) + 0.8p U (7) (same as in Figure 3, top) at locations x = 4 and x = 7. Note the significantly "more discrete" behaviour of the empirical and rearrangement estimators in comparison with the MLE. Also note the lack of accuracy in the approximation at x = 4 when n = 100 (top left), which is more prominent for the rearrangement estimator. This occurs because x = 4 is a boundary point, in the sense that p 3 > p 4 , and is therefore least resilient to any global changes in p n . Lastly, note that the distribution functions satisfy 7) ) and (Y G ) (4,7) = gren(Y (4,7) ).
Proposition 3.4. Let θ = p r = . . . = p s , and let Y (r,s) denote a multivariate normal vector with mean zero and variance matrix {σ i,j } s i,j=r where σ i,j = αδ i,j − α 2 , for α −1 = s − r + 1. Let Z be a standard normal random variable independent of Y (r,s) , and let τ = s − r + 1. Then Note that the behaviour of gren(Y (r,s) ) and gren( Y (r,s) ) will be quite different since s x=r Y (r,s) x = 0 almost surely, but the same is not true for Y (r,s) .
Remark 3.5. To match the notation of Carolan and Dykstra (1999), note that τ gren( Y (r,s) ) is equivalent to the left slopes at the points {1, . . . , τ }/τ of the least concave majorant of standard Brownian bridge at the points {0, 1, . . . , τ }/τ . This random vector most closely matches the left derivative of the least concave majorant of the Brownian bridge on [0, 1], which is the process that shows up in the limit for the continuous case.
3.1.2 When p is strictly monotone at x.
In this situation, the three estimators p n,x , p R n,x and p G n,x have the same asymptotic behaviour. This is considerably different than what happens for continuous densities, and occurs because of the inherent discreteness of the problem for probability mass functions.
Proposition 3.6. Suppose that for some r, s ∈ N with s − r ≥ 0 the probability mass function p satisfies p r−1 > p r > . . . > p s > p s+1 . Then Remark 3.7. We note that the convergence results of Propositions 3.3 and 3.6 also hold jointly. That is, convergence of the three processes (Y (r,s) n , (Y R n ) (r,s) , (Y G n ) (r,s) ) may also be proved jointly in R 3(s−r+1) .

Convergence of the Process
We now strengthen these results to obtain convergence of the processes Y R n and Y G n in ℓ 2 . Note that the limit of Y n has already been stated in Theorem 3.1.
Theorem 3.8. Let Y be the Gaussian process defined in Theorem 3.1, with p a monotone decreasing distribution. Define Y R and Y G as the processes obtained by the following transforms of Y : for all periods of constancy of p, i.e. for all s ≥ r with s − r ≥ 1 such that The two extreme cases, p strictly monotone decreasing and p equal to the uniform distribution, may now be considered as corollaries. By studying the uniform case, we also study the behaviour of Y G (via Proposition 3.4), and therefore we consider this case in detail.
Corollary 3.9. Suppose that p is strictly monotone decreasing. That is, suppose that

The Uniform Distribution
Here, the limiting distribution Y is a vector of length y + 1 having a multivariate normal Corollary 3.10. Suppose that p is the uniform probability mass function on {0, . . . , y}, Figure 5: The relationship between the limiting process Y and the least concave majorant of its partial sums for the uniform distribution on {0, . . . , 5}. Left: the slopes of the lines L 1 , L 2 and L 3 give the values gren(Y ) 0 , gren(Y ) 1 = . . . = gren(Y ) 4 and gren(Y ) 5 , respectively. Right: the discrete Brownian bridge lies entirely below zero. Therefore, its LCM is zero, and also gren(Y ) ≡ 0. This event occurs with positive probability (see also Figure 6).
The limiting process gren(Y ) may also be described as follows. Let U(·) denote the standard Brownian bridge process on [0, 1], and write U k = k j=0 Y j for k = −1, . . . , y. Then we have equality in distribution of In particular we have that U −1 = U y = y j=0 Y j = 0. Thus, the process U is a discrete analogue of the Brownian bridge, and gren(Y ) is the vector of (left) derivatives of the least concave majorant of {(j, U j ) : j = −1, . . . , y}. Figure 5 illustrates two different realizations of the processes Y and gren(Y ).
Remark 3.11. Note that if the discrete Brownian Bridge is itself convex, then the limits Y, rear(Y ) and gren(Y ) will be equivalent. This occurs with probability The result matches that in part (iii) of Theorem 2.1. . For y = 9, it is equal to 0.0999. Figure 6 examines the behaviour of the limiting distribution of the MLE for several values of x. Since this is found via the LCM of the discrete Brownian bridge, it maintains the monotonicity property in the limit: that is, gren(Y ) x ≥ gren(Y ) x+1 . This can easily be seen by examining the marginal distributions of gren(Y ) for different values of x ( Figure 6, left). For each x, there is a positive probability that gren(Y ) x = 0. This occurs if the discrete Brownian bridge lies entirely below zero and then the least concave majorant is identically zero, in which case gren(Y ) x = 0 for all x = 0, . . . , y (as in Figure 5, right). The probability of this event may be calculated exactly using the distribution function of the multivariate normal. Figure 6 (right), shows several values for different y.

Limiting distributions for the metrics
In the previous section we obtained asymptotic distribution results for the three estimators.
To compare the estimators, we need to also consider convergence of the Hellinger and ℓ k metrics. Our results show that p R n and p n are asymptotically equivalent (in the sense that the metrics have the same limit). The MLE is also asymptotically equivalent, but if and only if p is strictly monotone. If p has any periods of constancy, then the MLE has better asymptotic behaviour. Heuristically, this happens because, by definition, Y G is a sequence of local averages of Y , and averages have smaller variability. Furthermore, the more and larger the periods of constancy, the better the MLE performs, see, in particular, Proposition 4.5 below. These results quantify, for large sample size, the observations of Figure 3.
The rate of convergence of the ℓ 2 metric is an immediate consequence of Theorem 3.8. Below, the notation Z 1 ≤ S Z 2 denotes stochastic ordering: i.e. P (Z 1 > x) ≤ P (Z 2 > x) for all x ∈ R (the ordering is strict if both inequalities are replaced with strict inequalities).
Corollary 4.1. Suppose that p is a monotone decreasing distribution. Then, for any If p is not strictly monotone, then ≤ S may be replaced with < S . The above convergence also holds in expectation (that is, E[||Y n || k k ] → E[||Y || k k ] and so forth). Furthermore, with equality if and only if p is strictly monotone.
Convergence of the other two metrics is not as immediate, and depends on the tail behaviour of the distribution p.
If p is not strictly monotone, then ≤ S may be replaced with < S . The above convergence also holds in expectation, and with equality if and only if p is strictly monotone.
Convergence of the Hellinger distance requires an even more stringent condition.
Corollary 4.3. Suppose that κ = sup{x : p x > 0} < ∞. Then If p is not strictly monotone, then ≤ S may be replaced with < S . The distribution of κ x=0 Y 2 x /p x is chi-squared with κ degrees of freedom. The above convergence also holds in expectation, and with equality if and only if p is strictly monotone.
Remark 4.4. We note that if x≥0 √ p x = ∞, then x≥0 |Y x | = ∞ almost surely, and if κ = ∞, then x≥0 Y 2 x /p x is also infinite almost surely. This implies that for the empirical and rearrangement estimators, the conditions in Corollaries 4.2 and 4.3 are also necessary for convergence. The same is true for the Grenander estimator, when the true distribution is strictly decreasing.
Proposition 4.5. Let p be a decreasing distribution, and write it in terms of its intervals of constancy. That is, let where where θ i > θ i+1 for all i = 1, 2, . . ., and where {C i } i≥1 forms a partition of N. Then This result allows us to explicitly calculate exactly how much "better" the performance of the MLE is, in comparison to Y and Y R . With R-valued random variables, it is standard to compare the asymptotic variance to evaluate the relative efficiency of two estimators. We, on the other hand, are dealing with R N -valued processes. Consider some process W ∈ R N , and let Σ W denote its covariance matrix (of size N × N). Then the trace norm of Σ W is equal to the expected squared ℓ 2 norm of W , where {λ i } i≥1 denotes the eigenvalues of Σ W . Therefore, Corollary 4.1 tells us that, asymptotically, Y G is more efficient than Y R and Y , in the sense that with equality if and only if p is strictly decreasing. Furthermore, Proposition 4.5 allows us to calculate exactly how much more efficient Y G is for any given mass function p.
Suppose that p has exactly one period of constancy on r ≤ x ≤ s, and let τ = s−r+1 ≥ 2. Further, suppose that p In particular, if p is the uniform distribution on {0, . . . , y}, then we find that E ||Y R || 2 2 = y/(y + 1), whereas E ||Y G || 2 2 behaves like log y/(y + 1), and is much smaller. Note that if p is strictly monotone, then we obtain as required. Also, if p is the uniform probability mass function on {0, . . . , y}, we conclude that where log y − 0.5 < y i=1 (i + 1) −1 < log(y + 1). Lastly, consider a distribution with bounded support, and fix r < s where p is strictly monotone on {r, . . . , s}. That is, we have that p r−1 > p r > . . . > p s > p s+1 . Next definep byp x = p x for x < r and x > s, andp x = s x=r p x /(s − r + 1) for x ∈ {r, . . . , s}. Then the difference in the expected Hellinger metrics under the two distributions is where τ = s − r + 1. Therefore, the longer the intervals of constancy in a distribution, the better the performance of the MLE.
Remark 4.6. From Theorem 1.6.2 of Robertson et al. (1988) it follows that for any This result may also be proved using the method used to show Proposition 4.5. Note that this pointwise inequality does not hold in general for Y G replaced with Y R .
Corollaries 4.1 and 4.2 then translate into statements concerning the limiting risks of the three estimators p n , p R n , and p G n as follows, where the risk was defined in (2.5). In particular, we see that, asymptotically, both p R n and p n are inadmissible, and are dominated by the maximum likelihood estimator p G n . Corollary 4.7. For any 2 ≤ k ≤ ∞, and any p ∈ P, the class of decreasing probability mass functions on N, The inequality in the last line is strict if p is not strictly monotone. The statements also hold for k = 1 under the additional hypothesis that x≥0 √ p x < ∞.

Estimating the mixing distribution
Here, we consider the problem of estimating the mixing distribution q in (1.1). This may be done directly via the estimators of p and the formula (1.2). Define the estimators of the mixing distribution as follows Each of these estimators sums to one by definition, however q n is not guaranteed to be positive. The main results of this section are consistency and √ n-rate of convergence of these estimators.
Theorem 5.1. Suppose that p is monotone decreasing and satisfies x≥0 xp x < ∞. Then all three estimators q n , q G n and q R n are consistent estimators of q in the sense that ρ(q n , q) → 0 almost surely as n → ∞ forq n = q n , q G n and q R n , whenever ρ(q, q) = H(q, q) or ρ(q, q) = ||q − q|| k , 1 ≤ k ≤ ∞.
To study the rates of convergence we define the the fluctuation processes Z n , Z R n , and Z G n as with limiting processes defined as and these convergences also hold in expectation. Also, nH 2 ( q n , q) → d and these again also hold in expectation.
As before, we have asymptotic equivalence of all three estimators if p is strictly decreasing (cf. Corollary 3.9). To determine the relative behaviour of the estimators q R n and q G n we turn to simulations. Since q n is not guaranteed to be a probability mass function (unlike the other two estimators), we exclude it from further consideration.
In Figure 7, we show boxplots of m = 1000 samples of the distances ℓ 1 (q, q), ℓ 2 (q, q) and H(q, q) forq = q R n (light grey) andq = q G n (dark grey) with n = 20 (left), n = 100 (centre) and n = 1000 (right). From top to bottom the true distributions are (a) p = p U (5) , (b) p = 0.2p U (3) + 0.8p U (7) , (c) p = 0.25p U (1) + 0.2p U (3) + 0.15p U (5) + 0.4p U (7) , and (d) p is geometric with θ = 0.75. We can see that q G n has better performance in all metrics, except for the case of the strictly decreasing distribution. As before, the flatter the true distribution is, the better the relative performance of q G n . Notice that by Corollary 3.9 and Theorem 5.2 the asymptotic behaviour (i.e. rate of convergence and limiting distributions) of the l 2 norm of q G n and q R n should be the same if p is strictly decreasing.
Remark 5.3. For κ = ∞, the process {xY n,x : x ∈ N} is known to converge weakly in ℓ 2 if and only if x≥0 x 2 p x < ∞, while the convergence is know to hold in ℓ 1 if and only if x≥0 x √ p x < ∞; see e.g. Araujo and Giné (1980, Exercise 3.8.14, page 205). We therefore conjecture that Z R n and Z G n converge weakly to Z R and Z G in ℓ 2 (resp. ℓ 1 ) if and only if x≥0 x 2 p x < ∞ resp. x≥0 x √ p x < ∞ .

Proofs
Proof of Remark 1.1. This bound follows directly from the definition of p, since In the next lemma, we prove several useful properties of both the rearrangement and Grenander operators.
Lemma 6.1. Consider two sequences p and q with support S, and let ϕ(·) denote either the Grenander or rearrangement operator. That is, ϕ(p) = gren(p) or ϕ(p) = rear(p).

For any increasing function
(6.6) 2. Suppose that Ψ : R → R + is a non-negative convex function such that Ψ(0) = 0, and that q is decreasing. Then, 3. Suppose that |S| is finite. Then ϕ(p) is a continuous function of p. for y ∈ S. These inequalities immediately imply (6.6), since, by summation by parts, and f is an increasing function.
2. For the Grenander estimator this is simply Theorem 1.6.1 in Robertson et al. (1988). For the rearrangement estimator, we adapt the proof from Theorem 3.5 in Lieb and Loss (1997). We first write Ψ = Ψ + + Ψ − , where Ψ + (x) = Ψ(x) for x ≥ 0 and Ψ − (x) = Ψ(x) for x ≤ 0. Now, since Ψ + is convex, there exists an increasing function Ψ ′ + such that Applying Fubini's theorem, we have that Now, the function I [qx≤s] is an increasing function of x, and for ϕ(p) = rear(p), for each fixed s we have that ϕ(Ψ ′ + (p − s)) x = Ψ ′ + (ϕ(p) x − s), since Ψ ′ + is an increasing function. Therefore, applying (6.6), we find that the last display above is bounded below by The proof for Ψ − is the same, except that here we use the identity 3. Since |S| is finite, we know that p is a finite vector, and therefore it is enough to prove continuity at any point x ∈ S. For ϕ = rear this is a well-known fact. Next, note that if p n → p, then the partial sums of p n also converge to the partial sums of p. From Lemma 2.2 of Durot and Tocquet (2003), it follows that the least concave majorant of p n converges to the least concave majorant of p, and hence, so do their differences. Thus ϕ(p n ) x → ϕ(p) x .

Some inequalities and consistency results: proofs
Proof of Theorem 2.1. (i). Choosing Ψ(t) = |t| k in (6.7) of Lemma 6.1 proves (2.4). To prove (2.3) recall that By Hardy et al. (1952), Theorem 368, page 261, (or Theorem 3.4 in Lieb and Loss (1997)) it follows that which proves the result for the rearrangement estimator. It remains to prove the same for the MLE. Let {B i } i≥1 denote a partition of N. By definition, for some partition. Jensen's inequality now implies that which completes the proof.
Proof of Corollary 2.2. For any p ∈ P, we have that Plugging in the discrete uniform distribution on {0, . . . , κ}, and applying part (ii) of Theorem 2.1, we find that nR 2 (p, p R n ) = nR 2 (p, p n ) = 1 − (κ + 1) −1 . Thus, for any ǫ > 0, there exists a p ∈ P, such that nR 2 (p, p R n ) = nR 2 (p, p n ) ≥ 1 − ǫ. Since the upper bound on both risks is one, the result follows.
Proof of Theorem 2.4. The results of this theorem are quite standard, and we provide a proof only for completeness. Let F n denote the empirical distribution function and F the cumulative distribution function of the true distribution p. For any K (large), we have that for any x > K, Fix ǫ > 0, and choose K large enough so that (1 − F (K)) < ǫ/6. Next, there exists an n 0 sufficiently large so that sup 0≤x≤K | p n,x − p x | < ǫ/3 and |F n (K) − F (K)| < ǫ/3 for all n ≥ n 0 almost surely. Therefore, for n ≥ n 0 This shows that || p n − p|| k → 0 almost surely for k = ∞. A similar approach proves the result for any 1 ≤ k < ∞. Convergence of H( p n , p) follows since for mass functions H(p, q) ≤ ||p − q|| 1 (see e.g. Le Cam (1969), page 35). Consistency of the other estimators, p R n and p G n now follows from the inequalities of Theorem 2.1.
Proof of Corollary 2.5. Note that by virtue of the estimators, we have that F R n (x) ≥ F n (x) and F G n (x) ≥ F n (x) for all x ≥ 0. Now, fix ǫ > 0. Then there exists a K such that x>K p x < ǫ/4. By the Glivenko-Cantelli lemma, there exists an n 0 such that for all n ≥ n 0 sup x≥0 |F n (x) − F (x)| < ǫ/4, almost surely. Furthermore, by Theorem 2.4, n 0 can be chosen large enough so that for all n ≥ n 0 sup x≥0 p G n,x − p x < ǫ/4(K + 1), almost surely. Therefore, for all n ≥ n 0 , we have that The proof for the rearrangement estimator is identical.
Then W n is tight in ℓ k .
Proof. Note that for k < ∞, compact sets K are subsets of ℓ k such that there exists a sequence of real numbers A x for x ∈ N and a sequence λ m → 0 such that k≥m |w x | k ≤ λ m for all m, for all elements w ∈ K. Clearly, if the conditions of the lemma are satisfied, then for each ǫ > 0, we have that P |W n,x | ≤ A x for all x ≥ 0, and x≥m |W n,x | k ≤ λ m for all m ≥ 1 − ǫ for all n. Thus, W n is tight in ℓ k .
Proof of Theorem 3.1. Convergence of the finite dimensional distributions is standard. It remains to prove tightness in ℓ 2 . By Lemma 6.2 this is straightforward, since Throughout the remainder of this section we make extensive use of a set equality for the least concave majorant known as the "switching relation". Let s n (a) = inf {k ≥ −1 : F n (k) − a(k + 1) = sup{F n (y) − a(y + 1)}} ≡ argmax L k≥−1 {F n (k) − a(k + 1)} (6.8) denote the first time that the process F n (y) − a(y + 1) reaches its maximum. Then the following holds For more background (as well as a proof) of this fact see, for example, Balabdaoui et al. (2009).
Proof of Proposition 3.3. Let F denote the cumulative distribution function for the function p. For fixed t ∈ R it follows from (6.9) that where Z n (y) = n 1/2 F n (y)−(n 1/2 p x +t)(y+1). Note that for any constant c, argmax L (Z n (y)) = argmax L (Z n (y) + c), and therefore we instead take Z n (y) = n 1/2 (F n (y) − F n (r − 1)) − (n 1/2 p x + t)(y − r + 1) = V n (y) + W n (y) − t(y − r + 1), where V n (y) = √ n ((F n (y) − F n (r − 1)) − (F (y) − F (r − 1))) , n −1/2 W n (y) = (F (y) − F (r − 1)) − p x (y − r + 1) Let U denote the standard Brownian bridge on [0, 1]. It is well-known that V n (y) ⇒ U(F (y))− U(F (r − 1)). Also, W n (y) → ∞ for y / ∈ {r − 1, . . . , s}, and it is identically zero otherwise. It follows that the limit of (6.10) is for any x ∈ {r, . . . , s}. Note that the process and therefore the probability above is equal to P gren(Y (r,s) ) x < t for x ∈ {r, . . . , s}. Since the half-open intervals [a, b) are convergence determining, this proves pointwise convergence of Y G n,x to gren(Y ) x . To show convergence of the rearrangement estimator fluctuation process, note that for sufficiently large n we have that p n,r−k > p n,x > p n,s+k for all x ∈ {r, . . . , s} and k ≥ 1. Therefore, ( p R n ) (r,s) = rear(( p n ) (r,s) ) and furthermore, since p x is constant here, (Y R n ) (r,s) = rear(Y (r,s) n ). The result now follows from the continuous mapping theorem.
Proof of Theorem 3.8. Let ϕ denote an operator on sequences in l 2 . Specifically, we take ϕ = gren or ϕ = rear. Also, for a fixed mass function p let Next, define ϕ p to be the local version of the ϕ operator. That is, for each i ≥ 1, Fix ǫ > 0, and suppose that q n → q in ℓ 2 . Then there exists a K ∈ T p and an n 0 such that sup n≥n 0 x>K q 2 n,x < ǫ/6. By Lemma 6.1, ϕ p is continuous on finite blocks, and therefore it is continuous on {0, . . . , K}. Hence, there exists a n ′ 0 such that for all n ≥ n ′ Applying (6.7), we find that for all n ≥ max{n 0 , n ′ 0 } which shows that ϕ p is continuous on ℓ 2 . Since Y n ⇒ Y in ℓ 2 , it follows, by the continuous mapping theorem, that ϕ p (Y n ) ⇒ ϕ p (Y ). However, both Y G n and Y R n are of the form √ n(ϕ( p n ) − p) = ϕ p (Y n ). To complete the proof of the theorem it is enough to show that converges to zero in L 1 ; that is, we will show that E[E n ] → 0. By Skorokhod's theorem, there exists a probability triple and random processes Y and Y n = √ n( p n − p), such that Y n → Y almost surely in ℓ 2 . Fix ǫ > 0 and find K ∈ T p such that x>K p x < ǫ/4. Next, let T K p = {0 ≤ x ≤ K : x ∈ T p }, and let δ = min x∈T K p (p x − p x+1 ). Then, there exists an n 0 such that for all n ≥ n 0 sup x≥0 | p n,x − p x | < δ/3, (6.12) sup x≥0 | 0≤y≤x ϕ( p n ) y − F (x)| < δ/6, (6.13) almost surely (see Corollary 2.5).
Now, consider any m ∈ T K p . It follows that any such m is also a touchpoint of the operator ϕ on p n . Here, by touchpoint we mean that m x=0 ϕ( p n ) x = m x=0 p n,x . From (6.12), it follows that inf x≤m p n,x > sup x>m p n,x , which implies that m is a touchpoint for the rearrangement estimator. For the Grenander estimator, we require (6.13). Here, . Therefore, the slope of F G n changes from m to m + 1, which implies that m is a touchpoint almost surely. Let p (s,r) n = { p n,s , p n,s+1 , . . . , p n,r }. An important property of the ϕ operator is if m < m ′ are two touchpoints of ϕ applied to p n , then for all m almost surely. It follows that and hence Since E n ≤ 2||Y n || 2 2 , with E[||Y n || 2 2 ] ≤ 1, we may apply Fatou's lemma so that Letting ǫ → 0 completes the proof. Corollaries 3.9 and 3.10 are obvious consequences of Theorem 3.8. Remark 3.11 is proved in the following section.

Limiting distributions for metrics: proofs
Proof of Corollary 4.1. We provide the details only in the k = 2 setting. The cases when k > 2 follow in a similar manner, since here ||x|| k ≤ ||x|| 2 for x ∈ ℓ 2 .
Convergence of ||Y n || 2 , ||Y R n || 2 and ||Y G n || 2 follows from Theorems 3.1 and 3.8 by the continuous mapping theorem. That ||Y || 2 = ||Y R || 2 is obvious from the definition of Y R . That ||Y G || 2 ≤ ||Y || 2 follows from Jensen's inequality and the definition of the gren(·) operator, since for any r < s, gren(Y (r,s) ) x is equal to the average of Y y over some subset of {r, . . . , s} containing the point x. If p is not strictly decreasing, then there exists a region, which we denote again by {r, . . . , s}, where it is constant. Then there is positive probability that (Y G ) (r,s) is different from Y (r,s) . In this case, we have that which finishes the proof of the stochastic ordering in the third statement. Convergence in expectation is immediate since To complete the argument note that sup C∈2 N | x∈C w x | = ||w|| 1 /2 for any sequence w such that x w x = 0 (note that the condition x √ p x < ∞ means that the sequences Y n and Y are absolutely summable almost surely). However, the result may also be proved by noting that the sequence Y n is tight in ℓ 1 using Lemma 6.2, since as m → ∞ under the assumption x≥0 √ p x < ∞. The proof that Y G n ⇒ Y G and Y R n ⇒ Y R in ℓ 1 is identical to the proof of Theorem 3.8, and we omit the details. Convergence of expectations follows since ||Y n || 1 is uniformly integrable, as by the Cauchy-Schwarz inequality. All other details follow as in the proof of Corollary 4.1.
Proof of Corollary 4.3. If κ < ∞, then we have that Y 2 x p x (6.14) by Theorem 3.1 and Theorem 2.4 for k = ∞. That this has a chi-squared distribution with κ degrees of freedom is standard, and is shown for example, in Ferguson (1996), Theorem 9. Convergence of means follows by the dominated convergence theorem from the bound H(p, q) ≤ ||p − q|| 1 (see e.g. Le Cam (1969), page 35) and Corollary 4.2. All other details follow as in the proof of Corollary 4.1.
Proof of Remark 4.4. Suppose first that x≥0 √ p x = ∞. Define P to be the probability measure P (A) = x∈A p x , and let W be the mean zero Gaussian field on ℓ 2 such that and W N is finite almost surely, it follows that x≥0 |Y x | = ∞ almost surely as well. That is, if x≥0 √ p x = ∞, then the random variable ||Y || 1 simply does not exist.
A similar argument works for the Hellinger norm. Assume that κ = ∞. Then and the Borel-Cantelli lemma shows that x≥0 W 2 x /p x is infinite almost surely.
Lemma 6.3. Let Z 1 , . . . , Z k be i.i.d. N(0,1) random variables, and let Z G i , i = 1, . . . , k denote the left slopes of the least concave majorant of the graph of the cumulative sums j i=1 Z j with j = 0, . . . , k. Let T denote the number of times that the LCM touches the cumulative sums (excluding the point zero, but including the point k). Then Proof. It is instructive to first consider some of the simple cases. When k = 1, the result is obvious. Suppose then that k = 2. We have Note that we ignore all equalities, since these occur with probability zero. It follows that where, by exchangeability it follows that On the other hand, we also have that since the random variablesZ = (Z 1 + Z 2 )/2 and Z 1 −Z are independent. The result follows. Next, suppose that k = 3. Then we have the following.
The choice of splitting the conditions between columns (a) and (b) is key to our argument. Note that the LCM creates a partition of the space {1, . . . , k}, where within each subset the slope of the LCM is constant. The number of partitions is equal to T . Here, column (a) describes the necessary conditions on the order of the slopes on the partitions, while column (b) describes the necessary conditions that must hold within each partition.
In the first row of the table, we find by permuting across all orderings of (123) that 1. Permutations of subgroups (column (a)), and 2. independence of column (b) from the random variables k i=1 (Z G i ) 2 and the indicator functions in column (a). Note that for any k > j ≥ 1, lettingZ = (Z 1 +Z 2 +. . .+Z k )/k which is independent ofZ for any choice of j < k.
To write down the proof for any k we must first introduce some notation.
• For any 1 ≤ m ≤ k, we may create a collection P of partitions of {1, . . . , k} such that the total number of elements in each partition is m. For example, when k = 4 and m = 2, then the elements of P are the partitions ({1}{234}), ({12}{34}) and ({123}{4}). Furthermore, for each partition, we may write down the number of elements in each subset of the partition. Here the sizes of the partitions are 1, 3 then 2, 2 and 3, 1. These partitions may be grouped further by placing together all partitions such that their sizes are unique up to order. Thus, in the above example we would put together 1, 3 and 3, 1 as one group, and the second group would be made up of 2, 2. From each subgroup we wish to choose a representative member, and the collection of these representatives will be denoted as τ (m). We assume that the representative τ is chosen in such a way that the sizes of the partitions are given in increasing order. Let r 1 denote the number of subgroups with size 1, and so on. Thus, for τ = ({1}{234}), we have r 1 = 1, r 2 = 0, r 3 = 1, . . . , r k = 0.
• Next, from τ (m) we wish to re-create the entire collection P. To do this, it is sufficient to take each τ and re-create all of the partitions which had the same sizes. Let σ m τ denote the resulting collection for a fixed partition τ . Thus, P is equal to the union of σ m τ over all τ ∈ τ (m). Note that the number of elements in σ m τ is given by m r 1 r 2 . . . r k .
We also use the notation R j = j i=1 r i , with R 0 = 0. Note that R k = m.
• For each σ j as defined above, we let where σ (l) j denotes σ j with its last l elements removed.
We are now ready to calculate E[ k i=1 (Z G i ) 2 1 T =m ]. By considering all possible partitions, this is equal to the sum over all τ ∈ τ (m) of the following terms By permuting each σ ∈ σ m τ , and appealing to the exchangeability of the Z i 's, this is equal to AVσ j Z>max{AV −1 σ j Z,...,AV −(|σ j |−1) σ j Z} , by independence of each AV σ j Z and each Z i − AV σ j Z for i ∈ σ j . Notice that the permutations of σ ∈ σ m τ do not account for permutations across all groups with equal "size". By considering furthermore all permutations between groups of equal size, we further obtain that the last display above is equal to 1 i + 1 , (6.15) using the notation of the Proposition 3.4. Without loss of generality we may assume that r = 0, and for simplicity we write Y for Y (r,s) . Let k = s + 1, and let Z 1 , . . . , Z k denote k i.i.d. N(0,1) random variables, l etZ denote their average, and let Z i = Z i −Z (which is independent ofZ). We then have that Therefore, by Lemma 6.3, to prove (6.15), it is sufficient to show that where T denotes the number of touchpoints of the LCM with the cumulative sums of the Z ′ i s.
To do this, we use the results of Sparre Andersen (1954). He considers exchangeable random variables X 1 , X 2 , . . . and their partial sums S 0 = 0, S 1 , S 2 , . . . , S n = n i=1 X i , and shows that the number H n of values i ∈ {1, . . . , n − 1} for which S i coincides with the least concave majorant (equivalently the greatest convex minorant) of the sequence S 0 , . . . , S n has mean given by as long as the random variables X 1 , . . . , X n are symmetrically dependent and P (S i /i = S j /j) = 0, 1 ≤ i < j ≤ n.
The vector X 1 , . . . , X n is symmetrically dependent if its joint cumulative distribution function P (X i ≤ x i , i = 1, . . . , n) is a symmetric function of x 1 , . . . , x n . This result is Theorem 5 in Sparre Andersen (1954). Clearly, we have that E[T − 1] = E[H k ], for X 1 = Z 1 , . . . , X n = Z k , which are exchangeable, and satisfy the required conditions. The result follows.
Proof of Remark 3.11. To prove this result we continue with the notation of the previous proof. Equality of gren(Y ) with Y holds if and only if the above partition T = {0, . . . , y}. By Theorem 5 of Sparre Andersen (1954), this occurs with probability 1/(y + 1)!.