Two-sample test for sparse high-dimensional multinomial distributions

Abstract

In this paper we consider testing the equality of probability vectors of two independent multinomial distributions in high dimension. The classical Chi-square test may have some drawbacks in this case since many of cell counts may be zero or may not be large enough. We propose a new test and show its asymptotic normality and the asymptotic power function. Based on the asymptotic power function, we present an application of our result to a neighborhood-type test which has been previously studied, especially for the case of fairly small p values. To compare the proposed test with existing tests, we provide numerical studies including simulations and real data examples.

Appendix: Proof of Lemma 1

We show the ratio consistency of $\hat{\sigma }_k^2$. To show the ratio consistency of $\hat{\sigma }_k^2$, by using $n_1 \asymp n_2$ and $\sigma _k^2 \asymp n^{-2} ||\mathbf{P}_1 + \mathbf{P}_2||^2_2$, it is sufficient to show

$$\begin{aligned}&\frac{\hat{\sigma }_k^2 -\sigma _k^2}{\sigma _k^2} \asymp \frac{ \sum _{i=1}^{k}({\hat{p}}^2_{1i} -\frac{{\hat{p}}_{1i}}{n_1} -p_{1i}^2)}{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \nonumber \\&\quad + \,\frac{\sum _{i=1}^{k}{\hat{p}}_{1i}{\hat{p}}_{2i} -\sum _{i=1}^{k}p_{1i}p_{2i} }{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} + \frac{ \sum _{i=1}^{k}({\hat{p}}^2_{2i} -\frac{{\hat{p}}_{2i}}{n_2} -p_{2i}^2)}{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \nonumber \\&\quad \overset{p}{\rightarrow }0. \end{aligned}$$

(25)

We first show the ratio consistency of $\sum _{i=1}^{k}\left( {\hat{p}}_{1i}^2 -\frac{{\hat{p}}_{1i}}{n_1}\right)$ for $\sum _{i=1}^{k}p_{1i}^2$. The case of the 2nd group ($\sum _{i=1}^{k}\left( {\hat{p}}_{1i}^2 -\frac{{\hat{p}}_{1i}}{n_1}\right)$ for $\sum _{i=1}^{k}p_{1i}^2$) can be proved similarly. Since $E({\hat{p}}_{1i} ^2) = p_{1i}^2 + \frac{p_{1i}(1-p_{1i})}{n_1} = (1-\frac{1}{n_1})p_{1i}^2 + \frac{p_{1i}}{n_1}$ where ${\hat{p}}_{1i} = \frac{N_{1i}}{n_1}$, we have $E \left( \frac{n_1}{n_1-1}({\hat{p}}_{1i}^2 - \frac{{\hat{p}}_{1i}}{n_1}) \right) = p_{1i}^2$. Thus we consider the following unbiased estimator of $\sum _{i=1}^k p_{1i}^2$: $\frac{n_1}{n_1-1}\sum _{i=1}^k({\hat{p}}_{1i}^2 - \frac{{\hat{p}}_{1i}}{n_1})$. To show $\frac{\frac{n_1}{n_1-1}\sum _{i=1}^k({\hat{p}}_{1i}^2 - \frac{{\hat{p}}_{1i}}{n_1}) - \sum _{i=1}^{k}p^2_{1i} }{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \overset{p}{\rightarrow }0$, we will show that the following quantity converges to 0 as follows:

$$\begin{aligned}&\frac{E \left[ \left( \left( \frac{n_1}{n_1-1}\right) \sum _{i=1}^k \left( {\hat{p}}_{1i}^2 - \frac{{\hat{p}}_{1i}}{n_1}\right) -\sum _{i=1}^k p_{1i}^2 \right) ^2 \right] }{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \nonumber \\&\quad = \frac{Var\left( \left( \frac{n_1}{n_1-1}\right) \sum _{i=1}^k \left( {\hat{p}}_{1i}^2 - \frac{{\hat{p}}_{1i}}{n_1}\right) \right) }{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \nonumber \\&\quad \le \frac{8}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \left( Var\left( \sum _{i=1}^k {\hat{p}}_{1i}^2\right) +Var\left( \sum _{i=1}^k \frac{{\hat{p}}_{1i}}{n}\right) \right) \nonumber \\&\quad =\frac{8}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} ((I) + (II)) \end{aligned}$$

(26)

where the last inequality in (26) is from $Var(X+Y) \le 2(Var(X)+Var(Y))$ and $n_1/(n_1-1) \le 2$. We decompose (I) into two parts:

$$\begin{aligned} (I)=Var\left( \sum _{i=1}^k {\hat{p}}_{1i}^2\right)= & {} \sum _{i=1}^k Var({\hat{p}}_{1i}^2) + \sum _{i \ne j} Cov\left( {\hat{p}}_{1i}^2, {\hat{p}}_{1j}^2\right) = (A) + (B). \end{aligned}$$

Using the results in Lemma.S2 in supplementary material, for some constants $C_1$ and $C_2$, we have

$$\begin{aligned} (A)= & {} \sum _{i=1}^k \left( E({\hat{p}}_{1i}^4) - (E({\hat{p}}_{1i}^2))^2 \right) \le C_1 \sum _{i=1}^k \left( \frac{p_{1i}^4}{n_1} + \frac{p_{1i}^3}{n_1} + \frac{p_{1i}^2}{n_1^2} +\frac{p_{1i}}{n_1^3} \right) \\ |(B)|= & {} \left| \sum _{i \ne j} \left( E({\hat{p}}_{1i}^2 {\hat{p}}_{1j}^2) -E({\hat{p}}_{1i}^2) E({\hat{p}}_{1j}^2) \right) \right| \\\le & {} C_2 \sum _{i\ne j} \left( \frac{p_{1i}^2 p_{1j}^2}{n_1} + \frac{p_{1i}^2 p_{1j}}{n_1^2} + \frac{p_{1i}p_{1j}^2}{n_1^2} +\frac{p_{1i}p_{1j}}{n_1^3} \right) . \end{aligned}$$

For all the terms in the above, we can show $\frac{(A)}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \rightarrow 0$ and $\frac{(B)}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \rightarrow 0$ as follows: first, note that $\frac{\max _i p_{ci}^2}{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \rightarrow 0$ since $\frac{\max _i p_{ci}^2}{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \le \frac{\max _i p_{ci}}{||\mathbf{P}_c ||_2^2} \rightarrow 0$ from Condition 2 in Theorem 1. For (A), using $\max _i p_{1i}^2 \le \max _i p_{1i} \rightarrow 0$ in result 2 in Lemma.S2 in the supplementary material, we have

$$\begin{aligned} \frac{\sum _{i=1}^k p_{1i}^4}{n ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{\max _i p_{1i}^2 \sum _{i=1}^k p_{1i}^2}{ n_1 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4} } = \frac{\max _i p_{1i}^2}{n_1 ||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \rightarrow 0,\\ \frac{\sum _{i=1}^k p_{1i}^3}{n ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{\max _i p_{1i} \sum _{i=1}^k p_{1i}^2}{ n_1 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4} } = \frac{\max _i p_{1i}}{n_1} \rightarrow 0,\\ \frac{\sum _{i=1}^k p_{1i}^2}{n_1^2 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{1}{ n_1 (n_1 ||\mathbf{P}_1 + \mathbf{P}_2||^2_2) } \rightarrow 0,~~\frac{\sum _{i=1}^k p_{1i}}{n_1^3 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4} }\\\le & {} \frac{1}{ n_1 (n_1||\mathbf{P}_1 + \mathbf{P}_2||^2_2)^2 } \rightarrow 0 \end{aligned}$$

where Condition 3 ($n||\mathbf{P}_1 + \mathbf{P}_2||^2_2\ge \epsilon >0$) and $n_1 \asymp n_2$ are used in the last steps as $n_1 \rightarrow \infty$. For (B), using $\sum _{i\ne j} p_{1i}^2 p_{1j}^2 \le ||\mathbf{P}_1 + \mathbf{P}_2||^2_2$ and $\sum _{i\ne j} p_{1i} p_{1j} \le \sum _{i=1}^k p_{1i} =1$, we have from Conditions 1–3 in Theorem 1

$$\begin{aligned} \frac{\sum _{i \ne j} p_{1i}^2 p_{1j}^2 }{n_1 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{1}{n_1} \rightarrow 0,~~~~~~\frac{\sum _{i \ne j} p_{1i}^2 p_{1j} }{n_1^2 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \rightarrow 0, \\ \frac{\sum _{i \ne j} p_{1i} p_{1j}^2 }{n_1^2 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{||\mathbf{P}_1 + \mathbf{P}_2||^2_2}{n_1^2 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4} } \le \frac{1}{n_1 ||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \rightarrow 0, \\ \frac{\sum _{i \ne j} p_{1i} p_{1j} }{n_1^3 ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\le & {} \frac{1}{n_1 (n_1||\mathbf{P}_1 + \mathbf{P}_2||^2_2)^2} \rightarrow 0. \end{aligned}$$

Similarly, for (II), we have

$$\begin{aligned} (II) = Var\left( \sum _{i=1}^k \frac{{\hat{p}}_{1i}}{n_1}\right)= & {} \sum _{i=1}^k Var\left( \frac{{\hat{p}}_{1i}}{n_1}\right) + \frac{1}{n_1^2} \sum _{i \ne j} Cov\left( {\hat{p}}_{1i}, {\hat{p}}_{1j}\right) \\= & {} \sum _{i=1}^k \frac{p_{1i}(1-p_{1i})}{n_1^3} - \sum _{i\ne j} \frac{p_{1i}p_{1j}}{n_1^3} \le \sum _{i=1}^{k}\frac{p_{1i}}{n_1^3} =\frac{1}{n_1^3}. \end{aligned}$$

Therefore, we have $\frac{(II)}{n_1^3||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \le \frac{1}{n_1^3||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \rightarrow 0$ which leads

$$\begin{aligned} \frac{ \sum _{i=1}^{k}\left( {\hat{p}}^2_{1i} -\frac{{\hat{p}}_{1i}}{n_1}\right) -\sum _{i=1}^{k}p_{1i}^2}{||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \overset{p}{\rightarrow }0. \end{aligned}$$

(27)

The ratio consistent estimator of $\sum _{i=1}^{k}\left( {\hat{p}}_{2i}^2 - \frac{{\hat{p}}_{2i}}{n_2}\right)$ can be also proved in the same way.

For $\sum _{i=1}^{k}{\hat{p}}_{1i}{\hat{p}}_{2i}$, we show

$$\begin{aligned}&\frac{E((\sum _{i=1}^{k}{\hat{p}}_{1i}{\hat{p}}_{2i})^2)}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} = \frac{\sum _{i=1}^k Var(N_{1i}N_{2i}) + \sum _{i\ne j}\mathrm{Cov}(N_{1i}N_{2i},N_{1j}N_{2j})}{||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \\&\quad \asymp \frac{\sum _{i=1}^k p_{1i}^2 p_{2i}}{n ||\mathbf{P}_1 + \mathbf{P}_2||_2^{4} }+ \frac{\sum _{i=1}^k p_{1i} p_{2i}^2}{n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} + \frac{1}{n^2 ||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \\&\quad - \frac{\sum _{i\ne j}p_{1i}p_{2i}p_{1j}p_{2j}}{n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \\&\quad \asymp \frac{\max _i p_{1i} }{n||\mathbf{P}_1 + \mathbf{P}_2||^2_2} +\frac{\max _i p_{2i} }{n||\mathbf{P}_1 + \mathbf{P}_2||^2_2} + \frac{\sum _{i=1}^k p_{1i} p_{2i}^2}{n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}}\\&\quad + \frac{1}{n^2 ||\mathbf{P}_1 + \mathbf{P}_2||^2_2} - \frac{ (\mathbf{P}_1\cdot \mathbf{P}_2)^2 }{n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \rightarrow 0 \end{aligned}$$

where the last term converges to 0 since $\frac{ (\mathbf{P}_1\cdot \mathbf{P}_2)^2 }{n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} \le \frac{||\mathbf{P}_1 + \mathbf{P}_2||^2_2}{2n||\mathbf{P}_1 + \mathbf{P}_2||_2^{4}} = \frac{1}{2n||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \rightarrow 0$ from Condition 3 in Theorem 1. Therefore

$$\begin{aligned} \frac{\sum _{i=1}^{k}{\hat{p}}_{1i}{\hat{p}}_{2i} - \sum _{i=1}^{k}p_{1i}p_{2i} }{ ||\mathbf{P}_1 + \mathbf{P}_2||^2_2} \overset{p}{\rightarrow }0 \end{aligned}$$

(28)

Combining (27) and (28), we have (25) which leads to the ratio consistency of $\hat{\sigma }_k^2$.