Sample size and positive false discovery rate control for multiple testing

: The positive false discovery rate (pFDR) is a useful overall mea- sure of errors for multiple hypothesis testing, especially when the underlying goal is to attain one or more discoveries. Control of pFDR critically depends on how much evidence is available from data to distinguish between false and true nulls. Oftentimes, as many aspects of the data distributions are unknown, one may not be able to obtain strong enough evidence from the data for pFDR control. This raises the question as to how much data are needed to attain a target pFDR level. We study the asymptotics of the minimum number of observations per null for the pFDR control associated with multiple Studentized tests and F tests, especially when the diﬀerences between false nulls and true nulls are small. For Studentized tests, we con- sider tests on shifts or other parameters associated with normal and general distributions. For F tests, we also take into account the eﬀect of the num- ber of covariates in linear regression. The results show that in determining the minimum sample size per null for pFDR control, higher order statistical properties of data are important, and the number of covariates is important in tests to detect regression eﬀects.


Introduction
A fundamental issue for multiple hypothesis testing is how to effectively control Type I errors, namely the errors of rejecting null hypotheses that are actually true. The False Discovery Rate (FDR) control has generated a lot of interest due to its more balanced trade-off between error rate control and power than the traditional Familywise Error Rate control (1). For recent progress on FDR control and its generalizations, see (6-12, 14-16, 19) and references therein.
Let R be the number of rejected nulls and V the number of rejected true nulls. By definition, FDR = E[V /(R∨1)]. Therefore, in FDR control, the case R = 0 is counted as "error-free", which turns out to be important for the controllability of the FDR. However, multiple testing procedures are often used in situations where one explicitly or implicitly aims to obtain a nonempty set of rejected nulls. To take into account this mind-set in multiple testing, it is appropriate to control the positive FDR (pFDR) as well, which is defined as E[V /R | R > 0] (18). Clearly, when all the nulls are true, the pFDR is 1 and therefore cannot be controlled. This is a reason why the FDR is defined as it is (1). On the other hand, even when there is a positive proportion of nulls that are false, the pFDR can still be significantly greater than the FDR, such that when some nulls are indeed rejected, chance is that a large proportion or even almost all of them are falsely rejected (3,4).
The gap between FDR and pFDR arises when the test statistics cannot provide arbitrarily strong evidence against nulls (4). Such test statistics include t and F statistics (3). These two share a common feature, that is, they are used when the standard deviations of the normal distributions underlying the data are unknown. In reality, it is a rule rather than exception that data distributions are only known partially. This suggests that, when evaluating rejected nulls, it is necessary to realize that the FDR and pFDR can be quite different, especially when the former is low.
In order to increase the evidence against nulls, a guiding principle is to increase the number of observations for each null, denoted n for the time being. In contrast to single hypothesis testing, for problems that involve a large number of nulls, even a small increase in n will result in a significant increase in the demand on resources. For this reason, the issue of sample size per null for multiple testing needs to be dealt with more carefully. It is known that FDR and other types of error rates decrease in the order of O( log n/n) (13). In this work, we will consider the relationship between n and pFDR control, in particular, for the case where false nulls are hard to separate from true ones. The basic question to be considered is: in order to attain a certain level of pFDR, what is the minimum value for n. This question involves several issues. First, how does the complexity of the null distribution affect n? Second, is normal or t approximation appropriate in determining n? In other words, is it necessary to incorporate information on higher order moments of the data distribution? Third, what would be an attainable upper bound for the performance of a multiple testing procedure based on partial knowledge of the data distributions?
In the rest of the section, we first set up the framework for our discussion, and then outline the other sections.

Setup and basic approach
Most of the discussions will be made under a random effects model (10,18). Each null H i is associated with a distribution F i and tested based on ξ i = ξ(X i1 , . . . , X in ), where X i1 , . . . , X in are iid ∼ F i and the function ξ is the same for all H i . Let θ i = 1 {H i is true}. The random effects model assumes that (θ i , ξ i ) are independent, such that where π ∈ [0, 1] is a fixed population proportion of false nulls among all the nulls. Note that P (n) i of depend on n, the number of observations for each null. It follows that the minimum pFDR is (cf. (4)) α * = 1 − π 1 − π + πρ n , with ρ n := sup p (n) 1 p (n) 0 .
( 1.2) In order to attain pFDR ≤ α, there must be α * ≤ α, which is equivalent to (1 − α)(1 − π)/(απ) ≤ ρ n . For many tests, such as t and F tests, ρ n < ∞ and ρ n ↑ ∞ as n → ∞. Then, the minimum sample size per null is n * = min {n : (1 − α)(1 − π)/(απ) ≤ ρ n } . (1.3) In general, the smaller the difference between the distributions F i under false nulls and those under true nulls, the smaller ρ n become, and hence the larger n * has to be. Our interest is how n * should grow as the difference between the distributions tends to 0.

Outlines of other sections
Section 2 considers t tests for normal distributions. The nulls are H i : µ i = 0 for N (µ i , σ i ), with σ i unknown. It will be shown that if µ i /σ i ≡ r for false nulls, then, as r ↓ 0, the minimum sample size per null ∼ (1/r) ln Q α, π and therefore it depends on at least 3 factors: 1) the target pFDR control level, α, 2) the proportion of false nulls among the nulls, π, 3) and the distributional properties of the data, as reflected by µ i /σ i . In contrast, for FDR control, there is no constraint on the sample size per null. The case where µ i /σ i associated with false nulls are sampled from a distribution will be considered as well. This section also illustrates the basic technique used throughout the article.
Section 3 considers F tests. The nulls are H i : β i = 0 for Y = β T i X + ǫ, where X consists of p covariates and ǫ ∼ N (0, σ i ) is independent of X. Each H i is tested with the F statistic of a sample (Y ik , X k ), k = 1, . . . , n + p, where n ≥ 1 and X 1 , . . . , X n+p consist a fixed design for the nulls. Note that n now stands for the difference between the sample size per null and the number of covariates included in the regression. The asymptotics of n * , the minimum value for n in order to attain a given pFDR level, will be considered as the regression effects become increasingly weak and/or as p increases. It will be seen that n * must stay positive. The weaker the regression effects are, the larger n * has to be. Under certain conditions, n * should increase at least as fast as p.
Section 4 considers t tests for arbitrary distributions. We consider the case where estimates of means and variances are derived from separate samples, which allows detailed analysis with currently available tools, in particular, uniform exact large deviations principle (LDP) (2). It will be shown that the minimum sample size per null depends on the cumulant generating functions of the distributions, and thus on their higher order moments. The asymptotic results will be illustrated with examples of uniform distributions and Gamma distributions. An example of normal distributions will also be given to show that the results are consistent with those in Section 2. We will also consider how to split the random samples for the estimation of mean and the estimation of variance in order to minimize the sample size per null.
Section 5 considers tests based on partial information on the data distributions. The study is part of an effort to address the following question: when knowledge about data distributions is incomplete and hence Studentized tests are used, what would be the attainable minimum sample size per null. Under the condition that the actual distributions belong to a parametric family which is unknown to the data analyzer, a Studentized likelihood test will be studied. We conjecture that the Studentized likelihood test attains the minimum sample size per null. Examples of normal distributions, Cauchy distributions, and Gamma distributions will be given.
Section 6 concludes the article with a brief summary. Most of the mathematical details are collected in the Appendix.

Main results
Suppose we wish to conduct hypothesis tests for a large number of normal distributions N (µ i , σ i ). However, neither σ i nor any possible relationships among (µ i , σ i ), i ≥ 1, are known. Under this circumstance, in order to test H i : µ i = 0 simultaneously for all N (µ i , σ i ), an appropriate approach is to use the t statistics of iid samples Y i1 , . . . , Y i,n+1 ∼ N (µ i , σ i ): Suppose the sample size n + 1 is the same for all H i and the samples from different normal distributions are independent of each other. Under the random effects model (1.1), we first consider a case where distributions with µ i = 0 share a common characteristic, i.e., signal-noise ratio defined in the remark following Theorem 2.1.
Theorem 2.1. Under the above condition, suppose that, unknown to the data analyzer, when H i is false, µ i /σ i = r > 0, where r is a constant independent of i. Given 0 < α < 1, let n * be the minimum value of n in order to attain pFDR ≤ α. Then n * ∼ (1/r) ln Q α, π as r → 0+.
Remark. We will refer to r as the signal-noise ratio (SNR) of the multiple testing problem in Theorem 2.1.
Theorem 2.1 can be generalized to the case where the SNR follows a distribution. To specify how the SNR becomes increasingly small, we introduce a "scale" parameter s > 0 and parameterize the SNR distribution as G s (r) = G(sr), where G is a fixed distribution.

Preliminaries
Recall that, for the t statistic (2.1), if µ = 0, then T ∼ t n , the t distribution with n degrees of freedom (dfs). On the other hand, if µ > 0, then T ∼ t n,δ , the noncentral t distribution with n dfs and (noncentrality) parameter δ = √ n + 1µ/σ, with density t n,δ (x) = n n/2 √ π Γ(n/2) Apparently t n,0 (x) = t n (x). Denote a n,k = Γ n + k + 1 2 Γ n + 1 2 . Then It can be shown that t n,δ (x)/t n (x) is strictly increasing in x and (3)). Since the supremum of likelihood ratio only depends on n and r = µ/σ, it will be denoted by L(n, r) henceforth.

Proofs of the main results
We need two lemmas. They will be proved in the Appendix. The proofs of the main results are rather straightforward. The proofs are given in order illustrate the basic argument, which is used for the other results of the article as well.
Proof of Theorem 2.1. By (1.2), in order to get pFDR ≤ α, Let n * be the minimum value of n in order for the inequality to hold. Then by Lemma 2.1, as r = µ/σ → 0, n * r → ln Q α, π , implying Theorem 2.1.
Proof of Corollary 2.1. Following the argument for (1.2), it is seen that under the conditions of the corollary, the minimum attainable pFDR is Then the corollary follows from a similar argument for Theorem (2.1).

Main results
Suppose we wish to test H i : β i = 0 simultaneously for a large number of joint distributions of Y and X, such that under each distribution, Y = β T i X + ǫ i , where β i ∈ R p are vectors of linear coefficients and ǫ i ∼ N (0, σ i ) are independent of X. Suppose neither σ i or any possible relationships among σ i are known. Under this condition, consider the following tests based on a fixed design. Let X k , k ≥ 1, be fixed vectors of covariates. Let n + p be the sample size per null. For each i, let (Y i1 , X 1 ), . . . , (Y i,n+p , X n+p ) be an independent sample from Y = β T i X + ǫ. Assume that the samples for different H i are independent of each other.
Suppose that, unknown to the data analyzer, for all the false nulls H i , where δ > 0. This situation arises when all X k are within a bounded domain, either because only regression within the domain is of interest, or because only covariates within the domain are observable or experimentally controllable. Note that n is not the sample size per null. Instead, it is the difference between the sample size per null and the number of covariates in each regression equation. Given α ∈ (0, 1), let n * = inf {n : pFDR ≤ α for F tests on H i under the constraint (3.1)} .
It can be seen that n * is attained when equality holds in (3.1) for all the false nulls. The asymptotics of n * will be considered for 3 cases: 1) δ → 0 while p is fixed, 2) δ → 0 and p → ∞, and 3) p → ∞ while δ is fixed. The case δ → 0 is relevant when the regression effects are weak, and the case p → ∞ is relevant when a large number of covariates are incorporated.

Preliminaries and proofs
On the other hand, if β = 0, the F statistic follows the noncentral F distribution with (p, n) dfs and (noncentrality) parameter ∆, where
Then for x ≥ 0, which is strictly increasing, and First, it is easy to see that the following statement is true.
It follows that, under the constraint (3.1), the supremum of the likelihood ratio is attained when ∆ = (n + p)δ 2 and is equal to Therefore, under the random effects model (1.1), pFDR ≤ α is equivalent to K(p, n, δ) ≥ Q α, π . Theorem 3.1 then follows from the lemmas below and an argument as to that of Theorem 2.1. The proof of Theorem 3.1 is omitted for brevity. The proofs of the lemmas are given in the Appendix.
4. Multiple t-tests: a general case

Setup
Suppose we wish to conduct hypothesis tests for a large number of distributions F i in order to identify those with nonzero mean µ i . The tests will be based on random samples from F i . Assume that no information on the forms of F i or their relationships is available. As a result, samples from different F i cannot be combined to improve the inference. As in the case of testing mean values for normal distributions, to test H i : µ i = 0 simultaneously, an appropriate approach is to use the t statistics T i = √ nμ i /σ i , where bothμ i andσ 2 i are derived solely from the sample from F i , and n is the number of observations used to getμ i .
Again, the goal is to find the minimum sample size per null in order to attain a given pFDR level, in particular when F i under false H i only have small differences from those under true H i . The results will also answer the following question: are normal or t approximations appropriate for the t statistics in determining the minimum sample size per null?
We only consider the case where µ i is either 0 or µ 0 = 0, where µ 0 is a constant. In order to make the analysis tractable, the problem needs to be formulated carefully. First, unlike the case of normal distributions, in general, ifμ i andσ 2 i are the mean and variance of the same random sample, they are dependent andσ 2 i cannot be expressed as the sum of iid random variables. As seen below, the analysis on the minimum sample size per null requires detailed asymptotics of the t statistics, in particular, the so called exact LDP (2, 5). For Studentized statistics, there are LDP techniques available (17). However, currently, exact LDP techniques cannot handle complex statistical dependency very well. To get around this technical difficulty, we consider the following t statistics. Suppose the samples from different F i are independent of each other, and contain the same number of iid observations. Divide the sample from F i into two parts, Thenμ i andσ 2 i are independent, andσ 2 i is the sum of iid random variables. Second, the minimum attainable pFDR depends on the supremum of the ratio of the actual density of T i and its theoretical density under H i . In general, neither one is tractable analytically. To deal with this difficulty, observe that in the case of normal distributions, the supremum of the ratio equals We therefore consider the pFDR under the rule that H i is rejected if and only if T i > x, where x > 0 is a critical value. In order to identify false nulls as µ 0 → 0, Recall Section 2. Some analysis on (2.2) and (2.3) shows that, for normal distributions, the supremum of the likelihood ratio can be obtained asymptotically by letting x = c n √ n, where c n > 0 is an arbitrary sequence converging to ∞; specifically, given a > 0, as r ↓ 0 and n ∼ a/r, If, instead, x increases in the same order as √ n or more slowly, the above limit is strictly less than 1. Based on this observation, for the general case, we set x = c n √ n, with c n → ∞. In general, there is no guarantee that using c n growing at a specific rate can always yield convergence. Thus, we require that c n grow slowly.
Under the setup, suppose that, unknown to the data analyzer, when H i : Under the random effects model (1.1), the minimum attainable pFDR is The question now is the following: • Given α ∈ (0, 1), as d → 0, how should N increase so that α * ≤ α?

Main results
By the Law of Large Numbers, as n → ∞ and m → ∞,X n → 0 and S m → σ w.p. 1. On the other hand, by our selection, z N → ∞. In order to analyze (4.2) as d → 0, we shall rely on exact LDP, which depends on the properties of the cumulant generating functions It is easy to see that g(t) = g(−t) for t > 0. Recall that a function ζ is said to be slowly varying at ∞, if for all t > 0, lim x→∞ ζ(tx)/ζ(x) = 1.
Theorem 4.1. Suppose the following two conditions are satisfied.
b) The density function g is continuous and bounded on (ǫ, ∞) for any ǫ > 0, and there exist a constant λ > −1 and a function ζ(z) ≥ 0 which is increasing in z ≥ 0 and slowly varying at ∞, such that Fix α ∈ (0, 1). Let N * be the minimum value for N = m + n in order to attain α * ≤ α, where α * is as in (4.2). Then, under the constraints 1) m and n grow in proportion to each other such that m/N → ρ ∈ (0, 1) as m, n → ∞ and 2) z N → ∞ slowly enough, one gets where t 0 > 0 is the unique positive solution to Remark.
(1) By (4.5) and (4.6), N * depends on the moments of F of all orders. Thus, t or normal approximations of the distribution of T in general are not suitable in determining N * in order to attain a target pFDR level.
(2) If z N → ∞ slowly enough such that (4.5) holds, then for any z ′ N → ∞ more slowly, (4.5) holds as well. Presumable, there is an upper bound for the growth rate of z N in order for (4.5) to hold. However, it is not available with the technique employed by this work.
(3) We define N as n + m instead of n + 2m because in the estimator S m , each pair of observations only generate one independent summand. The sum n + m can be thought of as the number of degrees of freedom that are effectively utilized by T .
Following the proof for the case of normal distributions, Theorem 4.1 is a consequence of the following result.
Indeed, by display (4.2) and Proposition 4.1, if dN → T ≥ 0, then the minimum attainable pFDR has convergence (4.8) In order to attain pFDR ≤ α, there must be α * ≤ α, leading to (4.5). The proof of Proposition 4.1 is given in the Appendix A3.

Examples
Example 4.1 (Normal distribution). Under the setup in Section 4.1, let To see the connection to Theorem 2.1, observeX n = σZ/ √ n and S m = σW m / √ m, where Z ∼ N (0, 1) and W 2 m ∼ χ 2 m are independent. Since z N ↑ ∞ slowly, so is a m := n/m z N . Let r m = (d/σ) n/(m + 1). Then where T m,δ denotes the cumulative distribution function (cdf) of the noncentral t distribution with m dfs and parameter δ, and T m the cdf of the t distribution with m dfs. Comparing the ratio in (2.2) and the above ratio, it is seen that the difference between the two is that probabilities densities in (2.2) are replaced with tail probabilities. Since Since m * /N * → ρ, the asymptotic of N * given by Theorem 2.1 is identical to that given by Theorem 4.1.
Example 4.2 (Uniform distributions). Under the setup in Section 4.1, let F = U (− 1 2 , 1 2 ) in (4.1). Then for t > 0, and for t < 0, Λ(t) = Λ(−t). Thus condition a) in Theorem 4.1 is satisfied. It is easy to see that condition b) is satisfied as well, with λ = 0 and ζ(x) ≡ 1 in (4.4). Then by (4.5), Therefore, condition a) in Theorem 4.1 is satisfied. Because the value of λ in (4.4) is invariant to scaling, in order to verify condition b), without loss of generality, It suffices to consider the behavior of k(x) as x ↓ 0. We need to analyze 3 cases.

Optimal split of sample
For the t statistics considered so far, m/N is the fraction of degrees of freedom allocated for the estimation of variance. By (4.5), the asymptotic of N * depends on the fraction in a nontrivial way. It is of interest to optimize the fraction in order to minimize N * . Asymptotically, this is equivalent to maximizing (1 − ρ)t 0 as a function of ρ, with t 0 = t 0 (ρ) > 0 as in (4.6).
Example 4.1 (Continued) By (4.9), it is apparent that the optimal value of ρ is 1/2. In other words, in order to minimize N * , there should be equal number of degrees of freedom allocated for the estimation of mean and the estimation of variance for each normal distribution. In particular, if m ≡ n − 1, then ρ = 1/2, and the resulting t statistic has the same distribution as √ n − 1Z/W n−1 , where Z ∼ N (0, 1) and W n−1 ∼ χ n−1 are independent, which is the usual t statistic of an iid sample of size n.

Motivation
In many cases of multiple testing, only limited knowledge is available on the distributions from which data are sampled. The knowledge relevant to a null hypothesis is expressed by a statistic M such that the null is rejected if and only if the observed value of M is significantly different from 0. In general, as the distribution of M is unknown, M has to be Studentized so that its magnitude can be evaluated.
On the other hand, oftentimes, despite the complexity of the data distributions, it is reasonable to believe they have an underlying structure. Consider the scenario where all the data distributions belong to a parametric family {p θ }, such that the distribution under a true null is p 0 , and the one under a false null is p θ * for some θ * = 0. A question of interest is: under this circumstance, what would be the optimal overall performance of the multiple tests? The question is in the same spirit as questions regarding estimation efficiency. However, it assumes that neither the existence of the parameterization nor its form is known to the data analyzer and all the machinery available is the test statistic M .
As before, we wish to find out the minimum sample size per null required for pFDR control, in particular, as the tests become increasingly harder in the sense that θ * → 0. Our conjecture is that, asymptotically the minimum sample size per null is attained if M "happens" to be ∂[ln p 0 ]/∂θ. By "happens" we mean that the data analyzer is unaware of this peculiar nature of M and uses its Studentized version for the tests. This conjecture is directly motivated by the fact that the MLE is efficient under regular conditions. Although a smaller minimum sample size per null could be possible if M happens to be the MLE, due to Studentization, the improvement appears to diminish as θ → 0. Certainly, had the parameterization been known, the (original) MLE would be preferred. The goal here is not to establish any sort of superiority of Studentized MLE, but rather to search for the optimal overall performance of multiple tests, when we are aware that our knowledge about the data distributions is incomplete and beyond the test statistic, we have no other information.
The above conjecture is not yet proved or disproved. However, as a first step, we would like to obtain the asymptotics of the minimum sample size per null when Studentized ∂[ln p 0 ]/∂θ is used for multiple tests. We shall also provide some examples to support the conjecture.

Setup
Let (Ω, F ) be a measurable space equipped with a σ-finite measure µ. Let {p θ : θ ∈ [0, 1]} be a parametric family of density functions on (Ω, F ) with respect to µ. Denote by P θ the corresponding probability measure. Under the random effects model (1.1), each null H i is associated with a distribution F i , such that when H i is true, F i = P 0 , and when H i is false, F i = P θ , where θ > 0 is a constant. Assume that each H i is tested based on an iid sample {ω ij } from F i , such that the samples for different H i are independent, and the sample size is the same for all H i .
Condition 4 For any q > 0, there is θ ′ = θ ′ (q) > 0, such that Remark. By Condition 1, for any interval I in [0, 1], the extrema of r θ (ω) over θ ∈ I are measurable. Thus the expectation in (5.2) is well defined. For brevity, for θ ∈ [0, 1] and n ≥ 1, the n-fold product measure of P θ is still denoted by P θ , and the expectation under the product measure by E θ . We shall denote by ω, ω ′ , ω i , ω ′ i generic iid elements under a generic distribution on (Ω, F ). Denote For m, n ≥ 1, denote Sincel θ (ω) =ṗ θ (ω)/p θ (ω), from Conditions 1-4 and dominated convergence, it follows that E 0l0 = 0 and As a result, for θ > 0 close to 0, E θl0 (ω) > 0. This justifies using the upper tail of √ nX n /S m for testing. The multiple tests are such that whereX in and S im are computed the same way asX n and S m , except that they are derived from ω i1 , . . . , ω in , ω ′ i1 , . . . , ω ′ i,2m iid ∼ F i , N = n + m, and z N → ∞ as N → ∞. Then, under the random effects model, the minimum attainable pFDR is The question now is the following: • Given α ∈ (0, 1), as θ ↓ 0, how should N increase so that α * ≤ α?

Main results
Denote the cumulant generating functions Note that the expectation is taken under P 0 . b) Under P 0 , X has a density f continuous almost everywhere on R. Furthermore, either (i) f is bounded or (ii) f is symmetric and X L ∞ (P0) < ∞.
c) Under P 0 , the density g of X − Y is continuous and bounded on (ǫ, ∞) for any ǫ > 0, and there exist a constant λ > −1 and a function ζ(z) ≥ 0 increasing in z ≥ 0 and slowly varying at ∞, such that d) There are s > 0 and L > 0, such that Fix α ∈ (0, 1). Let N * be the minimum value of N = n + m in order to attain α * ≤ α, where α * is as in (5.5). Then, under the constraints 1) m and n grow in proportion to each other such that m/N → ρ ∈ (0, 1) as m, n → ∞ and 2) z N → ∞ slowly enough, one gets where t 0 is the unique positive solution to (4.6), and Remark. By symmetry, to verify (5.8), it is enough to only consider u > 0. Moreover, (5.8) holds if its left hand side is a bounded function of u.
Following the proofs of the previous results, Theorem 5.1 is a consequence of Proposition 5.1, which will be proved in Appendix A4.
Remark. Because the Cauchy distributions have infinite variance, t tests cannot be used to test the nulls. The example shows that even in this case, Studentized ℓ 0 (ω) can still distinguish between true and false nulls.

Summary
Multiple testing is often used to identify subtle real signals (false nulls) from a large and relatively strong background of noise (true nulls). In order to have some assurance that there is a reasonable fraction of real signals among the signals "spotted" by a multiple testing procedure, it is useful to evaluate the pFDR of the procedure. Comparing to FDR control, pFDR control is more subtle and in general requires more data. In this article, we study the minimum number of observations per null in order to attain a target pFDR level and show that it depends on several factors: 1) the target pFDR control level, 2) the proportion of false nulls among the nulls being tested, 3) distributional properties of the data in addition to mean and variance, and 4) in the case of multiple F tests, the number of covariates included in the nulls.
The results of the article indicate that, in determining how much data are needed for pFDR control, if there is little information about the data distributions, then it may be useful to estimate the cumulant generating functions of the distributions. Alternatively, if one has good evidence about the parametric form of the data distributions but has little information on the values of the parameters, then it may be necessary to determine the number of observations per null based on the cumulant functions as well. In either case, typically it is insufficent to only use the means and variances of the distributions.
The article only considers univariate test statistics, which allow detailed analysis of tail probabilities. It is possible to test each null by more than one statistic. How to determine the number of observations per null for multivariate test statistics is yet to be addressed.

References
The right hand side has a finite sum over k. By dominated convergence, This yields 2). To show 3), by similar argument, given 0 < c < 1, for n ≫ 1, Therefore, as nr → ∞, L(n, r) ≥ ce nr → ∞.
In order to prove Lemma 3.4, we need the following result.
3) Λ * is smooth and strictly convex on I Λ , and On the other hand, The next lemma is key to the proof of Proposition 4.1. Basically, it says that the analysis on the ratio of the extreme tail probabilities can be localized around a specific value determined by Λ and the index λ in (4.4). As a result, the limit (4.7) can be obtained by the uniform exact large deviations principle (LDP) in (2), which is a refined version of the exact LDP (5).
Proof of Proposition 4.1. Recall that d N → 0 and N → ∞, such that d N N → T . First, we show that, given ǫ > 0, there is z 0 > 0, such that Let z 0 > 0 and η > 0 such that (A3.3) holds. Fix z ≥ z 0 . Denote a = a(z) = (ν 0 − δ)/z and b = b(z) = (ν 0 + δ)/z. Because of (A3.3), in order to show (A3.4), it suffices to establish Let G m (x) be the distribution function of S m . Then From these equations, it is not hard to see that (A3.5) follows if we can show To establish (A3.6), observe that for N > 1 large enough and x ∈ [a, b], zx − d N ∈ [a/2, ν 0 + δ]. Therefore, τ N (x) := η Λ (zx − d N ) is not only well defined but also continuous and strictly positive on [a, b]. By Theorem 3.3 of (2), as N → ∞, the following approximation holds, which is a uniform version of the exact LDP due to Bahadur and Rao (5, Theorem 3.7.4).
Because τ N (x) → η Λ (zx) uniformly on [a, b] and the latter is strictly positive and continuous on [a, b], the above inequality yields sup x∈ [a,b] By the above approximations to P (X n ≥ zx − d N ) and P (X n ≥ zx), in order to prove (A3.6), it is enough to show By Taylor expansion and Lemma A3.1, where ξ = ξ(x) ∈ (0, 1). Therefore, Therefore (A3.5) is proved. Now that (A3.4) holds for any given ǫ > 0, as long as z ≥ z 0 = z 0 (ǫ), with z 0 being large enough, by the diagonal argument, we can choose z N > 0 in such as way that z N → ∞ slowly as N → ∞ and lim N →∞ This finishes the proof of the theorem.

A3.2. Proof of Lemma A3.2
The proof needs a few preliminary results. The first lemma collects some useful properties of Ψ.
Proof. We only show Ψ(t) → −∞ as t → −∞ and (A3.7), which are properties specifically due to condition b) in Theorem 4.1. The proof of the rest of Lemma A3.3 is standard.
To get Ψ(t) → −∞ as t → ∞, it suffices to show ∞ 0 e −tu 2 /2 g(u) du → 0 as t → ∞. For later use, it will be shown that, given s ≥ 0, The proof is based on several truncations of the integral. Given 0 < η < 1, there is 0 < ǫ < 1, such that On the other hand, x s+λ e −tx 2 /2 dx.
The right hand side is of the same order as ∞ 0 x s+λ e −tx 2 /2 dx, which in turn is of the same order as t −(λ+s+1)/2 . As a result, as t → ∞.
It can be seen that, for N > D/ min(ǫ, r 0 , r 1 , . . . , r p ), For the latter ones, by the choice of z and r i , u i − 2r i > 0 and (u i + r i )/z < σ 2 . Therefore, by the LDP, Similarly, lim(1/N ) ln(1/A 0 ) ≥ J z (ν) + η. Since there is only a finite number of The proof is thus complete.

A4.1. Proof of the main result
This section is devoted to the proof of Proposition 5.1. The proof is based on several lemmas. Henceforth, let N = m + n and ν 0 = Λ ′ (t 0 ), where t 0 is the positive solution to (4.6). It will be assumed that as m → ∞ and n → ∞, m/N → ρ ∈ (0, 1), where ρ is fixed.

A4.2. Proof of Lemmas
We need the next result to show Lemma A4.1.
To show the second part of the lemma, for each n ≥ 1, which yields Let k → ∞ and take lim on both ends. By the assumption, Thus we get Because a is arbitrary, the lemma is proved.
It is easy to check that under the assumptions of Theorem 5.1, all the statements in Lemmas A3.1 and A3.3 hold for Λ and Ψ defined in (5.6), with X =l 0 (ω), Y =l 0 (ω ′ ). Therefore, Lemma A3.2 can be applied.
Then E 0 (e aŪm | S m = t) = m i=1 φ vi (a/m)1 {v i = 0} P 0 (dζ | S m = t). (A4.11) Case i: f is bounded In this case, is the conditional density of U given V = v and φ v (z) = e zu h v (u) du. Since f is continuous almost everywhere and bounded, by condition a) of Theorem 5.1, there is r > 0 such that sup v e r|u| f (u + v)f (u − v) du < ∞, and by dominated convergence, as v → 0, g(v) → g(0) = f 2 ∈ (0, ∞). It follows that there is c > 0, such that {φ v (z), v ∈ [−c, c]} is a family of smooth functions of z ∈ [−r, r] with uniformly continuous and bounded φ ′ v (z) and φ ′′ v (z). Given η > 0, decrease c if necessary so that