Multiple Comparison

Multiplicity Issues

Statistical evidence is obtained by rejecting the null hypothesis at a “small” prespecified significance level α, say 0.05 or 0.01, which is an acceptable level of probability of the type I error (the error of rejecting the “true” null hypothesis). If we have a family of multiple hypotheses in a confirmatory experiment and test them simultaneously at each level α, the overall or familywise type I error rate (FWER), i.e., the probability of rejecting at least one “true” null hypothesis in the family, may inflate and exceed α, even if there exist no treatment differences. We call such inflation of the FWER a multiplicity issue.

Usually there may be some correlation structure between test statistics, and the inflation of the FWER might not be so remarkable. However, if we have multiple hypotheses to be tested for confirmatory purpose, we should adjust for multiplicity so as to control the FWER within α. This is called multiplicity adjustment. Testing procedures for...