Skip to main content
SpringerLink
Log in

SCORE Study Report 8: Closed Tests for All Pairwise Comparisons of Means

  • Biostatistics
  • Published:
Drug information journal : DIJ / Drug Information Association Aims and scope Submit manuscript

Abstract

We compare five closed tests for strong control of family-wide type 1 error while making all pairwise comparisons of means in clinical trials with multiple arms such as the SCORE Study. We simulated outcomes of the SCORE Study under its design hypotheses, and used P values from chi-squared tests to compare performance of a pairwise closed test described below to Bonferroni and Hochberg adjusted P values. Pairwise closed testing was more powerful than Hochberg’s method by several definitions of multiple-test power. Simulations over a wider parameter space, and considering other closed methods, confirmed this superiority for P values based on normal, logistic, and Poisson distributions. The power benefit of pairwise closed testing begins to disappear with five or more arms and with unbalanced designs. For trials with four or fewer arms and balanced designs, investigators should consider using pairwise closed testing in preference to Shaffer’s, Hommel’s, and Hochberg’s approaches when making all pairwise comparisons of means. If not all P values from the closed family are available, Shaffer’s method is a good choice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika. 1988; 75:800–802.

    Article  Google Scholar 

  2. Sidak Z. Rectangular confidence regions for the means of multivariate normal distributions. J Am Stat Assoc. 1967;62:626–633.

    Google Scholar 

  3. Fisher RA. Combining independent tests of significance. Am Stat. 1948;2(S):30.

    Google Scholar 

  4. Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65–70.

    Google Scholar 

  5. Simes RJ. An improved Bonferroni procedure for multiple tests of significance. Biometrika. 1986; 73:751 –754.

    Article  Google Scholar 

  6. Hommel G. A comparison of two modified Bon-fcrroni procedures. Biometrika. 1988;75:383–386.

    Article  Google Scholar 

  7. Rom DM. A sequentially rejective test procedure based on a modified Bonferroni inequality, Biometrika. 1990;77:663–665.

    Article  Google Scholar 

  8. Westfall PH, Tobias RD. Multiple testing of general contrasts: truncated closure and the extended Shaffer-Royen method. J Am Stat Assoc. 2007; 102:487–494.

    Article  CAS  Google Scholar 

  9. Westfall PH. Young SS. Resampling-Based Multiple Testing. New York: Wiley: 1993.

    Google Scholar 

  10. Scheffe H. A method for judging all contrasts in the analysis of variance. Biometrika. 1953;40:87– 104.

    Google Scholar 

  11. Tukey JW. The problem of multiple comparisons. In: Braun HI, ed. The Collected Works of John W. Tukey: Multiple Comparisons, vol. 8. New York: Chapman and Hall: 1994.

    Google Scholar 

  12. Kramer CY. Extension of the multiple range test to group means with unequal numbers of replications. Biometrics. 1956;12:307–310.

    Article  Google Scholar 

  13. Grechanovsky E, Hochberg Y. Closed procedures are better and often admit a shortcut. J Stat Plan Infer. 1999;76:79–91.

    Article  Google Scholar 

  14. Shaffer JP. Modified sequentially rejective multiple test procedures. J Am Stat Assoc. 1986;81:826– 831.

    Article  Google Scholar 

  15. Hornmel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika. 1988;75:383–386.

    Article  Google Scholar 

  16. Marcus R, Peritz E, Gabriel KR. On closed testing procedures with special reference to ordered analysis of variance. Biometrika. 1976;63:655– 660.

    Article  Google Scholar 

  17. Wright SP. Adjusted P-values for simultaneous inference. Biometrics. 1992;48:1005–1013.

    Article  Google Scholar 

  18. Donoghue JR. Implementing Shaffer’s multiple comparison procedure for a large number of groups. Recent Developments in Multiple Comparison Procedures. Institute of Mathematical Studies, Lecture Notes—Monograph Series. 2004;47:1– 23.

    Google Scholar 

  19. Holland BS, Copenhaver MDP. An improved sequentially rejective Bonferroni test procedure. Biometrics, 1987;43:417–423.

    Article  Google Scholar 

  20. Westfall PH. Multiple testing of general constraints using logical constraints and correlations. J Am Stat Assoc. 1997;92:299–306.

    Article  Google Scholar 

  21. Sloane NJA. The On-Line Encyclopedia of Integer Sequences. 2007. www.research.att.com/~njas/sequences/.

  22. Bittman RM, Romano JP, Vallarino C, Wolf M. Optimal testing of multiple hypotheses with common effect direction. Biometrika. 2009;96:399– 410

    Article  Google Scholar 

  23. The SCORE Study. Manual of Policies and Procedures. Version 4.0. Bethesda, MD: National Eye Institute; 2008. National Technical Information Service: Product Code #PB2008–106870.

Download references

Author information

Authors and Affiliations

  1. The EMMES Corporation, MD, 401 N. Washington St., Suite 700, 20850, Rockville, USA

    Neal Oden PhD & Paul C. Van Veldhuisen PhD

  2. Departments of Ophthalmology and Public Health Sciences, Penn State College of Medicine, Pennsylvania, Hershey, USA

    Ingrid U. Scott MD, MPH

  3. Fundus Photograph Reading Center, University of Wisconsin, Wisconsin, Madison, USA

    Michael S. Ip MD

Authors
  1. Neal Oden PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul C. Van Veldhuisen PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ingrid U. Scott MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael S. Ip MD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Neal Oden PhD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Oden, N., Van Veldhuisen, P.C., Scott, I.U. et al. SCORE Study Report 8: Closed Tests for All Pairwise Comparisons of Means. Ther Innov Regul Sci 44, 405–420 (2010). https://doi.org/10.1177/009286151004400405

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1177/009286151004400405

Key Words

Search

Navigation