Systematic review
Novel Effect Size Interpretation Guidelines and an Evaluation of Statistical Power in Rehabilitation Research

https://doi.org/10.1016/j.apmr.2020.02.017Get rights and content

Abstract

Objective

First, to establish empirically-based effect size interpretation guidelines for rehabilitation treatment effects. Second, to evaluate statistical power in rehabilitation research.

Data Sources

The Cochrane Database of Systematic Reviews was searched through June 2019.

Study Selection

Meta-analyses included in the Cochrane Database of Systematic Reviews with “rehabilitation” as a keyword and clearly evaluated a rehabilitation intervention.

Data Extraction

We extracted Cohen’s d effect sizes and associated sample sizes for treatment and comparison groups. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. The 25th, 50th, and 75th percentile values within the effect size distribution were used to establish interpretation guidelines for small, medium, and large effects, respectively. A priori power analyses established sample sizes needed to detect the empirically-based values for small, medium, and large effects. Post-hoc power analyses using median sample sizes revealed whether the “typical” rehabilitation study was sufficiently powered to detect the empirically-based values. Post hoc power analyses established the statistical power of each test based on the sample size and reported effect size.

Data Synthesis

We analyzed 3381 effect sizes extracted from 99 meta-analyses. Interpretation guidelines for small effects ranged from 0.08 to 0.15; medium effects ranged from 0.19 to 0.36; and large effects ranged from 0.41 to 0.67. We present sample sizes needed to detect these values based on a priori power analyses. Post hoc power analyses revealed that a “typical” rehabilitation study lacks sufficient power to detect the empirically-based values. Post hoc power analyses using reported sample sizes and effects indicated the studies were underpowered, with median power ranging from 0.14 to 0.23.

Conclusions

This study presented novel and empirically-based interpretation guidelines for small, medium, and large rehabilitation treatment effects. The observed effect size distributions differed across intervention categories, indicating that researchers should use category-specific guidelines. Furthermore, many published rehabilitation studies are underpowered.

Section snippets

Search procedure

We searched the Cochrane Database of Systematic Reviews20 for meta-analyses with rehabilitation as a keyword (k=313 reviews, as of June 2019). We included meta-analyses that (1) summarized the effects of an intervention; (2) reported effect sizes and sample sizes; (3) aligned with rehabilitation (ie, targeted functioning among those with health conditions); and (4) were nonpharmacological and nonsurgical. Three meta-analyses were excluded after screening because they did not pertain to an

Results

The 3381 effect sizes were classified as follows: (1) organ functions, n=1920; (2) skills and habits, n=864; (3) representations, n=327; and (4) multicomponent interventions, n=270. Exercise-based interventions were most common among treatments targeting organ function (n=1154, 60.1%). Cognitive training was the most frequently reported intervention targeting skills and habits (n=283, 32.8%). Didactic education was the most frequently reported treatment targeting representations (n=157, 48.0%),

Discussion

We leveraged 3381 effect sizes from rehabilitation studies to establish novel guidelines for interpreting the magnitude of rehabilitation treatment effects. These guidelines are uniquely relevant to rehabilitation research, and should therefore supplant the use of Cohen’s13,14 original guidelines. Second, this study retrospectively evaluated the statistical power of studies investigating rehabilitation treatment effects. Information from this study will enable rehabilitation investigators to

Conclusion

This study presents novel and empirically based interpretation guidelines for small, medium, and large rehabilitation treatment effects. Cohen’s effect size interpretation guidelines,13,14 which have been widely adopted across disciplines, do not accurately describe rehabilitation treatment effects. Furthermore, observed effect sizes differ across intervention categories, indicating that researchers should use category-specific guidelines. This study also demonstrates that rehabilitation

Suppliers

  • a.

    data.table package; Dowle et al.

  • b.

    R software; The R Foundation for Statistical Computing.

  • c.

    pwr package; Champely et al.

Acknowledgments

We thank Christopher Brydges, PhD for providing consultation regarding analyses. We also thank the reviewers for improving the clarity, methodological rigor, and potential impact of this article.

References (38)

  • K.J. Ottenbacher et al.

    Measures of effect size in the reporting of rehabilitation research

    Am J Phys Med Rehabil

    (1989)
  • L.A. Latif et al.

    Sample size calculation in physical medicine and rehabilitation: a systematic review of reporting, characteristics, and results in randomized controlled trials

    Arch Phys Med Rehabil

    (2011)
  • K.J. Ottenbacher et al.

    Statistical conclusion validity of rehabilitation research. A quantitative analysis

    Am J Phys Med Rehabil

    (1990)
  • K.E. Musselman

    Clinical significance testing in rehabilitation research: what, why, and how?

    Phys Ther Rev

    (2007)
  • P. Page

    Beyond statistical significance: clinical interpretation of rehabilitation research literature

    Int J Sports Phys Ther

    (2014)
  • D. Hevey et al.

    The effect size statistic: useful in health outcomes research?

    J Health Psychol

    (1998)
  • J. Cohen

    Statistical power analysis for the behavioural sciences

    (1988)
  • J. Cohen

    A power primer

    Psychol Bull

    (1992)
  • B. Thompson

    Effect sizes, confidence intervals, and confidence intervals for effect sizes

    Psychol Sch

    (2007)
  • Cited by (53)

    • Responsiveness of the Reaching Performance Scale for Stroke

      2023, Archives of Physical Medicine and Rehabilitation
    View all citing articles on Scopus

    Parts of this work were supported by grant funding from the National Institutes of Health (P2CHD065702). The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of Defense, Department of Veterans Affairs, or the United States government.

    Disclosures: none.

    View full text