1. Introduction

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in the sports science and medicine literature. MBI has laudable goals. The method arose out of a desire to address over-reliance on tests of statistical significance, inattention to estimation, and distortion from unpublished null results [1]. MBI was also developed specifically for use in sports science research, where it may be difficult to get large enough sample sizes to achieve traditional statistical benchmarks such as a power of 80% at an alpha of 0.05. It is indeed useful for researchers to recognize that other choices may be justifiable, and we appreciate that MBI may have helped researchers to move beyond a narrow focus on p < .05.

However, the development and dissemination of MBI have been unusual for a statistical method. MBI has never been published in a journal following peer review by statisticians, and equations with formal notation have never been provided. In fact, since its introduction into the peer-reviewed literature in 2006 [1], the method has been criticized by statisticians [2–8]. Specifically, critics have noted that MBI lacks a clear mathematical foundation, conflates frequentist and Bayesian statistical methods; provides inappropriate sample size calculators that may underestimate sample size needs; and provides inadequate control of Type I error rates [2–8].

These critiques have led some sports science journals to caution against the use of MBI or stop accepting MBI papers altogether [9,10]; and has also led to the method receiving unfavorable media attention [11–13]. Yet, MBI continues to be defended [14–17] and used (e.g., as of May 7, 2020, a Google Scholar search returned 350 hits for “magnitude based inferences” since January 1^st, 2019), thus necessitating further scrutiny of the method. We note that in late 2019, magnitude-based inference was renamed to magnitude-based decisions (MBD) [18]. We use the method’s original name because the studies we examined were published before the name change occurred; and we find that most published studies continue to use the term MBI (e.g., as of May 7, 2020, a Google Scholar search returned 55 hits for “magnitude based inferences” and only 17 hits for “magnitude based decisions” since January 1^st, 2020).

To date, reviews of the method have focused on theoretical issues [2–8] and have not systematically evaluated how the method is being used in practice. This is a particularly important gap as the Type I error rates associated with MBI are highly dependent on how the method is applied [5]. Evaluating the practical application of MBI is also important in addressing proponents’ claims about MBI [14–17], including that it represents an advancement over standard null hypothesis significance testing and that it has better Type I and Type II error rates than null hypothesis significance testing. Finally, an empirical investigation of MBI may have broader lessons for the statistical community. Many commentators have suggested that moving applied researchers away from significance testing and p-values will help them to draw more rational conclusions from their data. MBI provides a natural experiment to test this hypothesis.

To evaluate MBI empirically, we undertook a systematic review of 232 sports science and medicine studies that used MBI. We aimed to document the typical use case of MBI, including: (1) study design; (2) sample size; (3) the number of dependent variables and statistical tests related to the study’s main hypotheses; and (4) the choice of MBI-specific settings and parameters. We used these data to estimate the Type I error rates associated with the typical MBI study. We also made qualitative observations about the statistical content and conclusions of these studies.

2. Methods

2.3. Study selection

The resulting 491 results were combined with 84 results identified in a previous search [6], see the PRISMA flow diagram [20] in Fig 1. Articles were screened by title and abstract, and then full-text by KRL and a student assistant. Studies were included if they reported original empirical data (i.e., no meta-analyses, review articles) on human subjects. A total of 232 articles were included in the review (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flowchart.

PRISMA flowchart showing the screening of articles through the systematic review process.

https://doi.org/10.1371/journal.pone.0235318.g001

2.6. MBI description, settings, and parameters

MBI is implemented in downloadable Excel spreadsheets available at the website sportsci.org [18]. Though a detailed mathematical description has never been provided by the method’s developers, the statisticians Alan Welsh and Emma Knight published a statistical critique of MBI in 2015 in which they reverse engineered the formulas from the spreadsheets for the problem of comparing two means [3]. MBI involves several settings and parameters (defined below). Defaults for the parameters are provided in the spreadsheets.

MBI practitioners define a trivial range of effects that they would consider clinically irrelevant. The thresholds for harm and benefit—which we will denote −δ_h and δ_b, respectively—define this trivial range. The default setting for the trivial range is −δ_h = −0.20 to δ_b = +0.20 standard deviations [18].

MBI returns probabilities that an effect is negative (or harmful), trivial, or positive (or beneficial). These probabilities are based on interpreting p-values from two one-sided hypothesis tests as if they were Bayesian posterior probabilities [3,8]. For example, if p = .25 for the null hypothesis that the intervention is not beneficial (H₀: true effect ≤ δ_b), MBI would conclude that there is a 25% chance that the intervention is not beneficial and thus a 75% chance that the intervention is beneficial. MBI practitioners choose parameters called the maximum risk of harm and minimum chance of benefit—which we will denote η_1h and η_1b. An effect is deemed “unclear” when the harm and benefit probabilities exceed η_1h and η_1b, respectively. Otherwise, an effect is deemed “clear.”

When an effect is deemed clear, MBI returns a categorical inference such as “possibly positive,” “likely positive,” or “very likely positive.” These categories are determined by the MBI probabilities: ≤0.5% is most unlikely; 0.5–5% is very unlikely; 5–25% is unlikely; 25–75% is possible; 75–95% is likely; 95–99.5% is very likely; ≥99.5% is most likely or almost certainly [1,18,21]. For example, a beneficial/positive probability of 75% to 95% corresponds to “likely positive.” This is equivalent to rejecting the null hypothesis that the effect is not beneficial/positive (H₀: true effect ≤ δ_b) at a significance level of .25 but not .05. A “possibly positive” result is equivalent to rejecting H₀: true effect ≤ δ_b at a significance level of .75 but not .25.

MBI is divided into two types of inferences—“clinical” and “non-clinical/mechanistic”; correspondingly, the MBI spreadsheets return two sets of results, one for clinical MBI and one for non-clinical MBI. There are two main differences: (1) clinical MBI allows different values for η_1b and η_1h, whereas non-clinical MBI treats both directions equivalently, such that η_1b = η_1h, and (2) clinical and non-clinical MBI have different default values—0.5% and 5%, respectively—for η_1h (the “maximum risk of harm”). The choice of when to use clinical MBI versus non-clinical MBI is not clearly defined and a source of confusion in MBI [3]. As we will show later, the vast majority of MBI practitioners report the non-clinical/mechanistic MBI results even when reporting about interventions that affect human health or performance. We note that the MBI spreadsheets also return a third set of results called the “odds ratio interpretation” but this was only reported in nine papers that we reviewed, and the authors indicated that this option gave similar results to standard clinical MBI.

Fig 2 gives several examples of how inferences are called in non-clinical MBI. In non-clinical MBI, using the default values of η_1h = η_1b = 5%, an effect is deemed “unclear” if the harm and benefit probabilities both exceed 5%; this is equivalent to the 90% confidence interval for an effect overlapping both the positive (or beneficial) and negative (or harmful) ranges (Fig 2). The default η_1h for clinical MBI (0.5%) is much more stringent than for non-clinical MBI (5%), which means that it is harder to find “clear” positive effects with clinical MBI than with non-clinical MBI (at default values). When an observed effect is in the positive direction, it will be declared “unclear” in clinical MBI but “clear” in non-clinical MBI if the 99% confidence interval overlaps the negative range but the 90% confidence interval does not. We speculate that this is why most users report the non-clinical MBI results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example MBI inferences.

Ten hypothetical results and corresponding MBI inferences, assuming: a trivial range of -0.2 to 0.2 standard deviations, maximum risk of harm of 5%, and equivalent treatment of positive and negative directions (non-clinical MBI). MBI inferences correspond to the locations of the 50% and 90% confidence intervals relative to the negative (or harmful), positive (or beneficial), and trivial ranges. The result is deemed “unclear” if the 90% confidence interval spans the trivial range. Of note, minimal effect testing with α = 0.05, two-sided, would not arrive at conclusions of negative or positive for any of the examples shown. Equivalence testing with α = 0.05 would also fail to conclude equivalent (i.e., trivial difference) for any of the examples shown.

https://doi.org/10.1371/journal.pone.0235318.g002

3. Results

Our systematic review identified 232 studies that used MBI. The topics covered in these articles spanned the range of sport and exercise science and were published in highly cited journals in these fields, see Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The top five most frequent venues for MBI publications identified in our review.

https://doi.org/10.1371/journal.pone.0235318.t001

The studies covered a range of experimental designs, including randomized controlled trials, crossover trials, and observational studies, see Table 2. Sample sizes were extremely small: The median sample size was 10 per group (interquartile range: 8 to 15 per group) for multi-group studies and 14 (interquartile range: 10 to 24) for single-group studies. Only 15% of studies reported any a priori power analysis. In cases where authors reported sample size calculations based on the MBI calculators, they often reported very low sample size requirements, such as 5, 6, or 7 total participants [22–24]. We also note that many authors seemed to erroneously believe that use of MBI circumvents the need for an adequate sample size [25–29]. For example, Stanley et al. [25] wrote of MBI: “With this statistical approach, our sample size is not limiting.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics of the 232 articles identified in the systematic review, median [IQR] or N (%).

MBI settings were not discernible from all studies, as indicated.

https://doi.org/10.1371/journal.pone.0235318.t002

Focusing solely on analyses related to the main hypotheses (excluding baseline and other ancillary comparisons), the median number of dependent variables was 7 and the median number of statistical tests run was 30. Despite this multiplicity of outcomes, few studies (14%) defined one or more variables as “primary.”

The vast majority of studies used non-clinical MBI with the same values: trivial thresholds between -0.20 and +0.20 SD with η_1h = 5%. Authors explicitly noted the use of clinical MBI (3.4%), non-clinical MBI (16%) or both (1.3%) in a minority of papers. Given descriptions in the texts, we identified five additional papers where we believe clinical MBI was used, for a total of 16 papers (7%) that used clinical MBI. We believe that non-clinical MBI was used in almost all other cases (164 additional papers). In these cases, authors wrote a version of the following statements: “When the positive and negative values were both >5%, the inference was classified as unclear” or, equivalently, “If the 90% confidence interval overlapped the thresholds for the smallest worthwhile positive and negative effects, effects were classified as unclear”; this means they used η_1h = 5% and treated both directions equivalently, consistent with non-clinical MBI. We cannot rule out that authors used clinical MBI with both η_1h = 5% and η_1b = 5%, but this is less likely given that these are not the usual default choices for clinical MBI.

MBI provides categories of evidence, such as “possible”, “likely”, and “very likely” beneficial. We found that researchers employed MBI’s evidence thresholds much like significance thresholds: They highlighted results that met the bars of “possible”, “likely”, or higher in tables and figures with symbols to distinguish the achieved thresholds much like the use of symbols for p<0.05 or p<0.01; for examples, see: [30–32]. They then made claims about effects based on meeting a minimum MBI evidence threshold, usually either “possible” or “likely.” For example, Cruz et al. [33] explicitly define this bar, declaring: “A likely difference (>75%) was considered as the minimum threshold to detect meaningful differences,” as do Weaving et al. [34]: “The magnitude of difference was considered practically meaningful when the likelihood was ≥ 75%.” Others implicitly set a minimum threshold by only highlighting and drawing conclusions based on effects above a certain bar. For example, Lahart et al. [24] write in their abstract: “Magnitude-based inference analyses revealed likely at least small beneficial effects (effect sizes≥.20) on absolute and relative VO₂max (d = .44 and .40, respectively), and total and moderate PA (d = .73 and .59, respectively) in the intervention compared to the usual care group. We found no likely beneficial improvements in any other outcome” (emphasis added). One can infer from their description and ensuing positive recommendation (“This intervention has the potential for widespread implementation and adoption, which could considerably impact on post-treatment recovery in this population.”) that they used “likely” as their minimum evidence threshold, though this was never explicitly stated.

We found that most authors used a minimum evidence threshold of “possible” (38%) or “likely” (43%) (Table 2). We found no cases where authors either explicitly or implicitly set this bar higher than “likely” (Table 2). In fact, authors appeared to view the “likely” threshold as a high level of evidence. For example, Barrett et al. [35] state: “We adopted a conservative approach to inference whereby substantial effects were only declared clear when the probability likelihood for the effect was ≥75% (i.e., likely)” (emphasis added).

Almost half (47%) of studies performed null hypothesis testing alongside MBI. Most authors reported using the Excel spreadsheets to run their MBI analyses, but a few authors reported running frequentist statistical models in other programs, such as SAS, but then interpreted the resulting frequentist confidence intervals in MBI terms. We found no studies out of 232 that described their statistical approach as Bayesian.

S1 Appendix documents additional concerns about bias in the design and reporting of these studies. However, we note that these concerns are common in sports science studies and are not unique to studies using MBI.

In our review, we found that most MBI studies employed a narrow range of sample sizes, used non-clinical MBI, and used the same default MBI settings: thresholds for harm/benefit of 0.2 and maximum risk of harm of 0.05. Thus, we were able to estimate the Type I error rates that apply to the vast majority of the MBI literature. We estimated the Type I error rates associated with the “possible” and “likely” evidence thresholds since, in the studies we reviewed, authors consistently highlighted and made claims based on “possible” or “likely” positive/beneficial or negative/harmful inferences. Note that our estimates of Type I error would also apply to clinical MBI with η_1b = η_1h = 5%. In fact, clinical MBI with η_1b = η_1h = 5% (rather than η_1b = 25% and η_1h = 0.5%) was given as the default choice in at least one version of the MBI spreadsheets that we found online in August of 2019 (see S3 Appendix; [18]).

Fig 3A and 3B show the Type I error rates as a function of sample size for a two-group comparison of means when the outcome variance is smaller (3A) or larger (3B); patterns are similar for other statistical comparisons (see S1 Appendix). MBI creates pockets of high Type I error that—particularly for the “likely” threshold—coincide with the sample sizes that predominate in the MBI literature. Fig 3C and 3D demonstrate how easy it is for spurious “likely” effects to arise in small samples due to high chance fluctuation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. MBI’s Type I error rates.

A and B: Type I error rates for MBI’s “possible” (purple) and “likely” (red) thresholds, as well as standard hypothesis testing at α = 0.05 (blue) as a function of sample size. The statistical comparison is a two-group comparison of means. True effect size = 0, meaning there is no difference between the groups. (A) assumes variance of 0.364, as might arise in a pre-post study, whereas (B) assumes a variance of 1.0, as in a cross-sectional study. Shaded area shows the interquartile range of sample sizes of the reviewed studies; vertical reference line is the median sample size. Type I error rates were identical whether calculated mathematically or by simulation with 200,000 repeats (see S1 Appendix). C: MBI results from 5000 simulated trials where variance = 0.364 and n = 10 per group. D: MBI results from 5000 simulated trials where variance = 1.0 and n = 10 per group. Simulations and calculations use the MBI settings that predominate in the literature: trivial range of -0.2 to 0.2; maximum risk of harm of 5%; and equivalent treatment of positive and negative directions.

https://doi.org/10.1371/journal.pone.0235318.g003

For multi-group studies, the interquartile range of sample size was 8 to 15 per group. At these sample sizes, the Type I error rate for the “possible” threshold is 22% to 45% and for the “likely” threshold is 12% to 22% (Fig 3A and 3B). Consistent with these calculations, we noticed numerous studies in which effects associated with large p-values (for H_o = 0)—often in the 0.2 to 0.3 range—were declared “likely” or higher by MBI [36–52]. See Fig 4 for an example from the literature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. An example of MBI inferences in practice.

Left Panel (Reproduced from Parfey et al. [50], Fig 2C): Literature example where effects deemed “likely” by MBI are associated with large p-values. Confidence intervals are 95% CIs. Starred values are effects meeting MBI’s “likely” threshold. These results were interpreted as evidence of a difference between groups; the authors concluded: “Individuals with CLBP and PR manifested altered activation patterns during the hollowing maneuver compared to healthy controls.” Right panel: Simulation that shows the MBI inferences that are expected for a study of this type (n = 10 per group, cross-sectional) when the true effect is 0. Note that in both the real example and the simulation, most observed effects larger than 0.5 are deemed “likely”.

https://doi.org/10.1371/journal.pone.0235318.g004

In some examples, high p-values were reported alongside the MBI results. When p-values were reported alongside MBI results, we found that MBI descriptors were frequently prioritized above p-values in the interpretation of results. Indeed, the results were often used to justify the superiority of MBI over conventional approaches. Examples include:

“Although parametric analysis was unable to demonstrate significant differences, magnitude-based inferences indicated that the change in 1-RM squat showed a likely benefit from PA on increasing lower body strength and a very likely benefit for increasing lean body mass (LBM).”

(Hoffman et al. [43])

“Because a repeated measures ANOVA could not detect any significant difference in the variables between the fastest and slowest trials (P = .136 –>.999), magnitude-based inference approach was adopted to analyse the differences at each step.”

(Nagahara et al. [44])

“There were no statistically significant differences between the non-runners and runners groups in the CRP values for any of the coupling pairs analysed, however, the magnitude-based inferences enabled more precise tracking of differences between groups and some of the differences were substantially clear.”

(Lockie et al. [45])

“There were no statistical differences in NB between groups (P = 0.23); however, magnitude-based inferential statistics revealed likely small (mean effect±90% CI; 0.59±0.87) and moderate (0.80 ± 0.91) increases in NB for PULSE and INT compared to BOLUS and possible small increase (0.42 ± 1.00) for INT vs. PULSE. Conclusion: We conclude that the pattern of ingested protein, and not only the total daily amount, can impact whole-body protein metabolism.”

(Moore et al. [46])

In these instances, and others, the authors emphasized the more favorable MBI results in their conclusions, ignoring that the observed effects could easily have arisen due to sampling variability.

In many other examples, authors did not report p-values, but we back-calculated high p-values from confidence intervals [52]. For example, one study reported an improvement of 0.46 standard deviations in countermovement jumps with a 90% confidence interval of -0.10 to 0.91 standard deviations [52]; we calculated the two-sided p-value associated with H_o = 0 as: P(T₂₁>(.46/(.5/1.72)))+P(T₂₁<-(.46/(.5/1.72))) = .13. Notably, this paper reported that the intervention “almost certainly” improves countermovement jumps with a 100% probability of benefit [52].

Until now, our discussion has focused on a single test with default MBI parameters. Table 3 shows how changing the study design, variance, and MBI parameters affects the Type I error rates. We found that across most realistic scenarios, researchers have between a 20% to 52% chance of getting a “possible” or higher positive or negative inference when the true effect is zero or trivial. Increasing sample size or decreasing variance (relative to the base case) results in lower Type I error rates for the “likely” threshold, but higher rates for the “possible” threshold. Changing η_1h from 5.0% to 0.5% lowers the Type I error rates for these small sample sizes, but as previously noted, researchers rarely chose this value (n = 19, 8% of studies).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Type I error rates for MBI vary as a function of sample size, statistical comparison, variance, maximum risk of harm, and thresholds for harm/benefit.

Row 1 represents the typical MBI study; subsequent rows demonstrate how changing specific parameters alters the rates; parameters that remain unchanged from the base case are grayed out whereas altered parameters are bolded. Rates were calculated mathematically and also confirmed by simulation with 200,000 repeats (See S1 Appendix for description and S2 Appendix for code).

https://doi.org/10.1371/journal.pone.0235318.t003

MBI’s high Type I error rates are further compounded by multiple testing (Table 3). Though multiple testing is also an issue for standard hypothesis testing, the higher per-test Type I error rates of MBI serve to exacerbate the issue. The studies we reviewed reported a median of 7 dependent variables and 30 statistical tests related to the main hypothesis. With so many tests, users are virtually guaranteed to find some “possible” or higher effects by random chance when the intervention is ineffective (Table 3). MBI provides few tools for addressing multiple testing; in fact, the proponents of MBI have advised against controlling for multiple tests [21]. MBI also provides no means for handling repeated-measures data; thus, in papers with more than two time points, authors invariably presented MBI analyses for each time point separately, creating an obvious multiple testing issue. For example, in a parallel-group randomized trial, Townsend and colleagues [53] measured muscle biomarkers in an exercise and control group at 3 follow-up time points: 1 hour, 5 hours, and 48 hours. Using repeated-measures ANOVA, they found no significant time-by-group interactions for any outcome at the 0.05 level; however, the MBI analysis found two positive effects out of 15 tested (5 outcomes by 3 timepoints): a “likely” increase for exercise versus control in IkBα phosphorylation at 5 hours and a “very likely” increase for exercise versus control in total c-Myc content at 5 hours. They then concluded: “The main findings of this investigation indicated that phosphorylation of IkBα and total c-Myc content was increased in skeletal muscle following resistance exercise in novice lifters.”

4. Discussion

Through our systematic review, we have documented that studies using MBI are typically small in size and often make strong claims based on weak evidence. MBI authors routinely made claims based on achieving MBI inferences of “likely” and sometimes “possible” effects. These inferences carry high Type I error rates—12%-22% and 22%-45%, respectively, for the way that MBI is typically applied in the literature. MBI users predominantly applied MBI’s default settings, used a maximum risk of harm of 5%, and chose “non-clinical” rather than “clinical” MBI even when testing interventions in humans. This is likely related to “non-clinical” MBI providing stronger inferences for a given set of data as well as to a lack of precise guidelines about when to use non-clinical versus clinical MBI. Few studies reported a priori power calculations (15%) or specified a primary outcome (14%). Multiple testing was common with a median of 7 dependent variables and 30 tests related to the main hypotheses of interest; this exacerbates the Type I error issue.

The median sample size was 10 per group for multi-group studies and 14 total for single-group studies. As a benchmark, it is worth noting that a sample size of 10 per group has a mere 40% power to detect an effect size of 0.8, which is commonly described as a large effect (given α = 5%). Inadequate sample sizes and poor sample size planning are certainly not unique to MBI. However, we believe that the way that MBI has been advertised to the sports science community has encouraged the use of small studies. A central justification for MBI has been its ability to generate more “publication-worthy” results from under-powered studies. For example, proponents of the method state that MBI, “provides an avenue for publishing previously unpublishable effects from small samples” [14] and that, “With small studies, you get nonsignificant effects a lot, but these nonsignificant effects are often clear so the researcher can often make useful publishable conclusions for such a thing with MBI. You can get your stuff into press” [54]. Thus, it’s not surprising that MBI users predominantly employed small samples; rarely performed a priori sample size calculations, and even made statements suggesting that MBI circumvents the need for an adequate sample size [25–29].

Overinterpretation of statistical results is also not unique to MBI, but features of the method and its portrayal may exacerbate the problem. MBI has been described as a Bayesian procedure [14] and “definitely not” a form of hypothesis testing [15]. MBI’s developers state: “…we wish to avoid hypothesis tests of any kind, involving either the nil (zero) hypotheses or non-zero hypotheses, as in equivalence testing” [55]. Thus, it’s not surprising that users appear unaware that, underneath the hood, MBI is in fact running non-zero hypothesis tests (with liberal significance levels). MBI also misrepresents p-values as Bayesian posterior probabilities and affixes overly optimistic descriptors such as “likely” or “almost certainly” beneficial. Correspondingly, users routinely attribute higher levels of evidence to MBI inferences than the data warrant.

MBI’s proponents have repeatedly denied that MBI has elevated Type I error rates [14–17]. We have shown this to be false. MBI does not necessarily carry a high Type I error rate, because MBI’s error rates depend on the choice of MBI parameters and settings as well as study design and sample size. However, we found that MBI is predominantly applied in a way that results in high Type I error. For example, use of a maximum risk of harm of 0.5% keeps Type I error rates low for samples of 8 to 15 per groups, but only 8% of users chose this value. Despite conducting clinical research in human populations, authors primarily used non-clinical MBI rather than clinical MBI, likely because it has a default maximum risk of harm of 5% and thus makes it easier to find “clear” beneficial/positive effects.

There has been disagreement about what constitutes a Type I error in MBI. The developers of MBI sometimes refer to “clinical Type I error,” which they define as claiming benefit when the true effect is harmful; however, this ignores the larger source of Type I error, which is claiming benefit when the true effect is trivial [5]. The developers of MBI have also claimed that clinical MBI incurs a Type I error for a “possibly” or “likely” inference but that non-clinical MBI does not [17]. This would only be true if non-clinical MBI required a minimum actionable evidence threshold of “very likely.” It does not. Users routinely made claims about clinical interventions based on “possibly” and “likely” inferences regardless of the choice of clinical or non-clinical MBI. In fact, only 21% of users even specified whether they were using non-clinical or clinical MBI, suggesting that this distinction holds little meaning in practice.

We acknowledge that the appropriate tradeoff between Type I and Type II error may be different in sports science than in other biomedical fields, such as cancer research. Thus, there may be cases where a Type I error rate higher than 0.05 may be justified. However, we emphasize that: (1) 0.05 is itself considered a weak level of evidence [56] (2) 0.05 has undoubtedly been given undue importance in the medical literature, but this problem is not solved by setting a more liberal default threshold; (3) the Type I error rates that we have documented for MBI would be considered unusually weak levels of evidence in biomedicine. As a benchmark, p = .25 corresponds to a Shannon-information, or S-value, of 2, meaning the data provide just 2 bits of information against the null hypothesis of no effect—no more surprising than getting two heads in two coin tosses [57]; (4) high rates of multiple testing in sports science compound the issue; and (5) if researchers are going to allow a higher Type I error rate, they must be transparent about this choice and justify it a priori. In the MBI literature, we found no cases where MBI practitioners acknowledged higher Type I error rates, let alone justified them.

We found that the MBI spreadsheets encourage a black box approach to statistics [8]. For example, users predominantly employed the default thresholds for harm/benefit of 0.2 standard deviations presented in the MBI spreadsheets. This suggests that this choice was not based on any biological or clinical rationale and was likely not specified a priori. Of course, MBI is not unique in encouraging a black box approach to statistics. Many statistical programs allow users to implement statistical methods that they do not understand. However, MBI’s lack of rigorous review and documentation may exacerbate the problem [8].

The MBI spreadsheets are also not designed to handle many common statistical situations: repeated-measures or clustered data, interactions, count data, and adjustment for more than a single covariate [8]. The inability to appropriately handle repeated-measures data is particularly problematic given that repeated-measures were common in MBI studies. For instance, in a study on foam rolling [30], group differences were assessed at three different time points for 13 outcomes measures. With a more conventional ANOVA, these post-hoc tests might only be conducted following a statistically significant Group x Time interaction, or not at all. Multiple testing is also a problem for null hypothesis significance testing, but MBI’s higher per-test Type I error rates and inability to handle repeated measures compound the problem.

We acknowledge that Type I error rates for null hypothesis significance testing can also be greatly inflated due to “p-hacking,” the practice of manipulating data and analyses to produce p-values under 0.05 [58]. Though MBI does allow other researcher degrees of freedom (e.g., choice of MBI parameters and settings), MBI may be less prone to p-hacking because it does not have the same narrow focus on p < .05. Indeed, a positive outcome of MBI is that it has helped moved sports scientists away from overreliance on p < .05.

Proponents of MBI argue that small studies are unavoidable in sports science and thus that MBI is needed to ensure a path to publication for otherwise “unpublishable” null results [14]. We disagree that small studies are inevitable, and believe that we should encourage researchers to engage in sample size planning and to design better [59] and larger studies [60,61]. We also encourage the adoption of registered reports, which prevent reliance on statistical significance as a criterion for publication [62,63]. In the absence of preregistration, when researchers do run severely under-powered studies (e.g., due to logistical constraints), we do not believe that the appropriate path to publication is choosing an inferential method that misrepresents weak evidence as strong evidence. Rather, researchers can publish the results as a pilot or feasibility study [64] or report the results descriptively, avoiding inferential language [65]. Small studies—provided that are properly reported, without overstating evidence—can be useful for the design of larger trials as well as in meta-analyses. Additionally, we believe that many sports science researchers may be interested in making decisions about individual athletes rather than making inferences to wider populations, in which case the most appropriate study design may be an N-of-1 trial [66]. If practitioners are interested in seeing if an intervention is reliably beneficial for a specific athlete, it is more meaningful and more valid to adopt an N-of-1 trial than generalize from an underpowered study using a low standard of evidence for an effect. Finally, we note that when studying elite athletes, it’s possible that a relatively small sample will give adequate power due to the low variability in the population.

Proponents of MBI have additionally argued that MBI is necessary because it provides the only alternative to null hypothesis significance testing, for example writing: “Generations of statisticians before us have also criticized the null-hypothesis test, but magnitude-based inference appears to be the first practical alternative that properly takes into account the uncertainty arising from sampling variation” [15]. This argument is false; scientists are not limited to these two approaches and MBI is certainly not the first alternative to a monolithic statistical approach. Multiple other statistically sound alternatives are available. For example, MBI uses the same underlying hypothesis tests as minimal effect testing and equivalence testing/non-inferiority testing [67]. Thus, MBI practitioners could use these procedures instead [68]. Furthermore, sports scientists should understand that—when used correctly—p-values are valid tools of inference [69]. For example, Amrhein and colleague’s [70] recent headline-making call for the abandonment of significance testing did not advocate a ban on p-values, but instead encouraged scientists to use p-values correctly—by recognizing them as a continuous (rather than dichotomous) measure and as just one piece of evidence among many. Finally, many of the stated goals of MBI—such as giving a plain language interpretation of statistics that is understandable by coaches and athletes—can be validly achieved with a Bayesian analysis [4] or by careful incorporation of p-values with data on effect sizes, as well as consideration of strengths and weaknesses of the experiment and evaluation of the current literature.

The conclusions of past studies that have used MBI should be treated with skepticism. However, these papers do still contain valid information—such as means, standard deviations, and confidence intervals—that can be used to re-evaluate the studies’ conclusions. For example, one study reporting that an intervention “almost certainly” improved countermovement jumps also reported a 90% confidence interval for this improvement of -0.10 to 0.91 standard deviations [52]. It is clear from the confidence interval that the evidence of improvement is much weaker than the study’s conclusions imply. However, the standard error is calculable allowing the data from this study to be entered into a meta-analysis.

Furthermore, some MBI papers report the exact harm and benefit probabilities given by the MBI spreadsheets, which are useful because they reveal the p-values associated with specific one-sided hypothesis tests. The harm/negative probability is the p-value associated with the null hypothesis of harm (H_o: effect size ≤-δ_h); and subtracting this value from 1 gives the p-value associated with the null hypothesis of no harm (H_o: effect size ≥-δ_h). Similarly, the benefit/positive probability is the p-value associated with the null hypothesis of benefit (H_o: effect size ≥δ_b); and subtracting this value from 1 gives the p-value associated with the null hypothesis of no benefit (H_o: effect size ≤δ_b). One may use these p-values to perform equivalence testing and/or minimal effects testing [67,68]. We give a specific example of how this would be done in Table 4 using a study on foam rolling [30]. Our re-examination of these data suggest that the study’s conclusions were overly optimistic. Notably, this paper has been cited 258 times in Google Scholar (as of 2020-01-05); the first 10 citing papers that we checked all cited MacDonald and colleagues [30] as providing evidence of the benefits of foam rolling, demonstrating how overly optimistic conclusions are easily propagated through the literature and how correcting such errors is difficult once they are published.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Example of how MBI results can be re-interpreted by one-sided minimal effects testing and noninferiority testing with α = .05^a.

https://doi.org/10.1371/journal.pone.0235318.t004

In cases where papers do not report the exact MBI harm and benefit probabilities, it is possible to back-calculate the relevant p-values from confidence intervals. We include SAS and R code for doing this, specifically for between-group comparisons, in S2 Appendix. For ease of use, we have also created an online results converter (https://rehabinformatics.shinyapps.io/MBI_Converter/).

As a case-study, MBI has larger lessons for the statistical community. Some academics have suggested that moving researchers away from p-values and/or significance testing will improve applied statistical practice. MBI provides a natural experiment for testing this hypothesis. We found that MBI papers were no less prone to dichotomization or over-interpretation of findings; in fact, use of MBI generally led to greater exaggeration of results than standard significance testing. Other empirical investigations have drawn similar conclusions: A review of 31 papers published in Basic and Applied Social Psychology after the journal banned p-values found that many authors over-interpreted their results far beyond what the data supported [71]. Misuse of frequentist tools therefore appears to be a symptom of a deeper problem rather than the cause of bad statistics. To improve statistical practice in applied disciplines, we will have to address more fundamental problems such as poor statistical knowledge among applied researchers; an unsophisticated view of statistics as a black box set of recipes that provides simple answers; and endogenous and exogenous incentives to report “positive” findings.

5. Conclusions

We have conducted a systematic review of the MBI literature and undertaken mathematical analyses of MBI methods. We found that MBI has done direct harm to the sports science and medicine literature by causing authors to draw overly optimistic conclusions from their data. MBI has also promoted small studies, promulgated a simplistic “black box” approach to statistics, and reinforced the harmful view that the purpose of research is to get publishable results for the researcher. Sports scientists should stop using MBI. Past empirical studies that have used MBI may well include useful descriptive statistics, but their conclusions should be treated with skepticism. As a case-study, MBI has larger lessons for the statistical community: Our findings suggest that moving researchers away from significance testing and p-values does not improve applied statistical practice.

Supporting information

Acknowledgments

Authorship order for JAT, MLB, and EJK was determined by a random number generator in R. We would also like to thank Regina Johanns and Hayden Lee for their assistance with the systematic review.