Comment on “A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods”

Alexander P. Keil,1 Jessie P. Buckley,2,3 Katie M. O’Brien,4 Kelly K. Ferguson,4 and Alexandra J. White4 Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health (NIH), Department of Health and Human Services (DHHS), Bethesda, Maryland, USA Department of Environmental Health and Engineering, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA Epidemiology Branch, National Institute of Environmental Health Sciences, NIH, DHHS, Research Triangle Park, North Carolina, USA

Recently, Day et al. 1 compared weighted quantile sum regression (WQSr) with quantile-based g-computation (QGC) using simulations and a worked example. 2 They wrote that "mixture componentspecific coefficients estimated by [QGC] were far more biased than those of any of the WQSr models." 1 Their results do not support this claim because bias was not assessed in isolation.
Instead, Day et al. assessed mean absolute percent error (MAPE), a measure of accuracy that combines bias and variability. To assess bias, we repeated one of their simulations (correlated mixture, b 1 = 0:2) and examined mean percent bias (MPB: average bias/truth) of the component-specific coefficients. MPB of their weighted quantile sum regression bootstrap sample permutation test (WQSBSPT) approach was 2.5-8 times higher than that for QGC (Table 1). MPB for the "mixture effect" was 80 times higher for WQSBSPT than QGC. Thus, the results support a countermanding claim: QGC was far less biased than WQSBSPT. We consequently disagree that their results suggest "caution when interpreting [QGC] coefficients." 1 Day et al.'s simulations assume unidirectional causal exposures, an ideal setting for maximizing WQSr accuracy. A reanalysis using our previously published simulation 2 with 1 causal exposure and 13 noise exposures yielded a 3-fold better component coefficient MAPE for QGC than WQSBS-Split (12% and 36%, respectively). Similar to Day et al.'s simulations, this simulation assumes no counteracting exposures. Thus, accuracy results do not generalize across different plausible scenarios.
We also note a fundamental flaw in some simulations. When WQSr methods failed to return a result, Day et al. imputed b 1 = 0. When analyzing data simulated with b 1 = 0, the authors therefore imputed an estimate with perfect accuracy (no bias, no variability), which exaggerated WQSr performance. Because 86% of Weighted quantile sum regression, random subset, repeated holdout (WQSRS-RH) fits failed to return a result when b 1 = 0, their WQSRS-RH results are not credible. Other WQSr results were also impacted.
Finally, we respond to the authors' coffee/alcohol analogy. The analogy says that the effects of coffee and alcohol would cancel out one another in a QGC regression such that QGC would erroneously report no joint effect of substance use. If the joint effect of coffee and alcohol on a health outcome of interest is truly null, QGC will tell us so 2 and also estimate the independent effects of alcohol and coffee. That is no error; it is crucial to understand when therapeutic agents counteract hazardous agents in a mixture. Alternatively, WQSr will yield biased independent effects of alcohol and coffee and no joint effect. 3 All authors declare they have no actual or potential competing financial interest.
Note to readers with disabilities: EHP strives to ensure that all journal content is accessible to all readers. However, some figures and Supplemental Material published in EHP articles may not conform to 508 standards due to the complexity of the information being presented. If you need assistance accessing journal content, please contact ehpsubmissions@niehs.nih.gov. Our staff will work with you to assess and meet your accessibility needs within 3 working days.