Presentation of Study Results: The Authors’ Responsibility

We read with interest the article by Kalkbrenner et al. (2012) in which they explored maternal smoking during pregnancy as a risk factor for autism spectrum disorders (ASD). We believe that the following shortcomings of the study did not allow an evaluation of the results and therefore that the paper provides little evidence to judge whether data suggest a “link.” 
 
The findings of Kalkbrenner et al. (2012) regarding “higher-functioning” ASD include three null associations and one association in the smallest subgroup of 375 cases (ASD-not otherwise specified; ASD-NOS) that was “statistically significant” only in sensitivity analysis. Therefore, we question their interpretation of the data when an effect was suggested in only one of the four tests of the same hypothesis. Furthermore, ASD-NOS is a difficult diagnostic subtype to understand because it includes, as the authors noted, a heterogeneous mixture of diagnoses. 
 
Although socioeconomic status (SES) is a well-known correlate of both smoking and ASD, the authors used only maternal education to control for SES; thus, residual confounding from other aspects of SES is likely (King and Bearman 2011; Rai et al. 2012). 
 
Kalkbrenner et al. (2012) did not appropriately control for confounders, and this affected sensitivity analysis central to their conclusions. In their sensitivity analysis for outcome misclassification, they did not correct for covariates, thus basing all of their interpretations on results that were contaminated by confounding. They could have used Monte Carlo methods (Bodnar et al. 2010) to adjust for confounding while accounting for outcome misclassification, obtaining confidence intervals that account for random simulation error, but they did not do this. Thus, the reported confidence intervals for the sensitivity analyses are likely to be too narrow. 
 
Kalkbrenner et al. (2012) did not quantitatively assess the impact of exposure misclassification. The quoted 0.8 concordance of smoking data on birth certificates with the medical record means that smoking exposures of > 125,000 persons in the sample were expected to be incorrectly classified. Sensitivity of maternal smoking on U.S. birth certificates is likely to be only 0.5 (Kharrazi et al. 1999). Epidemiologists ignore measurement error at great peril (Jurek et al. 2006) while correction procedures exist (MacLehose and Gustafson 2012). 
 
Finally, we would like to point out the difficulties of this article in communicating scientific results to the general public. Because, as Kalkenbrenner stated, “the study doesn’t say for certain that smoking is a risk factor for autism” (UWM News 2012), then it is the author’s responsibility to more carefully report to the media what the study actually does say. It is easy to blame journalists for the sensational findings that have been reported about this study (e.g., Goodwin 2012). However, given the historic legacy of blaming parents, particularly mothers, for their child’s diagnosis, we would better serve the communities for whom we do this research if we developed standard practices for reporting preliminary findings in ASD risk factor research. One suggestion would be to report these findings without discussion in media (e.g., Palmer 2011) and scholarly publications, as was done by Adam et al. (2011), who produced experimental data demonstrating that the speed of light was exceeded: 
 
Despite the large significance of the measurement reported here and the robustness of the analysis [p << 0.00006%], the potentially great impact of the result motivates the continuation of our studies in order to investigate possible still unknown systematic effects that could explain the observed anomaly. We deliberately do not attempt any theoretical or phenomenological interpretation of the results. 
 
We encourage caution when promoting findings of “potentially great impact” on public health that are of a preliminary nature and are not ready to be even interpreted.

A 343 due to disease misclassification, we (Camargo et al. 2011) concluded that our meta-analysis supports the IARC classification. This illustrates again that meta-analyses are not free from subjective decisions and interpretations.
In conclusion, meta-analyses are a quantitative statistical tool that, in some instances may inform causal inference, but they never alleviate the need for critical review of all available data; narrative reviews by an interdisciplinary IARC Working Group may be, in some cases, more informative than a synthetic meta-analysis. Therefore, although a comprehensive review of all original data is required, a comprehensive review of all meta-analysis may not be warranted, particularly when the meta-analyses are outdated or cover only a subset of the original studies. The current "Preamble to the IARC Monographs" (IARC 2006) provides the Working Group with all options to perform quantitative meta-analysis where appropriate and helpful for causal inference. Different approaches have been applied in the history of the IARC Monographs. The Volume 100 series of the IARC Monographs confirmed all Group 1 carcinogens identified during the 40-year history of the monographs, which in turn confirmed that the procedures of the IARC Monographs are robust. With more epidemiological studies becoming available for each agent, additional cancer sites being investigated, and relatively small effect estimates becoming center of the discussion, the need for meta-analyses is likely to increase.

Presentation of Study Results:
The Authors' Responsibility http: //dx.doi.org/10.1289/ehp.1205556 We read with interest the article by Kalkbrenner et al. (2012) in which they explored maternal smoking during pregnancy as a risk factor for autism spectrum dis orders (ASD). We believe that the following short comings of the study did not allow an evaluation of the results and therefore that the paper provides little evidence to judge whether data suggest a "link." The findings of Kalkbrenner et al. (2012) regarding "higher-functioning" ASD include three null associations and one association in the smallest subgroup of 375 cases (ASD-not otherwise specified; ASD-NOS) that was "statistically significant" only in sensitivity analysis. Therefore, we question their interpretation of the data when an effect was suggested in only one of the four tests of the same hypothesis. Furthermore, ASD-NOS is a difficult diagnostic subtype to understand because it includes, as the authors noted, a hetero geneous mixture of diagnoses.
Although socioeconomic status (SES) is a well-known correlate of both smoking and ASD, the authors used only maternal education to control for SES; thus, residual confounding from other aspects of SES is likely (King and Bearman 2011; Rai et al. 2012). Kalkbrenner et al. (2012) did not appropriately control for confounders, and this affected sensitivity analysis central to their conclusions. In their sensitivity analy sis for outcome misclassification, they did not correct for covariates, thus basing all of their interpretations on results that were contaminated by confounding. They could have used Monte Carlo methods (Bodnar et al. 2010) to adjust for confounding while accounting for outcome misclassification, obtaining confidence intervals that account for random simulation error, but they did not do this. Thus, the reported confidence intervals for the sensitivity analyses are likely to be too narrow. Kalkbrenner et al. (2012) did not quantitatively assess the impact of exposure misclassi fica tion. The quoted 0.8 concordance of smoking data on birth certificates with the medical record means that smoking exposures of > 125,000 persons in the sample were expected to be incorrectly classified. Sensitivity of maternal smoking on U.S. birth certificates is likely to be only 0.5 (Kharrazi et al. 1999). Epidemiologists ignore measurement error at great peril (Jurek et al. 2006) while correction procedures exist (MacLehose and Gustafson 2012).
Finally, we would like to point out the difficulties of this article in communicating scientific results to the general public. Because, as Kalkenbrenner stated, "the study doesn't say for certain that smoking is a risk factor for autism" (UWM News 2012), then it is the author's responsibility to more carefully report to the media what the study actually does say. It is easy to blame journalists for the sensational findings that have been reported about this study (e.g., Goodwin 2012). However, given the historic legacy of blaming parents, particularly mothers, for their child's diagnosis, we would better serve the communities for whom we do this research if we developed standard practices for reporting preliminary findings in ASD risk factor research. One suggestion would be to report these findings without discussion in media (e.g., Palmer 2011) and scholarly publications, as was done by Adam et al. (2011), who produced experi mental data demon strating that the speed of light was exceeded: Despite the large significance of the measurement reported here and the robustness of the analysis [p << 0.00006%], the potentially great impact of the result motivates the continuation of our studies in order to investigate possible still unknown systematic effects that could explain the observed anomaly. We deliberately do not attempt any theoretical or phenomenological interpretation of the results. We are grateful for the opportunity to comment on the importance of careful interpretation and communication of epidemiological results. We concur with Burstyn et al. that scientists have a great responsibility to present their findings accurately. Indeed, scientists have a duty not only to generate research results but also to engage in dialog about the interpretation and meaning of these results.

Presentation of Study
We also agree that our study on maternal smoking in pregnancy and the risk of autism spectrum disorders (ASDs) had limitations, including hetero geneous outcome groupings, residual confounding, and exposure misclassification (Kalkbrenner et al. 2012). We do not, however, share the conviction that these limitations void the validity and contribution of our study or that our results should not be accompanied with an interpretation.
Our study improved upon previous studies of tobacco exposures and ASDs in several ways. It incorporated population-based data from the United States. Much of the existing body of research on this subject has been conducted in Europe where social patterns of both smoking and ASD diagnoses differ from patterns in the United States. Furthermore, adjusting for maternal education, marital status, and maternal race/ethnicity was an improvement over many previous studies that were not able to adjust for these factors. Although residual social class confounding is possible, it would more likely have maskedrather than produced-the observed elevated associations. Finally, we included a large number of children with ASD and had some phenotypic information beyond a binary ASD/non-ASD classification. By including and reporting on several ASD functional sub classifications, we performed replicates within one study, increased the transparency and completeness of reporting, and enhanced the comparability with previous reports. The consistent pattern of results across subclassifications strengthened our interpretation that the overall pattern we observed was not due wholly to biases or random error.
Most puzzling to us is the null hypothe sis testing approach described by Burstyn et al. in their letter, in which associations were dichotomized to conclude a "link" when confidence limits excluded the null value, or "no link" otherwise. This dichotomy of complex results is not only a grave over simplification, but it is awash with assumptions that do not hold in observational epidemiology (Poole 2001;Savitz 1993).
Instead, we prefer a more meta-analytic mindset, in which a given study is considered as a contribution to a broader literature, with the weight of contribution proportional to its precision. Valid and precise results are interpreted for their public health or clinical importance, judged by the magnitude of effect.
As an illustration of the meta-analytic perspective, with no intention of definitively answering whether maternal smoking causes any subgroup of ASDs, we have performed a simple combination of results from our study (Kalkbrenner et al. 2012) with those from a similar recent population-based study of maternal smoking in pregnancy and ASDs (Lee et al. 2011). Using strict null-hypothesis testing interpretation, all original adjusted prevalence ratios support "no link" because the 95% confidence limits include the null value of 1.0 (Table 1). In contrast, a more nuanced interpretation of results, focusing on patterns of magnitude and precision, yields a conclusion that associations between maternal smoking and ASD may differ by the presence of co-occurring intellectual disabilities, consistent with the meta-analytic combined results.
In summary, the quality of our data, analytic approach, and interpretation improved upon previous studies of this important question. It is always possible to go further, and we are intrigued by the suggestion of Burstyn et al. to conduct an extended, multiple-bias sensitivity analysis. We explored the impact of underrecognition of ASD varying by social class (an important bias not previously addressed in the literature), but we did not evaluate this error together with the under reporting of tobacco use in pregnancy and residual social class confounding. The mathe mati cal methods to explore and correct such inter woven biases are being developed and made available, opening up the possibility of better modeling these errors. This analysis should be explored in future work, and on this point perhaps we are in agreement with Burstyn et al.