Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments

Background: Systematic-review methodologies provide objectivity and transparency to the process of collecting and synthesizing scientific evidence in reaching conclusions on specific research questions. There is increasing interest in applying these procedures to address environmental health questions. Objectives: The goal was to develop a systematic-review framework to address environmental health questions by extending approaches developed for clinical medicine to handle the breadth of data relevant to environmental health sciences (e.g., human, animal, and mechanistic studies). Methods: The Office of Health Assessment and Translation (OHAT) adapted guidance from authorities on systematic-review and sought advice during development of the OHAT Approach through consultation with technical experts in systematic review and human health assessments, as well as scientific advisory groups and the public. The method was refined by considering expert and public comments and through application to case studies. Results and Discussion: Here we present a seven-step framework for systematic review and evidence integration for reaching hazard identification conclusions: 1) problem formulation and protocol development, 2) search for and select studies for inclusion, 3) extract data from studies, 4) assess the quality or risk of bias of individual studies, 5) rate the confidence in the body of evidence, 6) translate the confidence ratings into levels of evidence, and 7) integrate the information from different evidence streams (human, animal, and “other relevant data” including mechanistic or in vitro studies) to develop hazard identification conclusions. Conclusion: The principles of systematic review can be successfully applied to environmental health questions to provide greater objectivity and transparency to the process of developing conclusions. Citation: Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122:711–718; http://dx.doi.org/10.1289/ehp.1307972


Engagement with the NTP Board of Scientific Counselors
The NTP Board of Scientific Counselors (BSC), our primary external advisory group, played a key role in reviewing and critiquing our plans and draft approaches for implementing systematic review into NTP literature-based health assessments. There were multiple public meetings of the BSC at which the development of a new systematic-review approach was discussed, and these meetings also provided members of the public a forum to be informed or to comment. At the June 2012 BSC meeting, we first outlined plans to develop an approach for systematic review and evidence integration in OHAT literature-based health assessments (NTP 2012a). Subsequent to that initial public vetting, a working group of the BSC met in August 2012 to review an initial Draft NTP Approach that built on guidance from authoritative sources and technical input for carrying out systematic review. This initial draft approach laid out a detailed structure for carrying out the steps of a systematic review and for synthesizing data to reach hazardidentification conclusions.
The working group was composed of experts knowledgeable in systematic-review methodology as well as hazard assessment (Supplemental Material, Table S1). They were charged to evaluate the suitability and transparency of the proposed approach for reaching evidence assessment conclusions from a body of evidence (collection of studies) for noncancer health effects. The working group chair presented the outcome of their deliberations to the BSC at its meeting in December 2012, and the BSC unanimously accepted the working group's report (NTP 2012b).
Overall, the working group commended the NTP for taking proactive steps to move forward the state of the science for hazard assessment. Two primary recommendations were (1) that the NTP approach document should not be a treatise as written in the initial draft, and instead, should provide a framework that defines the steps and structure for the systematic review, and (2) that a protocol should be prepared for each project with the specific details for that evaluation.
We carefully considered the working group's feedback and the initial Draft NTP Approach was revised based on all comments, including the restructuring of the approach as a framework document with project-specific details moved to protocols developed for each evaluation. At the BSC meeting, we presented our response to the working group's report along with a Revised Draft NTP Approach, which laid out our framework for systematic review and evidence integration and identified the protocol as the vehicle to detail a specific evaluation. We also proposed to carryout case studies to develop two project-specific protocols as recommended, and to apply the approach to test cases as a means of identifying needed refinement or revision to the approach.
In response to public input at the December BSC meeting, in February 2013, the NTP released for public comment the framework document−Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments -February 2013 (previously called "NTP Approach," now "OHAT Approach") (NTP 2013a). This was followed in April by release of the draft protocols for two systematic-review case studies: (1) the association of bisphenol A exposure and obesity and (2) the association of perfluorooctanoic acid and perfluorooctane sulfonate exposure and immunotoxicity (NTP 2013b, c).
In June 2013 we shared the draft OHAT Approach, draft protocols for the case studies, and public comments with the BSC at a public meeting. We presented how the draft addressed major technical or scientific issues in those comments including evaluation of study quality, the method for determining the initial rating for assessing confidence in the body of evidence, the impact of excluding studies based on quality, and use of other relevant data (in-vitro and mechanistic data, etc.). The BSC responded favorably to the Draft OHAT Approach, its consideration of public input, and our proposed means for resolving remaining issues. Their input supported us moving forward with finalizing the approach document (NTP 2013d).

Engagement with NTP-agency partners
Throughout development of the OHAT Approach, we periodically shared drafts and communicated our strategy with NTP-agency partners through webinars and briefings to the NTP Executive Committee (NTP 2013e). Several agencies (EPA and ATSDR) were actively considering modifications to their literature-based evaluation processes, and therefore discussions on our draft approach included potential for harmonizing data extraction, developing templates, and sharing data files to the extent possible within the differing agency mandates. We also readily accepted invitations from our partners to share details of the approach at their events (e.g., EPA's 2013 Systematic Review Workshop). It was important to understand any potential concerns with our proposal to use systematic-review methodology to assess the scientific evidence and reach hazard conclusions since many of these agencies use OHAT health hazard assessments in their evaluations.  Finally, as we worked toward finalizing the OHAT Approach, we sought to provide additional clarity on some issues in the public comments and provide a further opportunity for public input.

Stakeholder outreach and communication
On September 26, 2013, we held a public, informational webinar to address specific topics and themes in the comments and discuss our progress on the case studies (NTP 2013i). The topics included evaluating study quality and utility, determining the initial confidence rating in the body of evidence by study design features, and integrating evidence to reach conclusions. The case studies are currently in progress, and at the webinar we noted that once they are completed the   Study-design labels can distinguish between the relative strengths of study designs, but they are imprecise and often include a mix of design features that impact the ability of a study to address causality. Instead, four key study-design features can be used to differentiate the ability of the study to address causality as reflected in the confidence that exposure preceded and was associated with the outcome. Table S2 illustrates whether or not the four features are "likely" present, "may or may not" be present, or "unlikely" to be present in typical studies for each of the study designs listed.
In the OHAT Approach, the presence or absence of these four features is assessed on an outcome-specific basis to set the initial confidence rating for each study. "Controlled exposure" of subjects to the substance is the factor that distinguishes experimental studies from observational studies, and the experimental study design will also typically include the other three key features in both human and animal studies. The key feature that distinguishes between the relative strengths of observational epidemiologic study designs is "exposure prior to outcome," (i.e., the exposure assessment represents exposures that occurred prior to the development of the outcome). In these cases, it is unlikely that an association could be the result of reverse causation−where the outcome contributes to the exposure. Prospective cohort studies usually have three key study design features; however, when the exposures and outcomes are assessed at the start of a prospective study, these results will only have two key features and more closely resemble a cross-sectional study.
Studies without individual-level information on outcomes and other covariates cannot control for additional confounding variables and may lead to inappropriate inferences or an "ecologic fallacy." This limitation is captured with the third key feature "individual outcome data." An ecologic study can refer to exposures assessed via aggregate data (air pollution by zip code of residence) with individual subject outcome information (which would have the third feature), or it could refer to exposures and outcomes assessed on aggregate data (trends in a city's air pollution and hospitalizations for asthma) and not have the third feature.
Without a comparison group there is limited ability to evaluate the association of an exposure and outcome. The fourth key feature "comparison group used" distinguishes case series and case reports from the other study designs because they typically lack a comparison group.
Consideration of whether the comparison group used is appropriate is captured in the risk of bias assessment. Table S3. Level of evidence for health effects descriptors.

Evidence descriptors Definition
High level of evidence There is high confidence in the body of evidence for an association between exposure to the substance and the health outcome(s).
Moderate level of evidence There is moderate confidence in the body of evidence for an association between exposure to the substance and the health outcome(s).
Low level of evidence There is low confidence in the body of evidence for an association between exposure to the substance and the health outcome(s), or no data are available.
Evidence of no health effect There is high confidence in the body of evidence that exposure to the substance is not associated with the health outcome(s).

Inadequate evidence
There is insufficient evidence available to assess if the exposure to the substance is associated with the health outcome(s).