Toward the Assessment of Scientific and Public Health Impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program Using Available Data

Background In the past 15 years, asthma prevalence has increased and is disproportionately distributed among children, minorities, and low-income persons. The National Institute of Environmental Health Sciences (NIEHS) Division of Extramural Research and Training developed a framework to measure the scientific and health impacts of its extramural asthma research to improve the scientific basis for reducing the health effects of asthma. Objectives Here we apply the framework to characterize the NIEHS asthma portfolio’s impact in terms of publications, clinical applications of findings, community interventions, and technology developments. Methods A logic model was tailored to inputs, outputs, and outcomes of the NIEHS asthma portfolio. Data from existing National Institutes of Health (NIH) databases are used, along with publicly available bibliometric data and structured elicitation of expert judgment. Results NIEHS is the third largest source of asthma-related research grant funding within the NIH between 1975 and 2005, after the National Heart, Lung, and Blood Institute and the National Institute of Allergy and Infectious Diseases. Much of NIEHS-funded asthma research focuses on basic research, but results are often published in journals focused on clinical investigation, increasing the likelihood that the work is moved into practice along the “bench to bedside” continuum. NIEHS support has led to key breakthroughs in scientific research concerning susceptibility to asthma, environmental conditions that heighten asthma symptoms, and cellular mechanisms that may be involved in treating asthma. Conclusions If gaps and limitations in publicly available data receive adequate attention, further linkages can be demonstrated between research activities and public health improvements. This logic model approach to research impact assessment demonstrates that it is possible to conceptualize program components, mine existing databases, and begin to show longer-term impacts of program results. The next challenges will be to modify current data structures, improve the linkages among relevant databases, incorporate as much electronically available data as possible, and determine how to improve the quality and health impact of the science that we support.

Growing demand for performance measurement and accountability in public research funding has led to a shift in focus for research impact assessment (RIA) beyond publications to measure accountability of research investment in broader terms, such as human health and environmental improvements as well as economic benefits. Conventional bibliometric analysis, which assesses contributions to knowledge, may not be sufficient to assess public health impacts.
Recent critiques of approaches to evaluating government-funded research underscore the need for better models and stronger linkages between evaluation and decisions about funding allocations (Coryn et al. 2007). Trochim and colleagues (2008), for example, propose an extensive mixed-methods approach that draws from diverse data sources to evaluate a large research initiative of the National Cancer Institute. A more tightly circumscribed approach is reflected in the work of Kuruvilla and colleagues (2006), who developed a research impact framework to help researchers systematically identify impacts resulting from their work. The tool also provides a structured framework for comparing research impacts across projects and time but is limited in that it includes data from only one source, the researcher. These researchers did not develop a full logic model, but rather identified specific descriptive categories within four broad areas: research-related impacts, policy impacts, service impacts, and societal impacts.
Increased interest in documenting the public health impacts from publicly funded research led the National Institute of Environmental Health Sciences (NIEHS) Division of Extramural Research (DERT) to develop a logic model to measure the impact of its extramural grant-funded investments in asthma research and to identify indicators to monitor changes over time (Figure 1). This more specific asthma-related framework was derived from work published earlier (Engel-Cox et al. 2008) to categorize the encompassing range of possible impacts of environmental health research related to human health, the economy, and environment. Presented here is a case study of how new methodologies and data sources can move us toward an assessment of public health research impacts, identifying data structure improvements needed to hasten this movement. The types of research impacts reported here were those judged to be feasible to assess related to the field of asthma. Our objective in the present study was to apply this conceptual framework in assessing the impacts of DERT's extramural asthma research grants funded from 1975 to 2005.
We chose impacts of asthma research because of asthma's public health significance. Asthma's prevalence in the United States has nearly doubled since 1970, and it is now the most common chronic childhood disease, with a prevalence of 8.5% among children in 2004 (Moorman et al. 2007). In addition to morbidity and associated missed school days for children, asthma results in missed work days for parents, restrictions on physical activity, and other quality-of-life impairments. It is also well documented that persons from lowincome and minority households are disproportionately affected by asthma (Brim et al. 2008;Persky et al. 2007) and that children from these homes are more likely to be treated in emergency and urgent care facilities because they lack adequate health care insurance coverage (Inkelas et al. 2008).
volume 117 | number 7 | July 2009 • Environmental Health Perspectives President Clinton's 1997 Executive Order 13045 required each federal agency to identify and evaluate environmental health and safety risks that may hinder children's health (Clinton 1997 NIEHS supports a multifaceted asthma research program, including basic research into asthma genetics, respiratory pathogenesis, and mechanisms by which particulate matter, aerosols, and aeroallergens induce asthmatic response; new methods for exposure measurement to dust and airborne allergens associated with asthma; national surveys to assess the risks of exposure to allergens and other environmental agents associated with asthma; the effectiveness of interventions that lower allergen levels in the home; and the effectiveness of interventions for secondary prevention of asthma.

Approach
In 2006, DERT convened an expert panel to review the present state of knowledge about measuring the impact of research and to inform the development of the aforementioned conceptual framework that describes the agency's extramural research portfolio (Engel-Cox et al. 2008). Panel members were selected based on their recognized research contributions along the whole bench-to-bedside-to-community translational continuum. In addition, we refined the conceptual model specifically for the asthma research portfolio and conducted a feasibility assessment of data sources and potential metrics of interest. Expert panel members were also asked to help assess the feasibility of accessing different data sources in conducting the evaluation and the validity of different metrics. We then compiled information from those data sources identified and developed attribution scenarios of how research impacts could be linked to the identified grant funding.
The conceptual framework uses a logic model format (Kellogg Foundation 2004). Components include contextual conditions, inputs, research activities, outputs, outcomes, and ultimate outcomes, with emphasis on science, clinical policy and practice, environmental policy and practice, and health outcomes. We developed indicators for each component using publicly available data sources (Appendix 1) and a structured elicitation of subject matter experts' judgment.
In assessing the public health impacts of a research program, the most noticeable immediate outputs include the use of agency-supported research in subsequent research, such as citations in the scientific literature or in follow-up research studies. The Scientific Publication Information Retrieval and Evaluation System (SPIRES 2007) was used to link citations. Other possible immediate outcomes are the use of research in influencing the work of commissions, task forces, or advisory panels, and the dissemination of interventions beyond the initial research study. Potentially effective community-based interventions were linked to publications stemming from NIEHS-funded grant research. Appointees to these committees are often viewed as leaders in the field and as having influence on agency or science policy and guidelines. Composition of commissions, task forces, and agency advisory panels was reviewed to determine whether NIEHS-sponsored investigators participated. Immediate outcomes were assessed through bibliometric analysis, content analysis using the OmniViz content mapping and visualization software (Biowisdom, Inc., Cambridge, UK), and other database queries. Conceptual framework for NIEHS asthma-related research portfolio. ER, emergency department. This logic model lays out inputs, activities, outputs, and outcomes of the NIEHS asthma-related research portfolio. Boxes represent types of information collected for the portfolio analysis. Gray text indicates gaps in electronically accessible data from which queries can be made and potential areas for future work. Intermediate outcomes result as immediate outcomes become translated into impacts. Three main types of impacts include economic, environmental, and health and social. Economic impacts may include changes in technology that affect asthma, such as new patents or processes, or new drugs developed. We searched the U.S. patent database and U.S. Food and Drug Administration (FDA) databases of drugs to compile this information. Environmental impacts may include changes in regulations, policies, or legislation that eventually result in changes in environmental measures, such as outdoor or indoor air quality indicators derived from U.S. Environmental Protection Agency (EPA) databases and U.S. federal legislative databases. Finally, health and social changes may include changes in guidelines (National Guideline Clearinghouse 2009) or health legislation (from U.S. federal and state legislative databases) that may eventually result in behavioral changes for health care providers, health systems, families, schools, or other youth settings. In a preliminary feasibility assessment (Liebow et al. 2006), we determined that regulations, legislation, and formal policy statements rarely cite the evidence on which they are based, so that such policy instruments cannot be attributed directly to specific research projects or centers. Impacts that can be more clearly attributed include economic impacts such as patents, technology changes, and drug development, as well as health and social impacts that include guidelines and practice changes.

Inputs
Ultimate outcomes are a combined result of changes in intermediate outcomes over time and may include decreased asthma morbidity and mortality, incidence of asthma, emergency department utilization, decrease in hospitalization rates, use of rescue medicines, activity limitations among persons with asthma, and decrease in number of school or work days missed by persons because of asthma. These data were primarily compiled from asthma surveillance data available from the U.S. Centers for Disease Control and Prevention (CDC).

Results
Results are summarized for each conceptual framework element: context, inputs, activities, outputs, and outcomes.
Context. Annual congressional appropriations to NIH. Annual appropriations to NIH grew steadily from $4.3 billion in 1983 to $28.4 billion in 2005 (NIH Office of Budget 2008). During this same period, the NIEHS budget also grew, but at a slower pace, from $164.6 million to $720.2 million. The budgets of other NIH institutes and centers grew considerably faster, most notably at NIAID and NHLBI, growing from $278.9 million to nearly $4.3 billion and $624.1 million to $2.9 billion, respectively.
Inputs. Grants and research funding. Between 1983 and 2006, DERT spent $218 million on 161 individual asthma-related research grants, adding to a total of 736 grant years [the aggregate number of years in which all individual principal investigators (PIs) affiliated with a specific institution received funding from NIEHS] ( Table 1). Total extramural funding has increased over time, and asthma-related grant awards have risen as a percentage of the total extramural portfolio over time, climbing to a peak of 9.8% of the extramural portfolio in 2003 and dropping to 7.9% in 2005. The relative share of asthmarelated funding invested in research has also increased over time.
Research mechanisms. NIEHS employs a range of grant mechanisms to fund asthma-related research. For the purposes of this analysis, mechanisms have been grouped into four general categories: training grants, innovation research/technology transfer grants, center grants (large multiproject grants issued to institutions), and research grants ( Figure 2). Research grants are by far the most common type of award found in the asthma portfolio, but after 1999 the total funding for center grants (which include multiple coordinated projects as well as infrastructure grants) exceeds the funding for research grants. The number and funding of center grant awards fell between 2003 and 2005. The number of training grants related to asthma increased in the early 2000s, but the funds awarded for training grants are much lower, most likely because training grants provide only salary and tuition for predoctoral and postdoctoral trainees and do not support all researchrelated expenses. Innovation grants are the award type least commonly observed in the asthma portfolio.
Institutional profile. Before 1996, the number of grants, PIs, and institutions funded were nearly identical. After 1996, the number of individuals and institutions funded for asthma research from NIEHS has mirrored the increases seen in the numbers of grants funded, but at lower rates of growth [see Supplemental Material, Figure 1 ( Additionally, 63% of the funded institutions received funding for just one grant (although such grants may have continued over many years), and two institutions, Columbia University and Johns Hopkins University, received more than five unique asthma-related grants.
Activities. The annual number and funding levels of active grants are classified according to six general activity categories: basic science (e.g., cellular/molecular, genetics); organ system/disease process (e.g., respiratory); application of technology to disease (e.g., Small Business Innovation Research/ Small Business Technology Transfer); center grants (e.g., Children's Environmental Health because the current categories are not mutually exclusive (each grant can be classified in up to two categories) and center grants often encompass multiple projects and centralized facility services associated with several types of activities either directly or indirectly.
Outputs. Publications. The NIEHS asthma-related portfolio of 161 unique grants has 2,057 asthma-related journal publications associated with it (Table 2). Over the 30-year study period, the annual rate of publication from the asthma portfolio has roughly followed the portfolio's funding profile, with a sharp spike in publications emerging after 1998. During the 1990s, the rate of publications increased before the dramatic funding increase, possibly indicating that preliminary studies accumulated to make a compelling case for increased research support. As funding increased dramatically, a time lag is evident, likely attributable to the time it takes to conduct environmental health research and publish findings in peer-reviewed literature. Even with the expansion of electronic publishing, most journals currently retain a dual print/electronic format, and page limits persist, leaving a time lag. The top 10 journals for NIEHS-supported research findings are a mix of basic science, clinical, and environmental health ( Table 3). The total number of publications by a single PI ranged from 0 to > 100. Approximately 25% of PIs with grant awards were not associated with any publications, whereas approximately 7% of the NIEHSsupported asthma researchers were associated with grants generating > 100 publications. Approximately 75% of all the asthma-related publications acknowledging NIEHS support cited center grant awards.
Asthma-related interventions. A broad range of potentially effective communitybased interventions were reported in publications stemming from NIEHS-funded grant research. Highlights of NIEHS-supported intervention research include such findings as control of asthma in children entails collaborative efforts and community-wide environmental control measures (Clark et al. 1999); neighborhood asthma coalitions are associated with promising reductions in acute care rates among active program participants (Fisher et al. 2004); and intensive cleaning can produce significant reductions in cockroach allergen in homes with heavy initial cockroach infestations (McConnell et al. 2003).
Overall, evidence of outputs is widely available in publicly available data sources directly traceable to specific grants. These linkages have been possible only recently with the development of the SPIRES application (developed in large part at NIEHS and now available across NIH) that finds peer-reviewed articles in the PubMed database (http://www. ncbi.nlm.nih.gov/sites/entrez?db=pubmed) that cite NIH publications. There are some limitations in these data, however. Despite a recent congressional mandate (NIH Reauthorization Act of 2006), grantees do not always include grant numbers in publications, important journals are not included in the PubMed database, and older grant numbers are not always unique. These challenges are expected to improve over time, and we are reasonably confident that the outputs attributable to the asthma portfolio are representative of the majority of papers produced.
Immediate outcomes. Citations. The 2,057 publications attributed to the NIEHS
Role of the environment in asthma susceptibility. NIEHS also has a large concentration of asthma-related publications concerning environmental toxicants on the National Priority (Superfund) List (U.S. EPA 2008). Research focusing on the chemicals identified as the most hazardous among known or threatened releases has significant potential for public health improvement. Toxicants that are mentioned prominently in publications from NIEHS-sponsored research include air pollution ingredients (e.g., carbon monoxide and other products of combustion; polycyclic aromatic hydrocarbons, including naphthalene and other benzene compounds), environmental tobacco smoke and its constituent ingredients, agricultural chemicals, and lipopolysaccharides and other endotoxins.
Commissions, task forces, advisory panels, work groups. Development of expertise that then informs health policy changes is one public health outcome of interest to DERT. The 2005 membership of various expert groups commissioned to assess the need for policy change was cross-referenced with the roster of NIEHS asthma grant awardees. We identified multiple FDA and Institute of Medicine (IOM) commissions, task forces, advisory panels, and workgroups relevant to asthma. For example, recent FDA science advisory panels include the Anesthesiology and Respiratory Therapy Devices Panel and the Pulmonary-Allergy Drugs Advisory Committee. For IOM, groups examined included current IOM members, as well as more specialized groups such as the Environmental Health Sciences, Research, and Medicine Roundtable. We also identified a current advisory panel convened by the Joint Commission on the Accreditation of Healthcare Organizations (The Joint Commission) to recommend asthma-related hospital performance measures. In addition, we included advisory panels convened to develop the two major clinical guidelines for asthma care: The Global Initiative for Asthma and the National Asthma Education and Prevention Program (NAEPP). Ten NIEHS agency-funded PIs could be identified as current participants in IOM, Joint Commission, and NAEPP panels.
Intermediate outcomes. Economic impacts. Asthma can be a disabling chronic disease that results in lost work and school days, emergency department visits, and other adverse impacts with significant financial dimensions. Thus, research that leads to improvements in disease management and reducing the incidence of asthma can have substantial long-term Taking the abstracts of all asthma-related publications resulting from the NIEHS and comparison agency grant awards , the OmniViz program was used to assess word occurrence, distribution, and associations in the title and abstract of each publication to define the major discriminating themes and to cluster documents with related themes. Proximity is a measure of thematic similarity. The closer the papers are, the more similar they are. Six clusters emerge, each outlined in a box. The blue nodes indicate publications sponsored by the comparison agencies, whereas the yellow nodes indicate publications sponsored by NIEHS. Substantively, NIEHS has concentrated more of its efforts in the response mediators and environmental exposures domains.  Research resulting in patents is not always traceable to the funding organization (Anderson et al. 1996, Narin 1994, because the findings may be cited as "prior art" and only indirectly attributable to the funding organization. A search of the U.S. Patent Office online database (all patents since 1790) and focusing on the Government Interest field yielded 66 patents with ES listed as a part of a grant number. Of these, four patents (all to one PI) matched to a specific NIEHS-funded grant. Emerging efforts at NIEHS and NIH are working to improve data structures and to link patent data with grant funding.
Technology development. Drug development is based on clinical trials research, supported by basic research in biochemistry and pathogenesis, and followed by postmarketing surveillance and clinical case reports. Clinical trials are often sponsored by pharmaceutical companies, although NIH has reaffirmed its commitment to support clinical research over the past decade (Zerhouni 2003). We searched FDA drug approval dockets for publications cited that acknowledge NIEHS sponsorship, and identified 39 known asthma medications.
Information about the new drug applications for each of the drugs was searched on the FDA Web site. Bibliographic references for each drug were compiled and then matched against the master bibliographic database associated with asthma-related grants. NIEHS-sponsored research results in publications concerning asthma drugs cromolyn, albuterol, epinephrine, and fluticasone. Although the association is somewhat weak, we were able to link NIEHS research to drug research. Many might argue whether this is appropriate, as the main mission of NIEHS is to study the environmental factors contributing to susceptibility rather than to develop therapeutic treatments. However, because many of the mechanistic processes are related, it is natural that there would be at least some overlap.
Additional impacts and attribution difficulties. Ideally, the NIEHS asthma program logic model (Figure 1)  Indicators for all of these impacts have been included in a monitoring and evaluation database that NIEHS has developed for periodic research impact assessment (unpublished database). However, in none of these cases were we able to link trends to specific research activities because either a) it was not feasible to collect the data, or b) the data sources available had no connection to publications or grants to make such a linkage. For example, legislative databases often include a legislative history and expert testimony in support of legislation, but in the most optimistic scenario, this background information takes the form of "research has shown that . . ." without actually citing any specific researcher's work. It is difficult to obtain information without interviewing individuals about the role of science funded by NIEHS in developing legislation or regulation. Future assessments will address these issues with some of the end users of such research, that is, regulators and legislators.

Limitations
We have assumed from the outset that to be transparent and sustainable, periodic evaluations of the research portfolio and its public health impacts should be based on indicator data that are • Readily accessible to NIEHS staff • Consistently organized over the assessment period • Electronically available • Documented with respect to population covered, geographic area covered, and interval or frequency of collection. Through this initial effort we have discovered that indicator data meeting these criteria are generally available to support conventional bibliometric analyses of research publications. Certain limitations apply, however. Beyond publications, indicators of other activities, outputs, and outcomes are not as well supported. Refinements to current data structures within the agency, such as estimating the portion of a given center grant related to a disease, condition, or health end point, would assist with public health impact assessments. Conference papers and presentations, often a channel for early release of findings, rarely include publicly available information about funding sources. Commission/taskforce membership is difficult to cross-reference with grant recipients because there are no central repositories of active group rosters for comparison with author lists. In addition, although current members of such groups are often listed on agency Web sites, past commission/task force membership lists are not easily available. Thus, it is difficult to know whether agency-sponsored PIs participated in such activities in the past. Legislation, which can have a profound impact on public health, almost never cites research findings that may . Distribution of genes of interest across grants, by agency. This CoMet visualization highlights the specific genetic foci (columns) associated with each agency's asthma-related research portfolio (rows). The warmer the color (i.e., yellow, orange, and red), the greater the number of grants awarded for research on a particular genetic location or complex. The cooler the color (green, blue, purple), the fewer the number of grants awarded for a particular genetic location or complex. This visualization shows the varied agency interests in specific genes, indicating unique niches and areas of overlap. NIEHS, although not awarding as many grants as its NHLBI and NIAID counterparts, is focusing its resources in a manner mostly consistent with these two institutes. A gene that is hot in NIEHS-supported grants but studied at only one other institute is glutathione S-transferase, an oxidative stress gene often associated with environmental exposures such as ozone. well be what has prompted the legal intervention. Additional work is underway to survey asthma researchers funded by NIEHS (1975NIEHS ( -2005 and likely end users of this research to obtain their views on dissemination channels most likely to have policy and public health impacts. Additionally, grant data available in the Information for Management, Planning, Analysis, and Coordination (IMPAC II) database at the time of this analysis were restricted to title and abstract, thus limiting the extent of information that could be extracted. For example, additional information about associated genes is likely available in the "specific aims" sections of grant applications but until recently were not searchable. Similarly, annual progress reports, which are not searchable within the IMPAC II database, contain a wealth of information about the actual research conducted (not just the proposed research) and about results and outcomes that may not be publishable findings and thus might not emerge in a bibliometric analysis.
The ultimate outcome measures, representing indicators of decreased asthma morbidity and mortality, are also limited. Ultimate outcomes take the longest time to realize and may not be observed for ≥ 10 years after research findings are first reported. Although ultimate outcome measures should be included in our long-term assessments of scientific and public health impacts, they are the least direct of all, and attribution to a particular research project or program will likely never be definitive.

Conclusions
Although conventional bibliometric methods can be useful to understand improvements in knowledge relating to public health advances, these methods alone are insufficient to assess the broad range of contributions made possible by research funding. Instead, using a conceptual framework and reviewing a broad variety of existing electronic databases can help to shed additional light on the types of outcomes that are possible and salient for assessing research impact. NIEHS has taken a significant step toward implementing a framework that helps us think globally about specific mechanisms by which research investments contribute to public health improvements. Although the nature of scientific discovery is well enough infused with contingency and serendipity to thwart any effort to achieve definitive attribution of public health improvements to a specific research grant award, a portfolio analysis like the one presented here clearly points to more immediate outcomes that, in the aggregate, become the necessary and sufficient conditions under which such public health improvements are achieved.
The call for increased transparency and accountability in federal science policy requires that we think strategically about modifications to current data structures that allow us to track expenditures and research results over time. Improved linkages between relevant databases (such as searchable patent and investigational new drug applications with the federal grant database) will assist with attribution of near-term and intermediate outcomes to research funding. Improved data structures and the ability to search progress reports and volume 117 | number 7 | July 2009 • Environmental Health Perspectives specific aims side by side with grant titles and abstracts will increase access to scientific output data useful for impact assessment. As the use of publication data increases, grantees will become even more aware of the importance of acknowledging grant support, creating a more complete picture of the publication products of the extramural research portfolio. This in turn may increase the likelihood that grant support will be properly acknowledged in publications and downstream clinical and public health applications.
Making sense of the vast and highly segmented landscape of research reports and publications produced with public grant support is a challenge, but new content analysis tools, coupled with visualization techniques, show great promise in taking documents in diverse formats to highlight patterns over time and across narrowly defined subspecialty areas.
NIEHS efforts reported here and in the companion article by Engel-Cox et al. (2008) demonstrate that it is possible to conceptualize program components, mine existing databases, and begin to show longer-term impacts of program results. The next challenges will be to modify current data structures, improve the linkages among relevant databases, and determine how to improve the science that we support. DERT's charge is to distribute resources in ways that produce the best science to achieve a healthier population. We can now begin to ask questions such as: How should DERT structure its funding portfolio to create the best chances to produce innovation? How should investigators be made aware of ways in which their work can inform policies and environmental management regimes that, in turn, lead to public health improvements? To create interdisciplinary synergies, what is the ideal portfolio blend of institutional depth and breadth of research investments? We cannot answer these questions now, but the framework allows us to sharpen the focus of our thinking about how to structure information to answer these questions in the future.