Elsevier

Technology in Society

Volume 27, Issue 4, November 2005, Pages 437-451
Technology in Society

Assessing the science–society relation: The case of the US National Science Foundation's second merit review criterion

https://doi.org/10.1016/j.techsoc.2005.08.001Get rights and content

Abstract

The science–society relation exhibits a tension between scientific autonomy and societal control of the direction and scope of scientific research. With the 1997 formulation of two generic merit review criteria for the assessment of National Science Foundation proposals—one for intellectual merit, and a second for ‘broader impacts’—this tension between science and society took on a unique institutional expression that has yet to work itself out into a well-accepted balance of complementary interests. This article examines some of the issues associated especially with the second ‘broader impacts’ criterion.

Introduction

Since its early modern origins, modern natural science has struggled to develop appropriate standards for quality assessment. In particular, one of the main issues has been the extent to which science ought to be judged only on its own terms. The history of science provides extensive literature on the effort to establish science as an autonomous human activity independent especially of religious or political manipulation. The experience of Galileo Galilei (1564–1642) and efforts to break free of church criticism of the heliocentric theory is perhaps the most well-known case in the religious arena. The Communist effort to promote the genetic theories of T.D. Lysenko (1898–1976) is an oft-cited case in the political arena. Such historical lessons taught us to espouse the Enlightenment ideal of a science that exists independently of religious or political influence for the ultimate benefit of society as a whole. From the very beginning, however, questions have occasionally been raised about whether scientific autonomy might be carried so far as to create an imbalance in the science–society relation. The Romantic response to Enlightenment science, for instance, questioned whether science could indeed stand on its own. Are there not times when scientific knowledge distorts lived reality? Is technological power not only a boon but also a danger to human welfare? This debate may be seen in terms of a conflict between advocates of internal and external criteria for evaluating science, with internalists championing autonomy while externalists argue for more societal control over the direction and scope of scientific research. Nevertheless, there are some on the inside (i.e. actual scientists) who recognize the importance of external considerations. Indeed, in the 1960s Alvin Weinberg, the physicist administrator of Oak Ridge National Laboratory, argued that external criteria had a proper and significant role to play in any scheme for assessing science.

In post World War II America, the debate surrounding the formation of the US National Science Foundation (NSF) also reflected this ambivalence: the strong autonomy advocacy of Vannevar Bush's Science—The Endless Frontier (1945) was moderated by the more pragmatic arguments of the Steelman Report (1947), which advocated more limited scientific autonomy in the name of public benefit. Although by the time the NSF was actually created in 1950 many of Bush's specific proposals for its formation were abandoned, his notion of the strong autonomy necessary for basic scientific research was institutionalized with the creation of the NSF and the development of protocols for the internal peer review of proposals to be funded by the federal government. We can credit Bush's rhetorical genius for arguing that a large degree of scientific autonomy was in fact necessary for producing the kind of societal benefits desired; and we can blame either the wishful thinking or woeful logic of many societal decision makers for the assumption that scientific autonomy was, therefore, sufficient for producing societal benefits. Decision makers have often made this assumption, however, with the result that tension persists between advocates of internal criteria of scientific merit and advocates of broader external criteria for assessing science.

With the 1997 formulation of two basic merit review criteria for the assessment of NSF proposals, this tension between science and society, internal autonomy and external evaluation, took on a unique institutional expression that has yet to work itself out into a well-accepted balance of complementary interests. This article examines some of the issues associated especially with the second ‘broader impacts’ criterion in an effort to contribute to the further evolution of a discussion of a distinctive issue in what may be termed the philosophy of science policy.

Section snippets

Background

In 1997 the Naztional Science Board (NSB), the NSF's policy branch, approved two new generic merit review criteria to replace the four that had been in effect since 1981.1 The two criteria approved in 1997 and currently used to evaluate all NSF proposals are: (1) What is the intellectual merit of the proposed activity? and

The specter of philosophical issues

The NAPA Report notes that ‘many reviewers either ignore Criterion 2 or in some cases regard it as irrelevant in the review of proposals’, that many reviewers ‘perceive Criterion 1 (scientific merit) and Criterion 2 (broader or societal impact) as in competition with each other’, and that many reviewers either ‘disregard Criterion 2 altogether or simply merge social value into scientific merit’ ([1], p. 13). Among the major recommendations of the NAPA panel is that ‘there is a need to improve

A brief historical outline of issues surrounding Criterion 2

One of the main reasons behind the 1997 restructuring of NSF's generic merit review criteria was the desire to link public investment in science with societal benefits, to demonstrate, in other words, that the people were getting a good return on their investment.3

Science policy and Criterion 2

The issues of whether, to what degree, how, and under what constraints scientific research should be publicly funded were raised long before Vannevar Bush wrote Science—the Endless Frontier.8 Nevertheless, it was Bush's answers to these questions that eventually led to the establishment of the National Science Foundation.9

Conclusion

Scientists and engineers deal professionally with what we might term broadly ‘matters of fact’, questions that are, in principle, resolvable by empirical means. Whether the theory of evolution correctly infers a common ancestry for all living things is just such a matter of fact, as is the question of whether life exists elsewhere in the universe: both are susceptible, in principle, to empirical testing. On the other hand, we also often encounter the opposite sort of question, a sort not

Acknowledgements

An earlier version of this paper served as part of a proposal to NSF to examine their merit review criteria. The author would like to thank several anonymous reviewers for their helpful criticisms. The author would also like to thank Bob Frodeman for insightful comments on many iterations of that proposal, as well as for pointing the author toward a philosophical examination of Criterion 2. Thanks also to Roger Pielke for some excellent guidance on sources on the linear model. Finally, the

Britt Holbrook received his PhD in philosophy from Emory University in August 2004. He has served as Research Assistant Professor within the Department of Philosophy and Religion Studies at the University of North Texas since January 2005. He has authored an entry on NSF's second merit review criterion for MacMillan Reference's Encyclopedia of Science, Technology, and Ethics as well as co-authoring (with Robert Frodeman) an article on the policy ramifications of Criterion 2 for Ogmius

References (28)

  • National academy of public administration (NAPA). A study of the national science foundation's criteria for project...
  • NSF. Task force on merit review's discussion report (NSB/MR...
  • NSF. NSF in a changing world (NSF...
  • NSF. Task force on merit review's final recommendations (NSB/MR...
  • NSF. Important notice no....
  • NSF. ‘Dear colleagues’ letter to PIs and reviewers (NSF...
  • NSF. Important notice no....
  • Senate report...
  • Senate report...
  • NSF. FY2000 report on the NSF merit review system (NSB...
  • NSF. FY2001 report on the NSF merit review system (NSB...
  • NSF. Important notice no....
  • NSF. Report of the advisory committee for GPRA performance assessment (NSF...
  • NSF. FY2003 performance and accountability report (NSF...
  • Cited by (60)

    • Evaluating the NSF broader impacts with the Inclusion-Immediacy Criterion: A retrospective analysis of nanotechnology grants

      2021, Technovation
      Citation Excerpt :

      Other scholars argue that broader impact requirements are based on a flawed linear model of R&D that does not match the reality of R&D (J. B. Holbrook and Frodeman 2007). Rather, models such as Pasteur's Quadrant, a two by two grid of consideration of use and quest for fundamental understanding, better match reality (Fisher et al., 2006; J. B. Holbrook 2005; Stokes 1997). Some argue that it would be more efficient to fund grants specifically designed to have impacts rather than forcing a natural science grant to include BIs.

    • Co-producing science for sustainability: Can funding change knowledge use?

      2020, Global Environmental Change
      Citation Excerpt :

      While some studies reported benefits of funding approaches that encourage interaction with various practitioners (e.g., DeLorme et al., 2016; Moser, 2016), others are concerned with unintended consequences and perverse incentives arising from interventions by funders (Lövbrand, 2011). Finally, studies also point to researchers sidestepping changes in rules to maintain the status quo, in spite of the best laid plans of funders and science policy-makers to effect change in research practices (Davis and Laas, 2014; Holbrook, 2012; Reale and Zinilli, 2017). In this study, we analyze research projects funded by the National Estuarine Research Reserve System (NERRS, a program of the U.S. National Oceanic and Atmospheric Administration– NOAA).

    View all citing articles on Scopus

    Britt Holbrook received his PhD in philosophy from Emory University in August 2004. He has served as Research Assistant Professor within the Department of Philosophy and Religion Studies at the University of North Texas since January 2005. He has authored an entry on NSF's second merit review criterion for MacMillan Reference's Encyclopedia of Science, Technology, and Ethics as well as co-authoring (with Robert Frodeman) an article on the policy ramifications of Criterion 2 for Ogmius (forthcoming).

    View full text