Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Original Article
  • Published:

Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene

Journal of Exposure Science & Environmental Epidemiology volume 11, pages 308–322 (2001)Cite this article

Abstract

This paper presents the results of the first phase of a study, conducted as an element of the National Human Exposure Assessment Survey (NHEXAS), to demonstrate the use of expert subjective judgment elicitation techniques to characterize the magnitude of and uncertainty in environmental exposure to benzene. In decisions about the value of exposure research or of regulatory controls, the characterization of uncertainty can play an influential role. Classical methods for characterizing uncertainty may be sufficient when adequate amounts of relevant data are available. Frequently, however, data are neither abundant nor directly relevant, making it necessary to rely to varying degrees on subjective judgment. Since the 1950s, methods to elicit and quantify subjective judgments have been explored but have rarely been applied to the field of environmental exposure assessment. In this phase of the project, seven experts in benzene exposure assessment were selected through a peer nomination process, participated in a 2-day workshop, and were interviewed individually to elicit their judgments about the distributions of residential ambient, residential indoor, and personal air benzene concentrations (6-day integrated average) experienced by both the non-smoking, non-occupationally exposed target and study populations of the US EPA Region V pilot study. Specifically, each expert was asked to characterize, in probabilistic form, the arithmetic means and the 90th percentiles of these distributions. This paper presents the experts' judgments about the concentrations of benzene encountered by the target population. The experts' judgments about levels of benzene in personal air were demonstrative of patterns observed in the judgments about the other distributions. They were in closest agreement about their predictions of the mean; with one exception, their best estimates of the mean fell within 7–11 μg/m 3 although they exhibited striking differences in the degree of uncertainty expressed. Their estimates of the 90th percentile were more varied with the best estimates ranging from 12 to 26 μg/m 3 for all but one expert. However, their predictions of the 90th percentile were far more uncertain. The paper demonstrates that coherent subjective judgments can be elicited from exposure assessment scientists and critically examines the challenges and potential benefits of a subjective judgment approach. The results of the second phase of the project, in which measurements from the NHEXAS field study in Region V are used to calibrate the experts' judgments about the benzene exposures in the study population, will be presented in a second paper.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8

Similar content being viewed by others

Notes

  1. Variability is the distribution of quantity over time, space, age, or other factors. Variability is, in theory, knowable and ultimately irreducible. Uncertainty (also referred to as knowledge or epistemic uncertainty (Paté-Cornell, 1996) can be characterized as the distribution of a quantity, which may have a true value but one which is currently unknown. Uncertainty is, in theory, often reducible by appropriate research.

  2. The NHEXAS is a major research program launched by US EPA's Office of Research and Development (ORD). The program was conceived to improve research methods for measuring population exposures to environmental contaminants, to provide data for validating exposure models, and more generally to provide improved data for environmental policy decisions (Sexton et al., 1995).

  3. Volatile organic compounds (VOCs) were collected on 3M OVM 3520 charcoal badges. In the earlier TEAM studies, the VOCs were collected on Tenax cartridges using battery-operated pumps over two 12-h periods representing a full 24-h period.

  4. Individuals were allowed to self-nominate, but only three of the experts on the panel chose to do so and their votes did not substantially alter the rank.

  5. The number of participants was determined by project resources.

  6. Heuristics in this context refer to cognitive devices or procedures by which individuals make judgments in the presence of uncertainty (Kahneman et al., 1982; Morgan and Henrion, 1990). See Appendix for a brief discussion of heuristics.

  7. For reasons that were beyond the control of the project, the interviews could not take place until 6–9 months after the workshop rather than immediately following as originally planned. John Evans was not available to participate in the interviews as originally scheduled.

  8. The protocol was pilot-tested on David MacIntosh, University of Georgia, and Barry Ryan, Emory University, and revised to the form used in the study.

  9. The purpose of the mental model was to help combat one of the common heuristic procedures relied on in giving judgments — that of availability (see Appendix).

  10. The 90% “credible” interval is that interval which the expert believes to include the true value with 90% probability.

  11. Several experts expressed a preference for detailed modeling of the quantities they were asked to predict, which was not possible for this study. Limited modeling support was available.

  12. For example, Expert G specified inputs to a simple microenvironmental model, which simulated variability in ambient, indoor, and personal exposures using Monte Carlo sampling techniques. We also estimated the distributions for the mean and 90th percentiles of ambient, indoor, and personal exposure distributions for Expert E using estimates of variability and uncertainty given by Expert E and two-dimensional Monte Carlo simulation techniques.

  13. In the general form of a microenvironmental model, presented below, personal exposure to an individual is the sum of exposures encountered in different locations in which the individual spends time. Each microenvironmental exposure is the product of the concentration encountered and the fraction of time spent in that microenvironment:

    where Ei=exposure to the ith individual, Cij=concentration encountered by the ith individual in the jth microenvironment, fij=fraction of time (e.g., 24-h day) spent by the ith individual in the jth microenvironment.

  14. While other experts also used the TEAM data, Expert E appears not to have made adjustments to his estimate to account for declines in benzene content of gasoline and other regulations, which have resulted in lower releases of benzene to ambient air.

  15. Formal decision analytic techniques known as VOI analyses have been developed to help integrate quantitative responses to these questions. Several examples exist of VOI techniques applied to environmental problems (Evans et al., 1988; Finkel and Evans, 1988; North et al., 1992; 1Taylor et al., 1993; Dakins et al., 1996; Thompson and Evans, 1997).

  16. The authors initially considered a disaggregated approach to this project, but the complexity of the model and the potential number of parameters for which sets of judgments would have to be elicited made it necessary to switch to a direct approach.

  17. Although a comparison between the results of such a model and the judgments expressed in this study would be valuable, the current BEADS model predictions include exposures to active smokers and occupationally exposed individuals, making a direct comparison inappropriate. Future work is planned to allow a comparison to be made between the two approaches.

Abbreviations

BEADS:

Benzene Exposure and Absorbed Dose Simulation model

CRARM:

Commission on Risk Assessment and Risk Management

ETS:

environmental tobacco smoke

NHEXAS:

National Human Exposure Assessment Survey

NRC:

National Research Council

TEAM:

Total Exposure Assessment Methodology

US EPA:

US Environmental Protection Agency

References

  1. Amaral DAL, Estimating uncertainty in policy analysis: health effects from inhaled sulfur oxides, PhD Thesis, Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA, 1983 (as cited in Morgan and Henrion, 1990)

  2. Bostrom A, Fischhoff B, and Morgan MG, Characterizing mental models of hazardous processes: a methodology and an application to Radon, J Soc Issues (1992) 48(4): 85–100

    Article  Google Scholar 

  3. Bostrom A, Atman C, Fischhoff B, and Morgan MG, Evaluating risk communications: completing and correcting mental models of hazardous processes, Part II, Risk Anal (1994) 14(5): 789–798

    Article  CAS  PubMed  Google Scholar 

  4. Browner C, Policy for risk characterization, Memorandum from Carol Browner, EPA Administrator, dated March 21, 1995

  5. Burke T Anderson H Beach N Colome S Drew RT Firestone M Hauchman FS Miller TO Wagener DK Zeise L and Tran N, Role of exposure databases in risk management, Arch Environ Health (1992) 47(6): 421–429

    Article  CAS  PubMed  Google Scholar 

  6. CARB (1991), Activity patterns of California residents, Final Report, Contract No. A6-177-33

  7. Cooke RM 1991 Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, New York, NY

    Google Scholar 

  8. CRARM (Commission on Risk Assessment and Risk Management), Risk assessment and risk management in regulatory decision making, Draft Report, Commission on Risk Assessment and Management, Washington, DC, June 1996

  9. Cullen AC and Frey HC, Developing distributions for use in probabilistic exposure assessment: a handbook for dealing with variability and uncertainty in models and inputs Unpublished draft 1997

  10. Dakins ME Toll JE Small MJ and Brand KP, Risk-based environmental remediation: Bayesian Monte Carlo analysis and the expected value of sample information, Risk Anal (1996) 16: 67–79

    Article  CAS  PubMed  Google Scholar 

  11. De Cock J, Kromhout H, Heederik D, and Burema J, Experts subjective judgment assessment of pesticide exposure in fruit growing, Scand J Work Environ Health (1996) 22(6): 425–432

    Article  CAS  PubMed  Google Scholar 

  12. Evans JS Hawkins NS and Graham JD, Uncertainty analysis and the value of information: monitoring for radon in the home, J Air Pollut Control Assoc (1988) 38: 1380–1385

    CAS  Google Scholar 

  13. Evans JS Graham JD Gray GM and Sielken RL, A distributional approach to characterizing low-dose cancer risk, Risk Anal (1994a) 14(1): 25–34

    Article  CAS  PubMed  Google Scholar 

  14. Evans JS Gray GM Sielken RL Smith AE Valdez-Flores C and Graham JD, Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency, Regul Toxicol Pharmacol (1994b) 20: 15–36

    Article  CAS  PubMed  Google Scholar 

  15. Finkel AM and Evans JS, Evaluating the benefits of uncertainty reduction in environmental health risk management, J Air Pollut Control Assoc (1988) 37: 1164–1171

    Google Scholar 

  16. Goldman LR Gomez M Greenfield S Hall L Hulka BS Kaye WE Lybarger JA Mckenzie DH Murphy RS Wellington DG and Woodruff T, Use of exposure databases for status and trends analysis, Arch Environ Health (1992) 47(6): 430–438

    Article  CAS  PubMed  Google Scholar 

  17. Goldstein BD Tardiff RG Baker SR Hoffnagle GF Murray DR Catizone PA Kester RA and Caniparoli DG, Valdez Air Health Study. Alyeska Pipeline Service, Anchorage, AK 1992

    Google Scholar 

  18. Graham JD Walker KD Berry M Bryan EF Callahan MA Fan A Finley B Lynch J Mckone T Özkaynak H and Sexton K, Role of exposure data bases in risk assessment, Arch Environ Health (1992) 47(6): 408–420

    Article  CAS  PubMed  Google Scholar 

  19. Habicht FH, Guidance on risk characterization for risk managers and risk assessors, Memorandum from F. Henry Habicht, EPA Deputy Administrator, dated February 26, 1992 (reprinted in Appendix B in NRC, 1994)

  20. Hawkins NC and Evans JS, Subjective estimation of toluene exposures: a calibration study of industrial hygienists, Appl Ind Hyg J (1989) 4: 61–68

    Article  CAS  Google Scholar 

  21. Hawkins NC and Graham JD, Expert scientific judgment and cancer risk assessment: a pilot study of pharmacokinetic data, Risk Anal (1988) 8(4): 615–625

    Article  CAS  PubMed  Google Scholar 

  22. Hora, SC, Acquisition of expert judgment: examples from risk assessment, J Energy Eng (1992) 188(2): 136–148

    Article  Google Scholar 

  23. Kahneman D, and Tversky A, On the psychology of prediction, Psychol Rev (1973) 80(4): 237–251

    Article  Google Scholar 

  24. Kahneman D, Slovic P, and Tversky A, Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge, UK 1982

    Book  Google Scholar 

  25. Kromhout H, Ostendorp Y, Heederik D, and Boleij JSM, Agreement between qualitative exposure estimates and quantitative exposure measurement, Am J Ind Med (1987) 12: 551

    Article  CAS  PubMed  Google Scholar 

  26. Macaluso M, Delzell E, Rose V, Perkins J, and Oestenstad K, Inter-rater agreement in the assessment of solvent exposure at a car assembly plant, Am Ind Hyg Assoc J (1993) 54: 351–359

    Article  CAS  PubMed  Google Scholar 

  27. MacIntosh DL Özkaynak H Xue J and Ryan PB, NHEXAS pre-field exposure assessment study: estimated benzene exposures and absorbed doses in EPA Region 5 and Arizona, Report to Karen Hammerstrom, U.S EPA, April 1995a

  28. Macintosh DL Xue J Özkaynak H Spengler JD and Ryan PB, A population-based exposure model for benzene, J Expos Anal Environ Epidemiol (1995b) 5(3): 375–403

    CAS  Google Scholar 

  29. Manne AS and Richels RG, The costs of stabilizing global CO2 emissions: a probabilistic analysis based on expert judgments, Energy J (1994) T15(1): 31–56

    Google Scholar 

  30. Matanoski G Selevan SG Akland C Bornschein RL Dockery D Edmonds L Greife A Mehlman M Shaw GM and Elliott E, Role of exposure databases in epidemiology, Arch Environ Health (1992) 47(6): 439–446

    Article  CAS  PubMed  Google Scholar 

  31. Morgan MG and Henrion M, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, New York 1990

    Book  Google Scholar 

  32. Morgan MG and Keith DW, Subjective judgments by climate experts, Environ Sci Technol (1995) 29: 468A–476A

    CAS  PubMed  Google Scholar 

  33. Morgan MG Morris SC, Henrion M Amaral DAL and Rish WR, Technical uncertainty in quantitative policy analysis — a sulfur air pollution example, Risk Anal (1984) 4(3): 201–216

    Article  Google Scholar 

  34. Nordhaus WD, Expert opinion on climatic change, Am Sci (1994) 82: 45–51

    Google Scholar 

  35. North DW Selker FK and Guardino T, Estimating the Value of Research: An Illustrative Calculation for Ingested Inorganic Arsenic. Decision Focus, Palo Alto, CA October 1992

    Google Scholar 

  36. NRC (National Research Council), Science and Judgment in Risk Assessment. National Academy Press, National Academy of Sciences, Washington, DC 1994

  37. Otway H, and von Winterfeldt D, Expert judgment in risk analysis and management: process, context, and pitfalls, Risk Anal (1992) 12(1): 83–93

    Article  CAS  PubMed  Google Scholar 

  38. Paté-Cornell ME, Uncertainties in risk analysis: six levels of treatment, Reliab Eng Syst Saf (1996) 54: 95–111

    Article  Google Scholar 

  39. Rinsky RA Smith AB Hornung R et al, Benzene and leukemia — an epidemiologic risk assessment, N Engl J Med (1987) 316(17): 1044

    Article  CAS  PubMed  Google Scholar 

  40. Robinson JP and Thomas J, Time Spent in Activities, Locations, Microenvironments: a California–National Comparison. Environmental Monitoring Systems Laboratory, US EPA, Las Vegas, NV 1991

    Google Scholar 

  41. Sexton K, Informed decisions about protecting and promoting public health: rationale for a national human exposure assessment survey, J Expos Anal Environ Epidemiol (1995) 5(3): 233–256

    CAS  Google Scholar 

  42. Siegel JE Graham JD and Stoto MA, Allocating resources among AIDS research strategies, Policy Sci (1990) 23: 1–23

    Article  Google Scholar 

  43. Sielken RL, Useful tools for evaluating and presenting more science in quantitative cancer risk assessments, Toxic Subst J (1989) 9: 353

    Google Scholar 

  44. Spedden SE, Judgments and perceptions of human risk from chemical carcinogens. ScD Thesis, Harvard University, Cambridge, MA (1992

  45. Spetzler CS and Staël von Holstein C-AS, Probability encoding in decision analysis, Manage Sci (1975) 22: 340–358

    Article  Google Scholar 

  46. Stewart PA Blair A Cubit DA et al, Estimating historical exposures to formaldehyde in a retrospective mortality study, Appl Ind Hyg (1986) 1: 34

    Article  CAS  Google Scholar 

  47. Taylor A Evans JS and Mckone TE, The value of animal test information in environmental control decisions, Risk Anal (1993) 13(4): 403–412

    Article  CAS  PubMed  Google Scholar 

  48. Thomas KW Pellizzari ED Clayton AC Perrit RL Dietz RND Goodrich RW Nelson WC and Wallace L, Temporal variability of benzene exposures for residents in several new jersey homes with attached garages or tobacco smoke, J Expos Anal Environ Epidemiol (1993) 3(1): 49–73

    CAS  Google Scholar 

  49. Thompson KM and Evans JS, The value of improved national exposure information for perchloroethylene: a case study for dry cleaners, Risk Anal (1997) 17(2): 253–271

    Article  Google Scholar 

  50. US EPA, Guiding Principles for Monte Carlo Analysis, Risk Assessment Forum, US Environmental Protection Agency, Washington, DC, EPA/630/R-97/001

  51. von Winterfelt D, and Edwards W, Decision Analysis and Behavioral Research. Cambridge University Press, Cambridge, England 1986

    Google Scholar 

  52. Wallace L, The Total Exposure Assessment Methodology (TEAM) study: an analysis of exposures, sources, and risks associated with four volatile organic chemicals, J Am Coll Toxicol (1989) 8(5): 883–895

    Article  Google Scholar 

  53. Wallace L, Environmental exposure to benzene: an update, Environ Health Perspect (1996) 104(suppl 6): 1129–1139

    CAS  PubMed  PubMed Central  Google Scholar 

  54. Wallace LA Pellizzari EA, Hartwell TD Whitmore R Zelon H Perritt R and Sheldon L, The California team study: breath concentrations and personal exposures to 26 volatile compounds in air and drinking water of 188 residents of Los Angeles, Antioch, and Pittsburgh, CA, Atmos Environ (1988) 22(10): 2141–2163

    Article  CAS  Google Scholar 

  55. Wallsten TS and Whitfield RG, Assessing the Risks to Young Children of Three Effects Associated with Elevated Blood Lead Levels. Argonne National Laboratory, Argonne, IL December 1986 ANL/AA-32

    Google Scholar 

  56. Winkler RL Wallsten TS Whitfield RG Richmond HM Hayes SR and Rosenbaum AS, An assessment of the risk of chronic lung injury attributable to long-term ozone exposure, Oper Res (1995) 43(1): 19–28

    Article  Google Scholar 

  57. Winn D, Lednar W, Williams T, and Van Ert M, The identification of occupational exposures by an industrial hygiene questionnaire. Presented to the American Industrial Hygiene Conference, New Orleans, May 1977. Occupational Health Studies Group, University of North Caroline School of Public Health (1977

  58. Wolff SK Hawkins NC, Kennedy SM and Graham JD, Selecting experimental data for use in quantitative risk assessment: an expert judgment approach, Toxicol Ind Health (1990) 6(2): 275–291

    Article  CAS  PubMed  Google Scholar 

  59. Wright GW and Ayton P (Eds.), Subjective Probability. John Wiley and Sons, New York, NY 1994

    Google Scholar 

Download references

Acknowledgements

This work would not have been possible without the experts who participated in this study. They have our esteem and thanks for their patience and attentiveness throughout the long, and often arduous, interviews. Barry Ryan of Emory University played a valuable role by serving as one of the pilot studies for the elicitation protocol. We are thankful to Paul Catalano, James Hammitt, and John Graham of the Harvard School of Public Health for their helpful comments on this study. This work was supported by US EPA Cooperative Agreement CR822038-03-1-3, HRSA, Bureau of Health Professions grant A03-AH01165-01, the Leslie Silverman Foundation, and the Harvard Center for Risk Analysis.

Author information

Authors and Affiliations

  1. Harvard School of Public Health, Boston, Massachusetts, USA

    KATHERINE D WALKER

  2. The University of Georgia, Athens, Georgia, USA

    JOHN S EVANS & DAVID MACINTOSH

Authors
  1. KATHERINE D WALKER
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. JOHN S EVANS
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. DAVID MACINTOSH
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to KATHERINE D WALKER.

Appendix

Appendix

Table A1: Heuristics in subjective judgments: some definitions

Rights and permissions

Reprints and permissions

About this article

Cite this article

WALKER, K., EVANS, J. & MACINTOSH, D. Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene. J Expo Sci Environ Epidemiol 11, 308–322 (2001). https://doi.org/10.1038/sj.jea.7500171

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/sj.jea.7500171

Keywords

This article is cited by

Search

Advanced search

Quick links