Abstract
The age-old question of the generalizability of the results of experiments that are conducted in artificial laboratory settings to more realistic inferential and decision-making situations is considered in this paper. Conservatism in probability revision provides an example of a result that (1) has received wide attention, including attention in terms of implications for real-world decision making, on the basis of experiments conducted in artificial settings and (2) is now apparently thought by many to be highly situational and not at all a ubiquitous phenomenon, in which case its implications for real-world decision making are not as extensive as originally claimed. In this paper we consider the questions of generalizations from the laboratory to the real world in some detail, both within the context of the experiments regarding conservatism and within a more general context. In addition, we discuss some of the difficulties inherent in experimentation in realistic settings, suggest possible procedures for avoiding or at least alleviating such difficulties, and make a plea for more realistic experiments.
On leave and visiting the International Institute for Applied Systems Analysis, Laxenburg, Austria, from June, 1973 to January, 1974 and the Department of Statistics, Stanford University, from February, 1974 to August, 1974
Supported in part by the National Science Foundation under Grant GA-31735.
Much of the material in this paper appeared in Winkler and Murphy (1973b). We are grateful to Academic Press, Inc., for their permission to include that material here.
The National Center for Atmospheric Research is sponsored by the National Science Foundation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Bibliography
Bartos, J.A., ‘The Assessment of Probability Distributions for Future Security Prices’. Indiana Univ., unpublished doctoral dissertation, Bloomington. 1969.
Brown, R. V., ‘Do Managers Find Decision Theory Useful?’. Harvard Business Review 48 (1970), 78–89.
Brown, R.V., Kahr, A., and Peterson, C.R., Decision Analysis for the Manager, Holt, Rinehart and Winston. New York. 1974.
Chinnis, J.O., and Peterson, C.R., ‘Inference About n Nonstationary Process’, Journal of Experimental Psychology 77 (1968), 620–625.
Chinnis, J.O., and Peterson, C.R., ‘Nonstationary Processes and Conservative Inference’, Journal of Experimental Psychology 84 (1970), 248–251.
Domas, P.A., and Peterson, C.R., ‘Probabilistic Information Processing Systems: Evaluation with Conditionally Dependent Data’, Organizational Behavior and Human Performance 7 (1972), 77–85.
Du Charme, W.M, ‘A Review and Analysis of the Phenomenon of Conservatism in Human Inference’, Rice Univ., unpublished manuscript. Houston. 1969.
Du Charme, W.M., and Peterson, C.R., ‘Intuitive Inference About Normally Distributed Populations’, Journal of Experimental Psychology 78 (1968), 269–275.
Du Charme, W.M., and Peterson, C.R., ‘Proportion Estimation as a Function of Proportion and Sample Size’, Journal of Experimental Psychology 81 (1969), 536–541.
Edwards,W., ‘Probabilistic Information Processing in Command and Control Systems’, Univ. of Michigan. Institute of Science and Technology, Report No. 3780-12-T, Ann Arbor. 1963.
Edwards, W.,‘Nonconservative Probabilistic Information Processing Systems’, Univ. of Michigan. Institute of Science and Technology, Report No. 5893-22-F. Ann Arbor. 1966.
Edwards, W., Conservatism in Human Information Processing, in B. Kleinmuntz (ed.), Formal Representation of Human Judgment, Wiley, New York, 1968.
Edwards, W., and Phillips, L.D., ‘Man as Transducer for Probabilities in Bayesian Command and Control Systems’, in M. W. Shelly and G. L. Bryan (eds,). Humm Judgments and Optimally, Wiley, New York, 1964.
Edwards, W., Phillips, L.D., Hays, W.L., and Goodman, B.C., ‘Probabilistic Information Processing Systems: Design and Evaluation’. IEEE Transactions on Systems Science and Cybernetics. SSC-4 (1968), 248–265.
Fried, L.S., and Peterson, C.R., ‘Information Seeking: Optimal Versus Fixed Stopping’. Journal of Experimental Psychology 80 (1969), 525–529.
Goodman, B.C., ‘Action Selection and Likelihood Ratio Estimation by Individuals and Groups’. Organizational Behavior and Human Performance 7 (1972), 121–141.
Gustafson, D.H., ‘Evaluation of Probabilistic Information Processing in Medical Decision Making’. Organizational Behavior and Human Performance 4 (1969), 20–34.
Howard, R.A., Matheson, J.E., and North, D.W., ‘The Decision to Seed Hurricanes’. Science 176 (1972), 1191–1202.
Kelly, C.W., and Peterson, C.R, ‘Probability Estimates and Probabilistic Procedures in Current-Intelligence Analysis Report on Phase I’. IBM. unpublished manuscript. Gaithersburg. Md., 1971.
Lichtenstein, S., and Feeney, G.J., ‘The Importance of the Data-Generating Model in Probability Estimation’, Organizational Behavior and Human Performance 3 (1968), 62–67.
Moskowitz, H., ‘Conservatism in Group Information Processing Behavior Under Varying Management Information Systems’. Purdue Univ., unpublished manuscript, Lafayette, Ind., 1971.
Murphy, A.H., and Winkler, R.L., ‘Scoring Rules in Probability Assessment and Evaluation’. Acta Psychologica 34 (1970), 273–286.
Murphy, A.H., and Winkler, R.L., ‘Subjective Probability Forecasting Experiments in Meteorology. Some Preliminary Results’. Bulletin of the American Meteorological Society 55 (1974). in press.
Murphy, A.H., Snapper, K.J., and Peterson, C.R., On the Process of Subjective Probability Forecasting’, National Center for Atmospheric Research, and Univ. of Michigan, unpublished manuscript, Boulder, Colorado and Ann Arbor, Michigan, 1972.
Peterson, C.R., and Beach, L.R., ‘Man as an Intuitive Statistician. ’Psychological Bulletin 68 (1967), 29–46.
Peterson, C.R., Schneider, R.J., and Miller, A.J., ‘Sample Size and the Revision of Subjective Probabilities’. Journal of Experimental Psychology 69 (1965), 522–527.
Peterson, C.R., Snapper, K.J., and Murphy, A.H., ‘Credible Interval Temperature Forecasts’. Bulletin of the American Meteorological Society 53 (1972), 966–970.
Phillips, L.D., ‘The ‘True Probability’ Problem’, Acta Psychologica 34 (1970), 254–264.
Phillips, L.D., Hays, W.L., and Edwards, W., ‘Conservatism in Complex Probabilistic Inference’. IEEE Transactions on Human Factors In Electronics HFE-7 (1966), 7–18.
Pitz, G.F., ‘On the Processing of Information: Probabilistic and Otherwise’, Acta Psychologica 34 (1970), 201–213.
Savage, L.J., ‘Elkitation of Personal Probabilities and Expectations’. Journal of the American Statistical Association 66 (1971), 783–801.
Schum, D.A., ‘Inferences on the Basis of Conditionally Nonindepcndent Data’. Journal of Experimental Psychology 72 (1966). 401–409
Schum, D.A., ‘Behavioral Decision Theory and Man-Machine Systems’, in K. De Greene (ed.), Systems Psychology. McGraw-Hill, New York, 1970.
Schum, D.A., Du Charme, W.M., and De Pitts, K.E., ‘Research on Human Multistage Probabilistic Inference Processes’, Organizational Behavior and Human Performance 10 (1973), 318–348,
Schum, D.A., Southard, J.F., and Wombolt, L.F., ‘Aided Human Processing of Inconclusive Evidence in Diagnostic Systems: A Summary of Experimental Evaluations’. Wright-Patterson Air Force Base. Ohio. Aerospace Medical Laboratory, AMRI.-TR-69-11, 1969
Slovic, P., ‘From Shakespeare to Simon: Speculations — and Some Evidence About Man’s Ability to Process Information’. Oregon Research Institute Research Monograph 12, Eugene. Oregon. 1972.
Slovic, P., and Lichtenstein, S., ‘Comparison of Baycsian and Regression Approaches to the Study of Information Processing in Judgment’. Organizational Behavior and Human Performance 6 (1971). 649–744.
Snapper, K.J., and Fryback, D.G., ‘Inferences Based on Unreliable Reports’. Journal of Experimental Psychology 87 (1971), 401–401
Staël von Holstein, C.A.S., Assessment and Evaluation of Subtecthe Probability Distributions, Stockholm School of Economics, Economic Research Institute. Stockholm, 1970.
Wallsten, T.S., ‘Subjects’ Probability Estimates and Subjectively Expected Utility Theory: Their Relationship and Some Limitations’. Univ. of North Carolina, unpublished manuscript. Chapel Hill. 1970.
Wheeler, G., and Beach, L.R., ‘Subjective Sampling distributions and Conservatism’, Organizational Behaiior and Human Performance 3 (1968), 36–46.
Winkler, R. L., ‘The Quantification of Judgment: Some Methodological Suggestions’. Journal of the American Statistical Association 62 (1967). 1109–1120.
Winkler, R.L., ‘Probabilistic Prediction: Some Experimental Results’. Journal of the American Statistical Association 66 (1971), 675–685.
Winkler, R.L., and Murphy, A.H., ‘Evaluation of Subjective Precipitation Probability Forecasts’, Proceedings of the First National Conference on Statistical Meteorology. American Meteorological Society, Boston. 1968. 133–141.
Winkler, R.L., and Murphy, A.H., ‘Information Aggregation in Probabilistic Prediction’, IEEE Transactions on Systems, Man. and Cybernetics SMC-3 (1973a), 154–160.
Winkler, R.L., and Murphy, A.H., ‘Experiments in the Laboratory and the Real World’. Organizational Behavior and Human Performance 10 (1973b). 252–270.
Youssef, Z.I., and Peterson, C.R., ‘Intuitive Cascaded Inferences’. Organizational Behavior and Human Performance 10 (1973), 349–358.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1974 D. Reidel Publishing Company, Dordrecht-Holland
About this chapter
Cite this chapter
Winkler, R.L., Murphy, A.H. (1974). On the Generalizability of Experimental Results. In: Staël Von Holstein, CA.S. (eds) The Concept of Probability in Psychological Experiments. Theory and Decision Library, vol 8. Springer, Dordrecht. https://doi.org/10.1007/978-94-010-2288-0_6
Download citation
DOI: https://doi.org/10.1007/978-94-010-2288-0_6
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-2290-3
Online ISBN: 978-94-010-2288-0
eBook Packages: Springer Book Archive