Abstract
The effects of base rates and payoffs on the shapes of rating receiver operating characteristic curves are inconsistent with the basic assumptions of signal detection theory (SDT), in particular the notion of a shifting decision criterion. Mueller and Weidemann (2008) propose that these unexpected phenomena are not due to problems with the decision- criterion construct but are instead due to two compounded effects: instability of the decision criterion across trials, and even greater instability in the flanking criteria that determine which confidence rating will be reported. There are several problems with the authors’ decision-noise hypothesis. First, even if their hypothesis about decision noise were taken for granted, the key feature of the ratings data that rejects the SDT model would remain a mystery. Second, the same violations of SDT that are exhibited in the ratings paradigm are also exhibited in the yes-no detection task when response time is substituted for confidence as a basis for analysis. Finally, the decision-noise hypothesis predicts that sensitivity will increase when one source of this variation-the response on a previous trial-is controlled for. This prediction was consistently violated in both the yes-no and ratings conditions of Mueller and Weidemann’s experiment. In an Addendum, we respond to Weidemann and Mueller’s (2008) reply to this Comment.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Balakrishnan, J. D. (1998a). Measures and interpretations of vigilance performance: Evidence against the detection criterion. Human Factors, 40, 601–623.
Balakrishnan, J. D. (1998b). Some more sensitive measures of sensitivity and response bias. Psychological Methods, 3, 68–90.
Balakrishnan, J. D. (1999). Decision processes in discrimination: Fundamental misrepresentations of signal detection theory. Journal of Experimental Psychology: Human Perception & Performance, 25, 1189–1206.
Balakrishnan, J. D. (2006). Objective analysis of classification behavior: Applications to scaling. In E. N. Dzhafarov & H. Colonius (Eds.), Measurement and representation of sensations (pp. 159–201). Mahwah, NJ: Erlbaum.
Balakrishnan, J. D., MacDonald, J. A., & Kohen, H. S. (2003). Is the area measure a historical anomaly? Canadian Journal of Experimental Psychology, 57, 238–256.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.
Katz, L. (1970). A comparison of Type II operating characteristics derived from confidence ratings and from latencies. Perception & Psychophysics, 8, 65–68.
Kornbrot, D. E. (2006). Signal detection theory, the approach of choice: Model-based and distribution-free measures and evaluation. Perception & Psychophysics, 68, 393–414.
Link, S. W. (1992). The wave theory of difference and similarity. Hillsdale, NJ: Erlbaum.
Link, S. W., & Heath, R. A. (1975). A sequential theory of psychological discrimination. Psychometrika, 40, 77–105.
Luce, R. D. (1986). Response times: Their role in inferring elementary mental organization. New York: Oxford University Press.
Mueller, S. T., & Weidemann, C. T. (2008). Decision noise: An explanation for observed violations of signal detection theory. Psychonomic Bulletin & Review, 15, 465–494.
Parducci, A., & Sandusky, A. (1965). Distribution and sequence effects in judgment. Journal of Experimental Psychology, 69, 450–459.
Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111, 333–367.
Treisman, M. (2002). Is signal detection theory fundamentally flawed? A response to Balakrishnan (1998a, 1998b, 1999). Psychonomic Bulletin & Review, 9, 845–857.
Treisman, M., & Williams, T. C. (1984). A theory of criterion setting with an application to sequential dependencies. Psychological Review, 91, 68–111.
Van Zandt, T. (2000). ROC curves and confidence judgments in recognition memory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 26, 582–600.
Weidemann, C. T., & Mueller, S. T. (2008). Decision noise may mask criterion shifts: Reply to Balakrishnan and MacDonald (2008). Psychonomic Bulletin & Review, 15, 1031–1034.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Balakrishnan, J.D., Macdonald, J.A. Decision criteria do not shift: Commentary on Mueller and Weidemann (2008). Psychonomic Bulletin & Review 15, 1022–1030 (2008). https://doi.org/10.3758/PBR.15.5.1022
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/PBR.15.5.1022