A tutorial on General Recognition Theory

Highlights

• General recognition theory provides a powerful mathematical framework for analyzing interactions between perceptual dimensions.

• Non-parametric statistical tools allow for analysis of dimensional interactions with minimal assumptions about the underlying perceptual and decisional representations.

• Parametric Gaussian models allow for powerful statistical analyses and compelling model visualizations.

Abstract

General Recognition Theory (GRT; e.g., Ashby and Townsend, 1986, inter alia) is a two-stage, multidimensional model of encoding and response selection. In this tutorial, we present the basic conceptual and mathematical structure of GRT and review the three notions of dimensional interaction defined in the GRT framework: perceptual independence, perceptual separability, and decisional separability. Experimental protocols and data closely linked to the GRT model are discussed, and two sets of empirical tests of dimensional interaction are presented. These test procedures are illustrated via functions the new R package mdsdt.

Introduction

Cognitive processing is fundamentally multidimensional. Even the simplest experimental stimuli must be defined with respect to multiple dimensions. Consider, for example, the auditory perception of a simple tone. Although the amplitude of the tone may be of primary interest to a researcher, a sinusoidal tone must also have frequency, duration, and phase. The perception of amplitude may depend, at least in part, on the frequency, duration, and/or phase of the tone. For example, a 100 Hz tone at a given amplitude will be perceived as substantially more quiet than a 1000 Hz tone with the same amplitude (Plack & Carlyon, 1995). Similarly, at very short times, a longer tone with a given amplitude will be perceived as louder (i.e., as having larger amplitude) than a shorter tone with the same amplitude (Plack & Carlyon, 1995). A full understanding of auditory perception must take this sort of interaction between acoustic dimensions into account. In general, a full understanding of how multiple dimensions are processed in any modality requires an account of the relationships between the dimensions.

General recognition theory (GRT, also known as multi-dimensional signal detection theory, or MDSDT) is a powerful framework for studying the relationships between dimensions in cognitive processing. It has been used to model perceptual interactions in vision (e.g.,  Kadlec and Hicks, 1998, Olzak and Wickens, 1997, Thomas, 2001b) and audition (e.g.,  Silbert, 2012, Silbert, 2014, Silbert, Townsend et al., 2009) as well as higher-level processes in memory (e.g.,  DeCarlo, 2003).

As useful as GRT is, a researcher interested in probing the relationships between dimensions in a particular domain may be intimidated by the mostly technical literature on GRT (Ashby and Townsend, 1986, Kadlec and Townsend, 1992a, Kadlec and Townsend, 1992b, Silbert, de Jong et al., 2009, Silbert and Thomas, 2013, Thomas, 1995, Thomas, 1999, Thomas, 2001a, Thomas, 2001b). This tutorial seeks to make the powerful tools offered by GRT readily available to the interested researcher. We begin with a brief overview of the structure of GRT. This is followed by a more technical discussion of three different types of dimensional interactions defined in the GRT framework. With these fundamentals in place, the tutorial then takes the reader carefully through multiple procedures for analyzing data and drawing GRT-based inferences based on the results of these analyses. Each procedure is illustrated using corresponding functions in the new R package mdsdt, making it fast and practical for the reader to apply these analyses to their own data.

GRT is a two-stage model of encoding and response selection. As such, it can be usefully thought of as a multidimensional extension of signal detection theory (SDT;  Green and Swets, 1966, Macmillan and Creelman, 2005). GRT, like SDT, models trial-by-trial variation in behavioral responses by assuming a two stage process of noisy encoding followed by deterministic response selection. Although, as noted above, GRT has been used to study non-perceptual processes, it has been applied most frequently to the study of perceptual interactions. We will focus here on perceptual encoding and subsequent response selection, noting here for clarity that this is not intended to imply that GRT is limited to this domain.

We consider noisy perception and deterministic response selection in turn.

GRT begins with the assumption that a substantial component of perception is stochastic. Although there are multiple possible sources for random perceptual variability (e.g., environmental and/or neural noise;  Ashby & Lee, 1993), the specific sources of perceptual variability are not (typically) modeled in GRT.

The basic idea of noisy perception is that from trial to trial, the perceptual effect of a given stimulus varies. Over time, this produces distributions of perceptual effects. With some additional structure (e.g., a mechanism for response selection), the statistical properties of these distributions generate predictions about observed data. This, in turn, allows the researcher to draw inferences about latent perceptual structures from observed data. Properties of these latent perceptual structures then license inferences about the presence or absence of perceptual dimensional interactions.

In the most general form of GRT, no specific assumptions are made about the properties of the stochastic component of perception. However, it is common to assume that noisy perception is usefully modeled with multivariate Gaussian distributions, analogous to the univariate Gaussian distributions of SDT. We discuss statistical analyses of GRT data for both cases below.

Noisy perception alone does not allow a perceptual model to make contact with data. We also need a mechanism for selecting a response given a noisy perceptual effect. In GRT, this mechanism typically takes the form of decision bounds, or curves that partition perceptual space into mutually exclusive response regions.

Response selection is deterministic in GRT. When a (random) perceptual effect occurs in a particular response region, the response associated with that region is always selected. Response probabilities are given by the joint properties of the (noisy) perceptual distributions and (deterministic) decision bounds.

More technically, the probability of emitting response $X$ to stimulus $Y$ is given by the integral of the perceptual distribution corresponding to stimulus $Y$ over the response region for response $X$.

In order to illustrate noisy perception and deterministic response selection, we provide a visual depiction of a 2×2 Gaussian GRT model. By ‘2×2’, we mean a model (and associated stimulus set) with four perceptual distributions comprising the factorial combination of two levels on each of two dimensions. For example, we may be interested in using GRT to analyze possible interactions between frequency and duration in the perception of sound. Silbert, Townsend et al. (2009) probed the perception of these dimensions in broadband noise in their first experiment. In this case, listeners simultaneously identified the frequency range and duration of the stimuli, and the stimulus set consisted of the factorial combination of low and high frequency levels (490–1490 Hz and 510–1510 Hz, respectively) and short and long duration levels (250 ms and 300 ms, respectively). Hence, the full set had four stimuli: low, short; low, long; high, short; and high, long.

Taking a top-down view, we can get a fairly comprehensive view of the Gaussian GRT model through plots of decision bounds and equal likelihood contours for each of the perceptual distributions.¹Fig. 1 illustrates a possible 2×2 model. In the large square panel, the plus-signs indicate the means of the perceptual distributions, while the circles and ellipses are equal likelihood contours, or sets of points on each perceptual distribution at the same height above the plane. The shape of the equal likelihood contours indicate presence or absence of correlation in each distribution, with circles indicating zero correlation and ellipses indicating non-zero correlation. The dimensions along which the perceptual distributions and decision bounds are defined are modeled perceptual dimensions corresponding to the physical dimensions of the stimuli.

When taking such a top-down view of a GRT model, it is important to keep in mind that random perceptual effects for a particular stimulus may occur inside or outside the equal likelihood contour, and that perceptual effects inside the contour are more likely than those outside (i.e., the perceptual distribution’s peak is inside the contour, at the mean of the distribution, and the height of the bell curve describing the distribution decreases with distance from the mean). A single equal likelihood contour provides a convenient visualization of a bivariate perceptual distribution, but it is a simplification.

The vertical and horizontal lines indicate decision bounds, and the labels indicate the response regions. In the smaller, rectangular panels below and to the left of the square panel, the marginal perceptual distributions are illustrated, with the solid lines corresponding to the closer distributions and the dashed lines corresponding to the more distant distributions (e.g., in the left panel, the solid lines illustrate the marginal low, short and low, long distributions, while the dashed lines illustrate the marginal high, short and high, long distributions).

As described above, the probability of a particular response to a given stimulus is given by the integral of the appropriate perceptual distribution in the appropriate response region. So, for example, the probability of a ‘low, long’ response to the high, long stimulus is the double integral of the top-right distribution over the top-left response region. Hence, the configuration of four perceptual distributions and two decision bounds produces 16 predicted response probabilities, one for each response region for each perceptual distribution.

This top-down view of a GRT model also provides a convenient visualization of dimensional interactions, which we consider in the following section.

As noted above, GRT provides powerful tools for analyzing perceptual interactions. The assumptions of GRT–noisy perception and deterministic response selection–provide the basis for three logically distinct ways in which dimensions may or may not interact in multidimensional perception. In this section, we review these notions of (lack of) interaction between dimensions defined in the GRT framework: perceptual independence, perceptual separability, and decisional separability. We emphasize that these three notions are theoretical constructs, and so are not directly observable. The goal in GRT-based analyses is to draw inferences about these unobservable constructs from properties of appropriately collected data. The primary aim of this tutorial is to illustrate for the reader how to do this.

Perceptual independence holds between two dimensions within a given perceptual distribution if, and only if, the perceptual effects on each dimension are stochastically independent within that perceptual distribution. If frequency and duration are perceptually independent in low, short stimuli, for example, perception of higher or lower frequency is not statistically associated with perception of shorter or longer duration when low, short stimuli are presented. If perceptual independence fails for frequency and duration in low, short stimuli, on the other hand, then there is such a statistical relationship when low, short stimuli are presented.

In Fig. 1, perceptual independence holds in the (circular) $A_1{B}_1$ (low, short) and $A_1{B}_2$ (low, long) distributions, while it fails in the (elliptical) $A_2{B}_1$ (high, short) and $A_2{B}_2$ (high, long) distributions.

Mathematically, let the joint density of perceptual effects for stimulus $A_i{B}_j$ be $f_{A_i{B}_j}(x,y)$, where $A_i$ indicates the $i$th level of the stimulus on the $x$ dimension and $B_j$ indicates the $j$th level on the $y$ dimension. In the example discussed above, $A$ indicates frequency and $B$ indicates duration, with $A_1$  = low frequency, $A_2$  = high frequency, $B_1$  = short duration, and $B_2$  = long duration. The marginal densities on each dimension are then:

$${\int }_{-\infty }^{\infty }{f}_{A_i{B}_j}(x,y)dy=g_{A_i{B}_j}\left(x\right)$$$${\int }_{-\infty }^{\infty }{f}_{A_i{B}_j}(x,y)dx=g_{A_i{B}_j}\left(y\right)\text{.}$$

Perceptual independence holds if, and only if Eq. (2) holds for all values of $x$ and $y$:

$$f_{A_i{B}_j}(x,y)=g_{A_i{B}_j}\left(x\right)g_{A_i{B}_j}\left(y\right)\text{.}$$

Because perceptual independence is a property of a single, unobserved perceptual distribution, it can, in principle, hold or fail for each perceptual distribution regardless of whether it holds or fails in any other.

Unlike perceptual independence, perceptual separability defines relationships across perceptual distributions. Perceptual separability holds on one dimension with respect to another dimension if the marginal perceptual distributions on the former do not vary across levels of the latter. So, for example, if the perception of frequency does not vary across levels of duration (i.e., if the perception of frequency is equivalent in short and long stimuli), we would say that frequency is perceptually separable from duration.

In Fig. 1, perceptual separability holds on the $x$ dimension with respect to the $y$ dimension (i.e., frequency is perceptually separable from duration). The $A_1{B}_1$ and $A_1{B}_2$ distributions are aligned vertically, as are the $A_2{B}_1$ and $A_2{B}_2$ distributions, and the marginal distributions illustrated in the bottom panel are clearly coincident across levels of duration ($B_1$ and $B_2$).

On the other hand, perceptual separability fails on the $y$ dimension with respect to the $x$ dimension (i.e., duration is not perceptually separable from frequency). The marginal perceptual distributions of perceptual effects on the $y$ dimension are closer together at the $A_1$ level (low frequency; solid lines in the left panel) than they are at the $A_2$ level (high frequency; dashed lines in the left panel), indicating that the perceptual salience of duration varies across levels of frequency ($A_1$ and $A_2$).

Mathematically, perceptual separability holds on the $x$ dimension with respect to the $y$ dimension (for $A$ with respect to $B$) if, and only if, Eq. (3) holds for all values of $x$ and $y$, while perceptual separability holds on the $y$ dimension with respect to the $x$ dimension (for $B$ with respect to $A$) if, and only if, Eq. (4) holds for all values of $x$ and $y$:

$$g_{A_i{B}_1}\left(x\right)=g_{A_i{B}_2}\left(x\right)i=1,2$$$$g_{A_1{B}_j}\left(y\right)=g_{A_2{B}_j}\left(y\right)j=1,2\text{.}$$

Eq. (3) states that the marginal perceptual effects at each level of the $x$ dimension (i.e., $A_1$ and $A_2$) are identically distributed at each level of the $y$ dimension (i.e., $B_1$ and $B_2$). Eq. (4) states that the marginal perceptual effects at each level of the $y$ dimension (i.e., $B_1$ and $B_2$) are identically distributed at each level of the $x$ dimension (i.e., $A_1$ and $A_2$).

Because perceptual separability concerns relationships between marginal perceptual distributions on a given dimension, it may hold or fail on either dimension regardless of whether or not perceptual independence holds in any of the distributions.

Finally, in the GRT framework, dimensions may interact with respect to response selection. Decisional separability holds on one dimension with respect to another if, and only if, the decision bound partitioning the former is parallel to the coordinate axis on the latter. Both decision bounds in Fig. 1 exhibit decisional separability.

If decisional separability holds, then decisions on one dimension do not depend in any way on another dimension. Complicating matters, there are a number of ways in which decisions on one dimension could depend on another dimension. A model could have linear bounds that are not parallel to any of the coordinate axes. In this case, decisions on one dimension would depend on multiple dimensions of the perceptual space. On the other hand, decisions on one dimension could depend on decisions on another dimension (e.g., responses concerning the frequency of a sound may depend on whether or not a sound has been labeled ‘short’ or ‘long’), in which case the model would have piecewise (linear) decision bounds (see, e.g.,  Silbert & Thomas, 2013).²

Another possibility is that responses are selected based on the relative likelihood of a given perceptual effect having been produced by each possible stimulus (i.e., by a Bayesian response rule). In this case, decisions are not directly determined by decision bounds. This type of model may be described as having implicit piecewise (quadratic or linear) decision bounds (see, e.g.,  Silbert & Thomas, 2014).

Recent theoretical work in the GRT framework has shown that failures of decisional separability are not, in general, identifiable (Silbert and Thomas, 2013, Thomas and Silbert, 2014).³ However, under the assumption that decisional separability holds, failures of perceptual independence and perceptual separability are identifiable. In addition, many failures of perceptual independence and/or perceptual separability are identifiable regardless of whether or not decisional separability holds. In particular, failures of perceptual separability that are caused by changes in the salience of one dimension across levels of the other are identifiable whether or not decisional separability holds. This type of failure of perceptual separability is illustrated in Fig. 1, wherein the distinction between short and long sounds is more salient at the high frequency level than at the low frequency level. Real-world examples of this kind of failure of perceptual separability has been found with speech stimuli, as well (Silbert, 2012, Silbert, 2014). For detailed discussion of these issues, see Silbert and Thomas (2013). In this tutorial, we will assume that decisional separability holds and focus on methods for analyzing perceptual interactions (or lack thereof).