Context effects on probability estimates

We found significant context effect on probability estimates when reward probability was 50% (p < 0.05, nonparametric bootstrap test) but not at 10% and 90% (p > 0.05, nonparametric bootstrap test). Subjects gave larger estimates when the 50% reward stimulus was experienced with a 10% reward stimulus than with a 90% reward stimulus (Fig 2A, the two bars in the center of the graph: the bar on the left indicates probability estimate of the 50% reward stimulus [S] when the other stimulus [OS] in the context was 10%; the bar on the right indicates 50% reward probability estimate when OS in the context was 90%). Looking at estimates of individual subjects, 27 out of 34 subjects’ estimates were larger in the [10%, 50%] context than in the [50%, 90%] context (Fig 2A, note the direction of tilt of the black lines which represent each single subject’s mean probability estimates), showing that most subjects gave larger estimates to the 50% reward stimulus when it was paired with a 10% reward stimulus than with a 90% reward stimulus. Importantly, we observed this effect despite the fact that frequency of reward the subjects experienced was the same between different contexts (two bars in the center of Fig 2B). This indicates that, for the 50% reward stimuli, context effect on probability estimates was not driven by their own reward history. To visualize context effect better for each reward probability, we plot the average difference in probability estimates between contexts (Fig 2C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The effects of context on probability estimates.

Experiment 1: (A) Subjects’ estimates of probability of reward associated with different stimuli. Stimuli carrying the same reward probability are plotted together. The vertical axis represents interval estimates: 1:[0%–5%], 2:[5%–20%], 3:[20%–35%], 4:[35%–50%], 5:[50%–65%], 6:[65%–80%], 7:[80%–95%], 8:[95%–100%]. The bars represent the mean probability estimates (across subjects), and the tilted black lines represent individual subjects’ data. Error bars represent ±1 SEM. (B) Frequency of reward the subjects experienced during the experiment. Conventions are the same as in Fig 2A. (C) Context-induced difference in probability estimates. For each reward probability—10%, 50%, 90%—the mean difference (across subjects) in probability estimates— Δ_10%, Δ_50%, Δ_90%—between contexts is plotted. In the equations that define Δ_10%, Δ_50%, and Δ_90% shown in the graph, conventions of the color codes for probability estimates () are the same as in Fig 2A. For example, Δ_50% is computed by subtracting (in blue), which represents probability estimate of the 50% reward stimulus in the [50%, 90%] context, from (in red), which represents probability estimate of the 50% reward stimulus in the [10%, 50%] context. We obtained Δ from each subject based on his or her mean probability estimates (across trials) and computed the mean of Δ across subjects. Error bars represent 95% bootstrap confidence interval of the mean Δ (across subjects). (D) Comparing the size of context effect between different reward probabilities. Bars indicate the mean difference in context effect (across subjects) for stimuli associated with different probabilities. For example, Δ_50%−Δ_10% compares context effect between 50% reward and 10% reward. Error bars represent 95% bootstrap confidence interval of the mean differences in context effect (across subjects). Data underlying these graphs can be found in https://osf.io/48j7m. OS, the other stimulus present in the same context as the stimulus of interest; S, stimulus of interest.

https://doi.org/10.1371/journal.pbio.3000634.g002

We also compared the size of context effect between different probabilities and found that the size of context effect on 50% probability estimate was significantly larger than either 10% or 90% (Fig 2D). But for 10% and 90%, the difference was not significant. These results were based on trials in the second-half of the functional magnetic resonance imaging (fMRI) session (Fig 2C and 2D). However, the results remained the same regardless of whether we used data from the entire session, the first-half of the session, or the second-half of the session. It is important to note that, even though the effect of context was not statistically significant at 10% and 90%, we did observe a trend that was consistent with the effect on 50%: probability estimate of a stimulus was inversely related to the probability of reward associated with the other stimulus present in the context. At 10% reward, subjects tended to give smaller estimates when the other stimulus carried a 90% than 50% reward (18 out of 34 subjects). At 90% reward, subjects tended to give larger estimates when the other stimulus carried a 10% than 50% reward (23 out of 34 subjects). If we analyzed just the second-half of the data, 20 and 18 subjects showed these results for 10% and 90%, respectively.

At the individual-subject level, we also found evidence for the context effect described above: individual subjects’ probability estimates of a stimulus (S) tended to be affected by both the actual reward frequency of S (f_S) and the other stimulus (f_OS) she or he experienced in the same context. For each reward probability, we regressed the mean probability estimates of each subject (across trials) against f_S and f_OS. For example, for 50% reward, each subject contributed two data points—his or her mean probability estimate of the 50% stimulus in the [10%, 50%] context and [50%, 90%] context . The values of the f_S and f_OS regressors for correspond, respectively, to the reward frequency of the 50% and 10% stimuli; the values of the f_S and f_OS regressors for correspond, respectively, to the reward frequency of the 50% and 90% stimuli. We found that in all three reward probabilities (10%, 50%, 90%), individual subjects’ probability estimates were positively correlated with f_S (p < 0.05). Consistent with the group average data for 50% reward, individual subjects’ probability estimates were negatively correlated with f_OS (p < 0.05). For 10%, the impact of f_OS (negative correlation) was also significant (p < 0.05); for 90%, the impact of f_OS was not significant (p = 0.1893).

Context-dependent probability estimates predict choice behavior

We also found that context effect on probability estimates further predicted subjects’ choice behavior. In a lottery decision task (a behavioral session) that followed the probability estimation task (fMRI session), subjects in each trial faced two stimuli they encountered in the fMRI session and were asked to choose the one they preferred so that one of their choices would be selected at random at the end of the experiment and realized to determine his or her payoff. Each stimulus they faced carried the same reward probability as in the fMRI session, and the reward magnitude was fixed across all stimuli so that subjects should pick the one she or he believed to have the larger probability of reward. Each pair was presented multiple times so that we could calculate choice probability (the fraction of trials in which subjects chose one lottery over the other) for each subject. We analyzed choice probability on the three pairs each consisting of stimuli carrying the same probability of reward but were experienced under different contexts. Subjects’ choice behavior was predicted by their context-dependent probability estimates: they preferred the 50% reward stimulus with larger probability estimate (in the [10%, 50%] context) to the one with smaller estimate (in the [50%, 90%] context) (middle bar in Fig 3A): choice probability was significantly larger than 0.5 (p < 0.05, nonparametric bootstrap test). In contrast, for the 10% and 90% pairs, subjects were indifferent between the stimuli carrying the same probability of reward (left and right bars in Fig 3A): choice probability was not significantly different from 0.5 (p > 0.05, nonparametric bootstrap test). However, it is worth noting that the direction of choice in those pairs was still qualitatively consistent with the probability estimates (Fig 2A): subjects tended to choose more often the stimuli with larger probability estimates.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Experiment 1: Choice probability in the post-fMRI lottery decision task was predicted by context-dependent probability estimates in the fMRI session.

(A) Three pairs of options, each representing a choice between two stimuli carrying the same reward probability but experienced in different contexts, are shown. For each pair, we plot the choice probability of the option highlighted in red. Each data point represents choice probability of a single subject. The bar indicates the mean choice probability averaged across all subjects. Error bars represent ±1 SEM. For each pair, we tested whether the mean choice probability is different from 0.5. The red star symbol indicates significant difference in choice probability from 0.5 based on 95% bootstrap confidence interval. (B–D) Relation between choice probability and probability estimate. For each pair of options described above, choice probability is plotted against the difference in probability estimates subjects provided in the fMRI task. Each data point represents a single subject. (B) 10% reward pair. (C) 50% reward pair. (D) 90% reward pair. Data underlying these graphs can be found in https://osf.io/48j7m.

https://doi.org/10.1371/journal.pbio.3000634.g003

The impact of context-dependent probability estimates on choice probability was not only observed at the group level—we found that individual differences in probability estimates also predicted choice probability (Fig 3B, 3C and 3D). For the 50% reward pairs, the Pearson correlation between individual subjects’ choice probability and probability estimate was 0.4113 (p = 0.0174). For 10% and 90% pairs, the correlation was also significant (10%: r = 0.5113, p = 0.0024; 90%: r = 0.4629, p = 0.0067). This indicates that, for 10% reward and 90% reward, even though the effect of context on probability estimate and on choice probability were not significant at the group level, individual subjects’ probability estimate and choice probability are significantly correlated. In summary, these results demonstrate that the effect of context was not only confined to subjects’ probability estimates—it further impacted their preferences for the stimuli.

Dynamics of probability estimates

We examined trial-by-trial probability estimates and found that context effect on 50% reward emerged relatively early and persisted throughout the experiment (Fig 4B). Another aspect of the dynamics worth mentioning is that subjects’ probability estimates clearly deviated from the actual frequency of reward that they experienced (the red and blue step-function traces in Fig 4B). Compared with frequency of reward, the subjects clearly underestimated the 50% probability of reward (blue curve) when it was experienced with a 90% reward stimulus in the [50%, 90%] context but overestimated 50% when it was experienced with a 10% reward stimulus in the [10%, 50%] context. In addition, the context effect observed here was not due to reward-frequency bias the subjects experienced in the pre-fMRI session (S2 Fig). By contrast, we did not observe these patterns on the 10% reward (Fig 4A) and 90% reward stimuli (Fig 4C). Qualitatively, however, we did see that subjects gave larger estimates to the 10% reward stimulus when it was experienced with a 50% reward stimulus (red in Fig 4A) than the 10% stimulus experienced with a 90% reward stimulus (blue in Fig 4A). For 90% reward, even though not statistically significant, the subjects gave larger estimates to the 90% reward stimulus in the [10%, 90%] context (red in Fig 4C) than in the [50%, 90%] context (blue in Fig 4C). For response time data, see S1 Fig.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Experiment 1: Comparison of trial-by-trial probability estimates between contexts showing how differences in probability estimates between contexts developed during the experiment.

Data includes the pre-fMRI practice session and the fMRI session. (A–C) The scale on the vertical axis indicates interval estimate (from 1 to 8 with 1 representing the smallest probability interval [0%–5%] and 8 representing the largest probability interval [95%–100%]). (A) Comparison of the 10% reward stimuli between contexts. Each data point represents the mean probability estimate of a single trial across subjects. Red: 10% stimulus in the [10%, 50%] context. Blue: 10% stimulus is in the [10%, 90%] context. The colored step function traces represent the corresponding mean frequency of reward the subjects experienced. (B) Comparison of the 50% reward stimuli between contexts. Red: 50% stimulus in the [10%, 50%] context. Blue: 50% stimulus in the [50%, 90%] context. (C) Comparison of the 90% reward stimuli between contexts. Red: 90% stimulus in the [10%, 90%] context. Blue: 90% stimulus in the [50%, 90%] context. The bumps on trial number 21 and 36, seen in Fig 4A and 4C, reflect the beginning of a new block of trials. Data underlying these graphs can be found in https://osf.io/48j7m. fMRI, functional magnetic resonance imaging.

https://doi.org/10.1371/journal.pbio.3000634.g004

A behavioral control experiment replicated context effects on probability estimates

We conducted an additional behavioral control experiment (Experiment 2; N = 20 subjects) to address a potential confound on the assignment of buttons to indicate probability estimates. In Experiment 1, because the boundary between the two buttons for small probabilities was set at 5% and at 95% for large probabilities instead of 10% and 90%, respectively (see the “Task” subsection in “Materials and methods”), it is possible that the nonsignificant context effect at 10% and 90% was because of a lack of sensitivity on the interval estimates to detect differences in probability estimates at 10% and 90% between different contexts. In the control experiment, a total of 10 buttons were used, each representing a 10% interval and therefore addressed the above concern (see subsection “Behavioral control experiment (Experiment 2)” in “Materials and methods”). We replicated the results observed in Experiment 1: For the 50% reward stimuli, subjects gave larger estimates when 50% was paired with 10% than with 90% (Fig 5B). For the 10% and 90% stimuli, no significant context effect was found (Fig 5A and 5C). Furthermore, as in Experiment 1, context effect and lack thereof on probability estimates predicted subjects’ choice behavior in the lottery decision task that followed the probability estimation task (Fig 5D). The subjects on average chose more often the stimulus with larger probability estimate when its reward probability was 50% (p < 0.05, nonparametric bootstrap test). By contrast, when stimulus reward probability was 10% and 90%, mean choice probability (across subjects) was not significantly different from 0.5 (p > 0.05, nonparametric bootstrap test), indicating that the subjects were indifferent between stimuli carrying the same reward probability but experienced in different contexts.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. A behavioral control experiment (Experiment 2) examining whether context effect and lack thereof on probability estimates found in Experiment 1 was because of the design of the interval-to-button mapping.

Conventions are the same as in Figs 3 and 4. The design of the experiment was identical to Experiment 1 except that there were 10—as opposed to 8—key buttons, each representing a unique interval of probability [0, 1] for the subjects to indicate probability estimates. (A) Probability estimates of the 10% reward stimuli. (B) Probability estimates of the 50% reward stimuli. (C) Probability estimates of the 90% reward stimuli. The scale on the vertical axis indicates interval estimate (from 1 to 10 with 1 representing the smallest probability interval [0%–10%] and 10 representing the largest probability interval [90%–100%]). (D) Choice probability in the lottery decision task following the probability estimation task. Error bars represent ±1 SEM. The red star symbol indicates significant difference in choice probability from 0.5 based on 95% bootstrap confidence interval. Data underlying these graphs can be found in https://osf.io/48j7m.

https://doi.org/10.1371/journal.pbio.3000634.g005

Context effects on probability estimates are driven by reference and uncertainty dependencies

To systematically explain context effects, we developed and tested a mathematical model for context-dependent probability estimation (see subsection “Modeling context-dependent computations for probability estimation: the Uncertainty- and Reference-Dependent (URD) model” in “Materials and methods” and Fig 6A for an illustration). Briefly, the model proposes that when computing probability estimate of a stimulus , the decision maker uses the frequency of reward (f_S) she or he experienced but is biased by the overall frequency of reward (f_overall) associated with a context. Here f_overall is the frequency of reward averaged across different stimuli in a context. The bias arises from treating f_overall as a reference point and comparing f_S with it (f_S−f_overall). For example, for a stimulus carrying 50% reward, its f_S would be greater than f_overall in the [10%, 50%] context in which the other stimulus carries a 10% reward and hence f_S−f_overall>0, but would be smaller than f_overall in the [50%, 90%] context in which the other stimulus carries a 90% reward and hence f_S−f_overall<0. The model therefore predicts that subjects would give larger probability estimates to the stimulus carrying 50% reward in the [10%, 50%] context than in the [50%, 90%] context because of the reference-dependent difference term (f_S−f_overall). However, simply having the reference-dependent term alone cannot explain why we did not find significant context effect at 10% and 90% reward. We reasoned that this is because of the level of uncertainty regarding which potential outcome (receiving a reward or no reward) would occur: When the level of uncertainty is larger (e.g., at 50% reward), the impact of (f_S−f_overall) on probability estimates is greater than when the level of uncertainty is smaller (e.g., at 10% or 90% reward). This is modeled by having (f_S−f_overall) weighted by the estimated level of uncertainty, which can be either the estimated variance or standard deviation of potential outcomes associated with the stimulus based on what the decision maker experienced in the recent past. Below is a version of the model that uses to weight the impact of (f_S−f_overall) (1)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Testing the URD model for context-dependent probability estimation.

(A) Illustration of the URD model. (B) Regression analysis. In three separate multiple regression analyses each examining a particular reward probability, we estimated the weight subjects assigned to frequency of reward associated with a stimulus (f_S) and the overall frequency of reward associated with the context (f_overall) and plot the mean regression coefficients across all subjects. Error bars represent ±1 SEM. The star symbol indicates statistical significance (testing the mean regression coefficient against 0 at p < 0.05). (C) We compare with for each reward probability separately. The vertical axis represents Δ, where . We computed Δ for each subject separately and plot the mean Δ across subjects. Error bars represent 95% bootstrap confidence interval of the mean Δ. Data underlying these graphs can be found in https://osf.io/48j7m. URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g006

Eq (1) can be rewritten as , which makes it easier to see several predictions of the model on the weights assigned to f_S and f_overall (see subsection “Modeling context-dependent computations for probability estimation: the Uncertainty- and Reference-Dependent (URD) model” in “Materials and methods”). These predictions were confirmed by multiple regression analysis on individual subjects’ probability estimates using f_S and f_overall as regressors (Fig 6B and 6C). First, the regression analysis confirmed that the regression coefficient of f_S was positive and significantly different from 0 for all three reward probabilities (Fig 6B). Second, the regression analysis confirmed that the coefficient of f_S associated with 50% reward was greater than that associated with 10% and 90% reward (Fig 6B). Third, the regression analysis confirmed that the coefficient of f_overall associated with 50% reward was negative and its magnitude was larger than that associated with the 10% reward and 90% reward stimuli (Fig 6B). Finally, the model makes the strong prediction that the regression coefficient of f_S () would be the same as , where is the regression coefficient of f_overall. This is what we found (Fig 6C). We tested Δ against 0, where and found that Δ was not significantly different from 0 in all three reward probabilities (p > 0.05, nonparametric bootstrap test). Together, these results provide support to the URD model and highlight that context effects on probability estimates are driven by the interaction of reference-dependent computation and estimated uncertainty associated with potential outcomes.

Model comparison

We further compared the URD model with normalization models that have been useful in describing context-dependent valuations [32,39,40]. Here, we consider two classes of normalization models, the divisive normalization (DN) model and the range normalization (RN) model. DN computes probability estimate of a stimulus by having its frequency of reward divided by the sum of the frequency of reward associated with the two stimuli in a context or divided by the average reward frequency of a context (Eqs 9 and 10 in “Materials and methods”). RN uses the difference between the maximum and the minimum stimulus reward frequency in a context in the denominator to compute probability estimates (Eq 11 in “Materials and methods”). Both model classes predict large context effect on probability estimates at 50% reward and nonsignificant effect at 10%, which are consistent with what we observed in this data set. However, both models also predict large context effect at 90% reward, which we did not observe in subjects’ probability estimates. For example, in the simplest form of divisive normalization without free parameters [32]——the normalized value of 10% reward in the [10%, 50%] context is 0.1 ÷ (0.1 + 0.5) = 0.167 (suppose that f_s = 0.1,f_OS = 0.5) and in the [10%, 90%] context is 0.1 ÷ (0.1 + 0.9) = 0.1 (suppose that f_s = 0.1,f_OS = 0.9). The difference is 0.067, a small difference. By contrast, for 90% reward, under [10%, 90%] the normalized value is 0.9 ÷ (0.9 + 0.1) = 0.9 (suppose that f_s = 0.9,f_OS = 0.1); under [50%, 90%] the normalized value is 0.9 ÷ (0.9 + 0.5) = 0.643 (suppose that f_s = 0.9,f_OS = 0.5). This indicates that DN would in principle predict large context effect at 90% reward but small effect at 10% reward. It is also worth mentioning that, in contrast to the URD model, both DN and RN models do not use information about reward variance or standard deviation when computing probability estimates.

A total of 9 different models—three versions of our model (URD), four versions of DN, and two versions of RN—were fitted to the trial-by-trial mean probability estimates (across subjects) using method of maximum likelihood (see subsection “Model fitting and model comparison” in “Materials and methods”). To compare these models, we computed Bayesian information criterion (BIC) and for each model estimated its 95% confidence interval using nonparametric bootstrap method [41] (Fig 7A). We did not include the BIC of RN-γ because of poor convergence in maximum likelihood estimation. The best models are the versions that considered probability distortion—overestimation of small probability and underestimation of large probability (see Eq 6 in “Materials and methods”; model labels including γ in Fig 7A). Among them, there was no significant difference between our model (URD-γ,URD-γ-λ) and DN models (DN-1-γ,DN-2-γ,DN-2). However, qualitatively, compared with our model fits (Fig 7B), the normalization models (Fig 7C) tended not to capture the data as well across different probabilities and contexts, especially at 90% reward.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Model fitting results: Group average data.

(A) Model comparison. We show BIC value of 8 models. Smaller BIC values indicate better models. Error bars represent 95% bootstrap confidence interval. (B) Model fitting results of the URD models (URD-γ,URD-γ-λ). (C) Model fitting results of the divisive normalization models (DN-1-γ,DN-2-γ) and range normalization model (RN). Each data point (black) in panels B and C represents the mean probability estimate (across all subjects) of a particular trial order. For example, in the [10%, 50%] context, when S is the 50% reward stimulus, OS would be the 10% reward stimulus; when S is the 10% reward stimulus, OS would be the 50% reward stimulus. Data underlying these graphs can be found in https://osf.io/48j7m. BIC, Bayesian information criterion; DN, divisive normalization; OS, the other stimulus present in the same context as the stimulus of interest; RN, range normalization; S, stimulus of interest; URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g007

We also fitted the same models at the individual-subject level (Fig 8A and 8B). The overall pattern of the model fits was very similar to the results from fitting the group average data. Finally, we implemented all the models considered above in the Rescorla–Wagner reinforcement-learning model framework and fitted them (S1 Text). We fit these models at the individual-subject level and plot the group average model fits (Fig 8C and 8D). The Rescorla–Wagner version of these models captured the dynamics of average probability estimates better compared with their non-Rescorla–Wagner counterparts (Fig 8A and 8B). Further, under the Rescorla–Wagner model framework, it is no longer obvious that our model described the data better than normalization models at 50% and 90% reward. However, these differences could be because of an increase in the number of free parameters in the Rescorla–Wagner versions of these models as a result of having to incorporate learning-rate parameters and our choice to fit each context separately rather than altogether (a strategy adopted in fitting the non-Rescorla–Wagner version of the models). Nonetheless, our model qualitatively described probability estimates equally well, if not better, compared with normalization models when fitting the group data, when fitting individual-subject data, and when fitting individual-subject data using Rescorla–Wagner version of the models. We also fitted other versions of these three model classes, namely, our URD model, the DN model, and the RN model. They in general did not produce significantly different results (S4 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Model fitting results: Individual data.

(A–B) Results from fitting the models at the individual-subject level. The mean model fits (averaged across subjects) are plotted along with subjects’ average probability estimate. (A) Results from URD models (URD-γ,URD-γ-λ). (B) Results from DN models (DN-1-γ,DN-2-γ) and RN model (RN). (C–D) Results from fitting the models implemented in the Rescorla–Wagner reinforcement-learning model framework. Models were fit at the individual-subject level. (C) Results from URD models (URD-γ,URD-γ-λ). (D) Results from DN models (DN-1-γ,DN-2-γ) and RN model (RN). (E) Model simulations on choice probability. Here, we show simulations from four of the models mentioned above. We plot choice probability based on simulated choice data according to the Rescorla–Wagner model parameter estimates from each subject. These results should be compared with subjects’ actual choice probability shown in Fig 3A. Conventions are the same as in Fig 3A. Data underlying these graphs can be found in https://osf.io/48j7m. DN, divisive normalization; OS, the other stimulus present in the same context as the stimulus of interest; RN, range normalization; S, stimulus of interest; URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g008

To show that the model fits can capture subjects’ choice behavior in the lottery decision task (post-fMRI session; Fig 3A), we simulated each subject’s choice behavior based on his or her model parameter estimates in the Rescorla–Wagner model framework and computed choice probability (Fig 8E). In general, all the four models illustrated here captured subjects’ actual choice probability equally well.

In summary, although the three model classes considered here—the URD model, DN model, and RN model—did not significantly differ in model comparison statistic, qualitatively the URD model tended to capture the context effect on probability estimates and lack thereof across different reward probabilities better.

fMRI results

We performed both univariate general linear model (GLM) analysis and multivoxel pattern analysis (MVPA) on the fMRI data. All the analyses focused on two time windows during a trial—when the visual stimulus was presented and subjects had to estimate reward probability (stimulus presentation) and when the reward outcome (whether subjects received a reward or no reward) was shown (reward feedback). For whole-brain GLM analysis, we performed cluster-level inference using Gaussian random field theory (familywise error corrected at p < 0.05) with a p < 0.001 (z > 3.1) cluster-forming threshold.

Average brain activity did not represent context effect on probability estimates

We found no evidence for neural representations of context effect on probability estimates based on average brain activity in response to different stimuli. First, at stimulus presentation, for each reward probability separately, we compared activity between different contexts and got null results at the whole-brain level. Second, we examined whether there were regions representing individual differences in context effect on 50% reward (the probability that showed significant context effect at the behavioral level). For each subject separately, we computed the difference in the mean probability estimates (across trials) between contexts and used it as a behavioral measure of context effect. We performed a group-level mixed-effect analysis on the difference in average brain activity in response to 50% reward stimuli between different contexts using individual subjects’ as a covariate and found no significant result at the whole-brain level. Third, region-of-interest (ROI) analysis on the valuation network in the VMPFC and ventral striatum (VS; using coordinates from the work by Clithero and Rangel [27]) also did not show significant result on the two analyses described above. Together, these findings indicate that context effect on probability estimates is not represented by stimulus-specific average brain activity (across trials) at the time of stimulus presentation. For results at the time of reward feedback, see “Replication of reward magnitude and prediction error signal in VMPFC and striatum” below.

Multivoxel patterns of activity in the dorsal anterior cingulate cortex, ventromedial prefrontal cortex, and intraparietal sulcus predict individual differences in context effect on probability estimates

We subsequently examined whether context effect on probability estimation can be represented in patterns of multivoxel activity by testing whether we can predict individual differences in context effect based on patterns of multivoxel activity. A between-subject, cross-validated Multivoxel Pattern Analysis (MVPA) was performed using a searchlight-based approach [42] looking at predefined ROIs in dACC, VMPFC, VS, aINS, and IPS. Among them, VMPFC, VS, and dACC are involved in representing subjective value in value-based decision making [26,27] and uncertainty-related statistics. The IPS and aINS, although less mentioned in the value-based decision making literature, had also been shown to be involved in representing probability and uncertainty in decision making [35].

We analyzed multivoxel activity patterns separately at the time of stimulus presentation and reward feedback. We found that, at both time windows, multivoxel patterns of dACC activity significantly predicted subjects’ context effect at 50% reward (Fig 9A, 9B and 9C). Multivoxel pattens of VMPFC activity predicted context effect at the time of reward feedback (Fig 9D and 9E), and multivoxel patterns of right IPS predicted context effect at the time of stimulus presentation (Fig 9F). For each subject separately, we computed the behavioral measure of context effect—the difference in mean probability estimates between contexts: mean probability estimate (across trials) of 50% reward stimulus in the [10%, 50%] context minus the mean probability estimate (across trials) of 50% reward stimulus in the [50%, 90%] context. For each subject separately, we computed the neural measure of context effect—the difference in mean BOLD response to the 50% stimuli between the [10%, 50%] context and [50%, 90%] context, which we estimated for each voxel separately. In the searchlight (a sphere with 8-mm radius) analysis, we trained a support vector regression (SVR) to learn and predict individual subject’s based on multivoxel patterns of activity within the searchlight and performed n-fold (n = 33 subjects) cross-validation. The correlation between the left-out subjects’ and the SVR-predicted was then computed. To assess whether prediction performance was above chance, we performed nonparametric permutation test as in the work by Schmack and colleagues [42]. For each searchlight, we constructed the null distribution of correlation coefficients by repeatedly doing the following: we randomly permuted the labels , trained the SVR based on them, and computed the correlation between actual and predicted . Prediction accuracy was considered significant if the probability of the true correlation was less than 0.05 Bonferroni-corrected for multiple comparisons given the number of voxels tested in an a priori ROI. For dACC, at stimulus presentation, 8 voxels survived Bonferroni correction (magenta voxels in Fig 9A); at reward feedback, 5 voxels survived Bonferroni correction (cyan voxels in Fig 9A). For VMPFC, 8 voxels survived Bonferroni correction; for right IPS, one voxel survived Bonferroni correction. The scatter plot in Fig 9B shows the result from the peak dACC voxel—searchlight centering on this voxel produces the largest correlation between predicted and actual —at stimulus presentation (r = 0.89, [x2, y30, z18]). The plots in Fig 9C and 9E, respectively, show the results from the peak dACC voxel at reward feedback (r = 0.83, [x2, y24, z20]) and the peak VMPFC voxel at reward feedback (r = 0.85, [x−12, y46, z0]). The plot in Fig 9F shows result from the peak right IPS at stimulus presentation (r = 0.82, [x48, y−42, z44]). The dashed lines (Fig 9B, 9C, 9E and 9F) represent perfect prediction, not regression fits.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. dACC, VMPFC, and rIPS represent context effect on probability estimates.

(A–C) dACC results. (A) In a between-subject, cross-validated MVPA, we found that patterns of multivoxel activity in dACC predicted context effect on individual subjects’ probability estimates using activity patterns at the time of stimulus presentation (magenta) and at reward feedback (cyan; p < 0.05, Bonferroni corrected for 1,350 voxels in the dACC ROI). The voxels shown are the ones that survived Bonferroni correction. (B) Actual plotted against predicted based on activity pattern—at the time of stimulus presentation—in the searchlight centered on the dACC voxel that produced the largest correlation between actual and predicted (referred to as the peak voxel). (C) Actual plotted against predicted based on activity pattern—at the time of reward feedback—of the peak voxel in dACC. (D–E) VMPFC results. (D) VMPFC voxels that significantly predicted behavioral context effect at the time of reward feedback (p < 0.05, Bonferroni corrected for 1,776 voxels in the VMPFC ROI). The voxels shown are the ones that survived Bonferroni correction. (E) Scatter plot using data from the peak VMPFC voxel. Conventions are the same as in panels B and C. (F) rIPS results (p < 0.05, Bonferroni corrected for 196 voxels in the rIPS ROI). Scatter plot using data from the peak rIPS voxel. The dashed line in panels B, C, E, and F represents the 45-degree line, indicating perfect prediction. Data underlying these graphs can be found in https://osf.io/48j7m. dACC, dorsal anterior cingulate cortex; MVPA, multivoxel pattern analysis; rIPS, right intraparietal sulcus; ROI, region of interest; VMPFC, ventromedial prefrontal cortex.

https://doi.org/10.1371/journal.pbio.3000634.g009

Replication of reward magnitude and prediction error signal in VMPFC and striatum

Examining brain activity at the time of reward feedback, we replicated previous findings on reward magnitude [26] and reward prediction error (RPE) [18,43–47] representations in the VMPFC and VS (Fig 10). In a closer look at VMPFC representations for reward magnitude with a larger cluster-forming threshold (z > 3.6), we found two separate clusters—one in the medial OFC (maximum z statistic = 3.76, [x8, y44, z−12]) and the other in the ACC (maximum z statistic = 4.31, [x2, y34, z2]; Fig 10A). We also found that VMPFC and VS represented RPE—the difference between reward outcome (1 = reward, 0 = no reward) and subjects’ trial-by-trial probability estimates (Fig 10B; ROI analysis). We used VMPFC and VS ROIs based on two previous meta-analysis papers on subjective-value representations in decision making (Fig 10B and 10C: Bartra and colleagues [26]; Clithero and Rangel [27]. We also used leave-one-subject-out (LOSO) method to construct ROI in VS based on the RPE contrast in GLM-1 (see subsection “General linear modeling of BOLD response” in “Materials and methods”).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Univariate GLM results replicating reward magnitude and reward prediction error representations in VMPFC and VS and showing significant reward-frequency representation in VS.

(A) Neural correlates of reward magnitude at the time of reward feedback. Cluster-level inference using Gaussian random field theory was performed (familywise error corrected at p < 0.05) with a p < 0.001 cluster-forming threshold. (B) ROI analysis on reward prediction error. We used VMPFC and VS ROIs from two meta-analysis papers on value-based decision making ([26]: shown as Bartra and colleagues; [27] shown as C&R) and VS ROI identified through LOSO method based on reward prediction error contrast at the time of reward feedback. (C) ROI analysis on frequency of reward associated with stimulus (f_S) and the overall reward frequency associated with a context (f_overall). *p < 0.05. Data underlying these graphs can be found in https://osf.io/48j7m. GLM, general linear model; LOSO, leave one subject out; ROI, region of interest; VMPFC, ventromedial prefrontal cortex; VS, ventral striatum.

https://doi.org/10.1371/journal.pbio.3000634.g010

Representation of reward-frequency statistics in ventral striatum

We examined the neural representations for frequency of reward associated with the stimulus (f_S) and the overall frequency of reward (f_overall)—two statistics critical to context-dependent probability estimation in our computational model. At the whole-brain level, we did not find any region that correlated with either one of these statistics. However, in ROI analysis we found that VS activity negatively correlated with f_overall (Fig 10C, LOSO ROI). The negative correlation was consistent with what we found in the behavioral analysis showing that f_overall was negatively correlated with subjects’ probability estimates (Fig 6B).

Four additional behavioral experiments

To further explore probability estimation at different reward probabilities and contexts, we conducted four additional behavioral experiments. In the previous two experiments (fMRI experiment—Experiment 1; behavioral control experiment—Experiment 2), we investigated context effects on three reward probabilities—10%, 50%, and 90%. In the new experiments, we included 30% and 70% that were not investigated in Experiments 1 and 2. These new experiments also allowed us to create new contexts to look at the original probabilities (10%, 50%, and 90%). We note that in Experiments 3 to 5 the probabilities examined were much closer to each other than Experiments 1 and 2. In these situations, the URD model would predict smaller context effects that might not reach statistical significance. Hence, we discussed these results qualitatively in terms of whether they were consistent with the model prediction.

In Experiment 3, we examined reward probabilities at 10%, 30%, and 50% (N = 20 subjects). The design principle was identical to Experiments 1 and 2 (Fig 1). Hence, three contexts—[10%, 30%], [10%, 50%], and [30%, 50%]—were implemented. In addition to examining context effect on 30% that was not investigated in Experiments 1 and 2, this experiment was further motivated by the following reasons. First, this experiment allowed us to examine 10% in a new context—[10%, 30%]. Second, different from Experiments 1 and 2 in which the 50% reward stimulus was the better stimulus in one context ([10%, 50%])—in terms of how likely subjects would receive a reward—but was the worse stimulus in the other ([50%, 90%] context), in this experiment the 50% reward stimuli always had the best prospect (compared with 10% and 30% reward). This experiment thus allowed us to further examine whether context effect on 50% (Experiments 1 and 2) can be purely driven by a better/worse asymmetry. In other words, if context effect on 50% were purely driven by better/worse asymmetry, we would not expect to see context effect on 50% in this experiment. By contrast, if context effect on 50% were driven by the interaction of reference-dependent computation and estimated uncertainty as predicted by the URD model, we would expect to see an effect of context on 50% reward in this experiment.

The results of Experiment 3 were consistent with our model prediction. For 30% reward, subjects gave larger probability estimates (red in Fig 11B) in the context in which the 30% reward stimulus was the better (more likely to receive a reward) stimulus ([10%, 30%] context) than in the context in which it was the worse stimulus ([30%, 50%] context; blue in Fig 11B). For 50% reward, subjects gave larger probability estimates in the [10%, 50%] context than in the [30%, 50%] context (Fig 11C). Because 50% was always the best stimulus, this pattern cannot be explained purely based on the better/worse asymmetry. By contrast, for 10% reward, probability estimates were virtually identical between different contexts (Fig 11A). Finally, we found that subjects’ choice behavior in the lottery decision task was consistent with their probability estimates (Fig 11D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Experiment 3: Examining context effects on probability estimates at 10%, 30%, and 50% reward and finding further support for the URD model.

Conventions are the same as in Fig 5. The design principle of the experiment was identical to the design in Fig 1B except that subjects faced 10%, 30%, and 50% reward probabilities instead of 10%, 50%, and 90% reward. (A) Probability estimates of 10% reward. (B) Probability estimates of 30% reward. (C) Probability estimates of 50% reward. (D) Choice probability in the lottery decision task after the probability estimation task. Error bars represent ± 1 SEM. Data underlying these graphs can be found in https://osf.io/48j7m. URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g011

In Experiment 4, we examined reward probabilities at 50%, 70%, and 90% (N = 20 subjects). The design principle was identical to Experiments 1 and 2 (Fig 1). Hence, three contexts—[50%, 70%], [50%, 90%], and [70%, 90%]—were implemented. This experiment thus allowed us to investigate context effect on a new reward probability at 70%. In addition, similar to the motivations for Experiment 3, we used this experiment to examine 50% and 90% that were present in Experiments 1 and 2 in greater depth. First, in contrast to Experiments 1 and 2 in which 50% was the better stimulus—in terms of how likely it was for subjects to get a reward—in one context and the worse stimulus in the other, here, the 50% reward stimuli always had the worst prospect (compared with 70% and 90% reward). Together with Experiment 3, these two experiments provided us the opportunity to further examine whether context effect on 50% observed in Experiments 1 and 2 arose from the interactions of uncertainty and reference-dependent computations or can simply be attributed to the better/worse asymmetry. Second, this experiment also allowed us to investigate 90% in greater depth by examining 90% in a new context—[70%, 90%]—in addition to the [10%, 90%] and [50%, 90%] contexts investigated in Experiments 1 and 2.

The results of Experiment 4 were consistent with our model prediction. For 70% reward, subjects gave larger probability estimates in the [50%, 70%] context in which it was the better stimulus (red in Fig 12B) than in the [70%, 90%] context where it was the worse stimulus (blue in Fig 12B). For 50% reward, subjects gave smaller probability estimate when the other stimulus carried a 90% reward ([50%, 90%] context; blue in Fig 12A) than when the other stimulus carried a 70% reward ([50%, 70%] context; red in Fig 12A). Because the 50% reward stimuli always had the worst prospect, the effect of context observed here cannot be simply attributed to the better/worse asymmetry. By contrast, for 90% reward, probability estimates were virtually identical between different contexts (Fig 12C). Finally, we found that subjects’ choice behavior in the lottery decision task was consistent with their probability estimates (Fig 12D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Experiment 4: Examining context effects on probability estimates at 50%, 70%, and 90% reward and finding further support for the URD model.

Conventions are the same as in Fig 5. The design principle of the experiment was identical to the design of Experiments 1 and 2 (Fig 1B) except that subjects faced 50%, 70%, and 90% reward probabilities instead of 10%, 50%, and 90% reward. (A) Probability estimates of 50% reward. (B) Probability estimates of 70% reward. (C) Probability estimates of 90% reward. (D) Choice probability in the lottery decision task after the probability estimation task. Error bars represent ± 1 SEM. The red star symbol indicates a significant difference from 0.5 based on 95% bootstrap confidence interval. Data underlying these graphs can be found in https://osf.io/48j7m. URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g012

In Experiment 5, we investigated reward probabilities at 30%, 50%, and 70% reward (N = 20 subjects). Similar to Experiments 1 and 2, 50% was the better stimulus in one context ([30%, 50%]; in terms of how likely it was to receive a reward) but was the worse stimulus in the other context ([50%, 70%]). However, different from Experiments 1 and 2, 30% and 70% were closer to 50% compared with 10% and 90%. This experiment thus preserved the better/worse asymmetry for the 50% reward stimuli but allowed us to examine whether context effect on 50% would change as a function of its distance in reward probability from the other stimuli.

The results of Experiment 5 were mixed. Consistent with our model prediction, for 50% reward, subjects gave larger probability estimate (red in Fig 13B) in the context in which the other stimulus carried a smaller reward probability (30%) than in the context where the other stimulus carried a larger reward probability (70%; blue in Fig 13B). Also consistent with our model prediction was that the size of the context effect on 50% reward was smaller compared with Experiments 1 and 2. However, inconsistent with our model prediction, for both 30% and 70%, probability estimates were virtually identical between different contexts toward the end of the session (Figs 10A and 13C, respectively). Finally, we found that subjects’ choice behavior in the lottery decision task was consistent with their probability estimates (Fig 13D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Experiment 5: Examining context effects on probability estimates at 30%, 50%, and 70% reward and finding further support for the URD model.

Conventions are the same as in Fig 5. The design principle of the experiment was identical to the design of Experiments 1 and 2 (Fig 1B) except that subjects faced 30%, 50%, and 70% reward probabilities instead of 10%, 50%, and 90% reward. (A) Probability estimates of 30% reward. (B) Probability estimates of 50% reward. (C) Probability estimates of 70% reward. (D) Choice probability in the lottery decision task after the probability estimation task. Error bars represent ± 1 SEM. Data underlying these graphs can be found in https://osf.io/48j7m. URD, uncertainty and reference dependent.

https://doi.org/10.1371/journal.pbio.3000634.g013

In summary, Experiments 3 to 5 allowed us to examine in more detail the impact of uncertainty and reference-dependent computations on probability estimation and largely replicated the effects observed in Experiments 1 and 2. First, we found that reference dependency acts strongly in driving context effect especially for stimulus whose reward probability was in the middle of the three probabilities tested in each experiment. This is similar to the reference-dependent gain/loss asymmetry observed in outcome evaluation [1,11] and indicates that people estimate reward probability relative to some reference point—the average reward probability experienced in the recent past. Second, we found that uncertainty plays a key role in driving context effect because regardless of its position among the three probabilities tested in different experiments, there was always—although to varying degree—context effect on 50% probability estimate. The same cannot be said for probabilities—30% and 70%—whose uncertainty was smaller than 50%. Third, in Experiment 5 ([30%, 50%, 70%]), we found that probability estimates at both 30% and 70% were virtually identical between different contexts. This suggests that the no-effect on 10% and 90% in Experiments 1 and 2 were less likely because of these probabilities reaching the boundary on the probability scale. However, this finding was also inconsistent with our model prediction—because there was a moderate level of uncertainty at both 30% and 70%, the URD model predicts a greater difference in probability estimate than what was actually observed. This implies that the impact of uncertainty is smaller than expected and that compared with reference point, the impact of uncertainty might be smaller. Nonetheless, the findings across these 5 experiments further solidify uncertainty dependency and reference dependency as two computational building blocks for context effects on probability estimation.

Finally, in Experiment 6 (N = 20 subjects), we tested whether monetary incentives would change the context effects observed in Experiments 1 and 2. Subjects were told that they would receive additional monetary bonus on a trial-by-trial basis based on how accurate his or her probability estimates were to the true probabilities. During the experiment, subjects were not given feedback on how accurate they were and the monetary bonus. Instead, they were told that their probability estimates were recorded and that at the end of the experiment they would receive the sum of additional bonus based on their trial-by-trial accuracy. We replicated our findings in Experiments 1 and 2 by showing significant context effect on 50% reward probability but not 10% and 90% (Fig 14A–14C). Further, context effect on 50% probability estimates was stronger than both 10% or 90%, which was also reflected in subjects’ choice behavior in the lottery decision task following the probability estimation task (Fig 14D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Experiment 6: Using an incentive-compatible design to examine context effects on probability estimates at 10%, 50%, and 90% reward and replicating the context effects seen in Experiments 1 and 2.

Conventions are the same as in Fig 5. The design principle of the experiment was identical to the design of Experiments 1 and 2 (Fig 1B) except that we provided monetary incentives to the subjects to give accurate probability estimates. (A) Probability estimates of 10% reward. (B) Probability estimates of 50% reward. (C). Probability estimates of 90% reward. (D) Choice probability in the lottery decision task after the probability estimation task. Error bars represent ± 1 SEM. The red star symbol indicates a significant difference from 0.5 based on 95% bootstrap confidence interval. Data underlying these graphs can be found in https://osf.io/48j7m.

https://doi.org/10.1371/journal.pbio.3000634.g014

Computational building blocks for context-dependent probability estimation

We developed a mathematical model for context-dependent probability estimation and tested whether it can describe the behavior observed in this study. In developing our model, we hypothesized that two computational building blocks—reference dependency and uncertainty dependency—play important roles in probability estimation. The first building block, reference dependency, describes that when estimating the probability of reward associated with a stimulus, the average probability of all stimuli a decision maker encounters in a context would serve as a reference point for comparison. People tend to overestimate reward probability if the average probability is smaller than the stimulus reward probability. In contrast, if the average probability is larger than the stimulus probability, people tend to underestimate its probability of reward. Reference dependency has long been suggested in the evaluation of outcomes, treating outcomes as gains or losses relative to some reference point [1], which has been formally modeled as the decision maker’s expectation of future outcomes based on recent experience [2,48]. However, there is very little work on examining whether probability estimation is reference-dependent [49]. Conceptually, the notion of reference point is similar to base rate (e.g., [50]). In the Bayesian framework, the predicted impact of base rate is such that probability estimates would be biased towards the base rate. By contrast, in the reference-dependency framework, probability estimates would be biased away from the reference point. Therefore, these two frameworks are conceptually distinct and would make opposite predictions.

The second building block, uncertainty dependency, should only exist in estimation problems that are uncertain in nature. Given that there is more uncertainty regarding which potential outcome would occur when estimating probabilities that are near 0.5, they are more susceptible to context effect. This uncertainty dependency predicts that the reference-point effect on probability estimation would be modulated by the degree of uncertainty, i.e., variance or standard deviation of experienced outcomes associated with the stimulus of interest such that the effect is stronger at intermediate probabilities (larger uncertainty) than extreme probabilities (smaller uncertainty). These predictions are consistent with what we found in the current study: when stimulus reward probability was at the extremes (10% or 90% chance of reward), context did not significantly impact subjects’ probability estimates. By contrast, when a stimulus carried a 50% reward, subjects overestimated reward probability when the average reward probability in a context was smaller than 50% but underestimated reward probability when the average reward probability was larger than 50%. These results argue against a purely categorical model in which probabilities are not coded numerically but in discrete categories with ordinal relationships. In contrast to our model, such a categorical model cannot account for the magnitude of the context effect at different probabilities observed in this study.

The model we developed in this study and reinforcement learning models [44,51,52] are not mutually exclusive. In fact, we incorporated all three model classes—our URD model, DN, and RN—into the Rescorla–Wagner reinforcement-learning model framework and fit them to individual subjects’ probability-estimate data. Compared with fits in the non-Rescorla–Wagner model framework (past outcomes are equally weighted to compute frequency statistics), the Rescorla–Wagner model fits tended to capture the trial-to-trial dynamics better, suggesting that decaying impact of outcomes into the past better captures the reward frequency statistics used by the decision maker when estimating probability of reward.

Comparison between context-dependent probability estimation and subjective-value computations

Although this study was inspired by previous research that examined context effect on valuation, it differs from previous work in that we asked whether and how context impacts probability estimation. Our work demonstrates similarities in that probability estimation is, like subjective-value estimation [2,48,53–63], context dependent. However, the results also highlight significant differences that are likely due to challenges imposed by estimating probability that are unique to valuation. Recent neurobiological research also has begun to investigate how context impacts subjective-value representations in the brain [6,28,29,31,32,39,40,64,65] and how context-dependent value signals relate to choice behavior [8,66].

Here, we discuss both similarities and differences between probability estimation and valuation, from the perspective of the tasks, computations, and algorithms carried out by the decision maker. First, although reward is present in both, a notable difference between valuation and probability estimation tasks is the presence or absence of uncertainty. The majority of valuation studies examined value computations in the absence of uncertainty (e.g., a stimulus is always associated with 3 drops of apple juice). In contrast, in our task subjects had to estimate the probability of receiving a reward associated with visual stimuli. Another difference in tasks is that although stimulus-reward associations are present in both, subjects’ own preferences for different rewards can clearly impact the subjective value of a stimulus but not its probability of reward. Instead, probability of an event such as reward is often determined by the outside world rather than subjective preferences that are internal in nature. Second, from the perspective of computations, by focusing on identifying computational building blocks for probability estimation and using them to guide model development, we were able to conclude that reference dependency is one crucial building block for both probability estimation and subjective-value computations. However, uncertainty dependency is unique to estimating rewards when there are uncertainties involved. Third, from the perspective of algorithms, our results suggest that there are perhaps more similarities between probability estimation and subjective-value computations than differences. The model comparison statistics showed that our context-dependent model (URD model) did not differ significantly from normalization models that have been useful in describing subjective-value computations. However, a key difference between these models lies at the large end of the extreme probabilities: at 90% reward, our model did not predict context effect—consistent with subjects’ behavior—but normalization models did. Our model offers a simple explanation to the lack of context effect here. That is, because the impact of reference point on probability estimates is scaled by the degree of uncertainty, the impact of reference point would be very small at 90% reward in which subjects almost always received a reward and hence there is very small uncertainty on whether she or he would receive or not receive a reward.

In contrast to our investigation on probability estimation, Palminteri and colleagues [30] focused on value representations—whether they are context-dependent during learning. They manipulated valence—using monetary gains and losses—of potential outcomes associated with different lotteries under consideration in a decision-making task in which subjects had to learn different options through outcome feedback after each choice. They showed that a relative, context-dependent reinforcement-learning model that incorporates reference dependency in the computation of option value better explained subjects’ behavioral data and brain activity in key valuation areas in VMPFC and striatum than the absolute reinforcement-learning model in which value computations are context-independent. Although our study focused on probability estimation and theirs on valuation, the computational models we separately developed shared a key feature in reference dependency. Together, our results highlight the importance of expectation as reference point in both value and probability computations. It should be noted that reference-dependent preference is an active area of research in behavioral and experimental economics in which expectation-based models for reference-dependent preferences have been proposed [2,48,67–70] and tested [61–63,71–74]. These models, including ours and in the work by Palminteri and colleagues [30], share the feature of expectations as reference point, but they also differ in some other key aspects, for example, the definition of expectation and how it is updated. Therefore, it is important in future work to investigate similarities and differences between these models and compare how well they describe behavioral and neural data on valuation and probability estimation.

Neural representations for context effects on probability estimation

Combining univariate and multivariate analyses, our fMRI results indicate that the neural implementations for probability estimation involve a network of brain regions, with dACC, VMPFC, and IPS representing context effects on probability estimates and VS representing the average reward frequency statistic necessary for context-dependent probability estimation. The VMPFC finding, together with previous studies showing its involvement in context-dependent valuation [6,29,30,75], established VMPFC as the common neurocomputational substrate for probability estimation and valuation. By contrast, the involvement of dACC may reflect aspects of probability estimation that is unique to valuation, which has been indicated in tracking uncertainty-related statistics, such as variance and volatility [17,21], comparing recent and distant reward history [22,23], reinforcement-guided learning [76], and decision making under uncertainty [77]. Similarly, the IPS had also been shown to represent valuation signals in decision under ambiguity [34] and decision under uncertainty [33].

A potential alternative interpretation of our dACC finding on representing context effect on 50% reward probability estimates is associated with response conflict or conflict monitoring [78–81] that also showed strong dACC involvement. Recall that in our fMRI experiment (Experiment 1), we assigned two buttons to represent the probability intervals close to 50%—one for the 35% to 50% interval and the other for the 50% to 65% interval. This forced the subjects to choose one of the two competing buttons—therefore initiating a response conflict—when indicating his or her probability estimates of the stimuli whose reward frequency was close to 50%.

We, however, believe that response conflict is less likely to explain our dACC finding for the following reasons. First, for the 50% reward stimuli, response conflict should be present in both the [10%, 50%] and [50%, 90%] contexts. And yet our fMRI MVPA analysis used the difference in brain activity in response to 50% stimuli between the two contexts. Therefore, if dACC represents response conflict that was present in both contexts, then the difference in dACC activity between these two contexts should not reflect response conflict. Hence, the dACC MVPA results we observed are perhaps less likely to be caused by response conflict. Second, the response conflict hypothesis does not specifically provide prediction on the direction and magnitude of context effect, i.e., 50% estimate is smaller in the [50%, 90%] context than the [10%, 50%] context, which we found dACC to represent by tracking individual differences in such behavioral metric. Therefore, even though dACC had been shown to represent conflict and the detection/monitoring of conflict, its activity in the context of our experiment cannot be directly attributed to response conflict or conflict detection/monitoring.

Probability estimation and experience-based decision making

By showing that probability estimation is context-dependent, our results provide insights into experience-based decision making—an area of research that investigates decision making where choosers acquire information about different options through experience—in that context-dependent probability estimation is another source of bias that could affect people’s choice behavior. This research had shown that people nonlinearly weight probability information learned through experience [15] even when their probability estimates matched well with objective probabilities [82]. However, context was typically not manipulated in these studies, and therefore it is not possible to examine how context-induced bias on probability estimation would affect choice. It would be interesting to examine how the presence of these two sources of bias—probability weighting and context-dependent probability estimation—would impact choice behavior. At the neural algorithmic level, representations of probability distortion had been shown in the striatum [83], lateral prefrontal cortex [20], and VMPFC [84]. Given that our results indicate that dACC, VMPFC, IPS (MVPA results), and striatum (average reward frequency) are critical to context effect, it remains open as to whether these two sources of biases are represented in the same brain regions and how they are being combined to impact experience-based decisions. In the future, it would also be important to examine, along with the stimulus standard deviation statistic, how other sources of variability in the environment could contribute to context effect. For example, how would the randomness or the rate of switching between different stimuli presented to the subjects influence context effects on probability estimates? How might this source of variability interact with the stimulus standard-deviation statistic to impact probability estimation?

In many choices we face, we are not explicitly given information about potential outcomes and their associated probabilities of occurrence. This makes probability estimation an essential computation for decision making under uncertainty. Finding that probability estimation is context-dependent not only demonstrates that it is subjective but, more importantly, that it is relative. Such relative subjective probability is represented in brain regions thought to be involved in subjective-value computations (VMPFC) and regions involved in extracting reward statistics from the environments in order to make decisions under uncertainty (dACC and IPS).

Probability estimation is a general problem organisms face in uncertain environments. Indeed, estimating probability through experience is an essential component of computational accounts of a wide array of cognitive tasks—from making sensory inferences, predicting future outcomes, to choosing between uncertain prospects [37,38,85]. Statistically optimal models provide formal accounts of cognition that require probability estimation of various events but has often failed to consider how context, namely, the presence of different events unfolding close in time, could impact probability estimation of each event of interest [86]. Thus, the robust context effects we observed and the computational building blocks identified suggest that context needs to be considered for the many cognitive computations that require estimation of probability.

Ethics statement

All subjects gave written informed consent prior to participation in accordance with the procedures approved by the Taipei Veteran General Hospital IRB. The experiments were conducted according to the principles expressed in the Declaration of Helsinki.

Subjects

A total of 137 subjects (67 women, mean age = 25.3 years) participated in this study. Among them, 37 right-handed subjects participated in the fMRI experiment (Experiment 1); three subjects did not complete the pre-fMRI session. To be consistent in data analysis across subjects, these three subjects’ behavioral and fMRI data were not analyzed. Each of the remaining 5 experiments was purely behavioral (Experiments 2 to 6) and had 20 subjects each. For the fMRI experiment, the subjects received 500 NTD (1 US Dollar = 30 New Taiwan Dollar or NTD) for their participation and received an additional bonus from the experiment. Their average earning was 1,120 NTD. For behavioral experiments, subjects received 225 NTD for their participation. The average additional bonus was 504 NTD (Experiment 2), 395 NTD (Experiment 3), 651 NTD (Experiment 4), 612 NTD (Experiment 5), and 581 NTD (Experiment 6). The difference in bonuses across experiments likely reflected the difference in reward probabilities used in these experiments.

Task

We designed a simple stimulus-outcome association task to investigate how context affects probability estimation. Below, we describe the trial sequence of the task and our manipulation of context. The task was programmed using the Psychophysics Toolbox in MATLAB [87,88]. Visual stimuli were projected from a LCD projector outside the scanner and viewed through a mirror mounted on the head coil. Responses were collected via two MRI-compatible button boxes each consisting of 4 buttons.

Procedure

The fMRI experiment (Experiment 1) consisted of 3 distinct phases—pre-fMRI, fMRI, and post-fMRI sessions. The experiment took approximately 80 min to complete.

Behavioral control experiment (Experiment 2)

One potential limit of Experiment 1 was how we assigned buttons to the intervals of reward probability. The button that included 10% corresponded to the interval between 5% and 20%, the button that included 90% corresponded to the interval between 80% and 95%. It is possible that context also played a role in subjects’ probability estimates on the 10% and 90% stimuli, but as long as the difference in probability estimates between different contexts was not large enough such that estimates in the two contexts corresponded to two different buttons, we would not be able to detect it. In contrast, we had a more sensitive measure for context effect on the 50% probability stimuli, because there were two buttons around 50%—one corresponding to the interval between 35% and 50% and the interval between 50% and 65%.

To rule out this potential confound, we conducted a behavioral control experiment by changing the button-to-probability assignment. In this experiment (N = 20 subjects), there were 10 keys, each corresponding to a 10% interval—[0%–10%], [10%–20%], [20%–30%], [30%–40%], [40%–50%], [50%–60%], [60%–70%], [70%–80%], [80%–90%], and [90%–100%]. For each stimulus, there were a total of 40 trials in the main session, which was equivalent to the fMRI session in Experiment 1 that had 30 trials for each stimulus. Both the interstimulus interval (time gap between stimulus presentation and reward back) and the intertrial interval were 1 s long. In addition, for all reward probabilities, the frequencies of reward each subject experienced matched with their corresponding objective probabilities of reward. This served to control the variability in frequency of reward (across subjects and across contexts for the same probability within a subject) caused by sampling with replacement. Otherwise, the control experiment was identical to Experiment 1.

Four additional behavioral experiments (Experiments 3 to 6)

The design of Experiments 3 to 6 was identical to Experiment 2 except that the reward probabilities tested were different. In Experiment 3, we tested 10%, 30%, and 50% reward in three different contexts: [10%, 30%], [10%, 50%], and [30%, 50%]. In Experiment 4, we tested 50%, 70%, and 90% reward in three different contexts: [50%, 70%], [70%, 90%], and [50%, 90%]. In Experiment 5, we tested 30%, 50%, and 70% reward in three different contexts: [30%, 50%], [30%, 70%], and [50%, 70%]. In Experiments 5 and 6, we included additional monetary bonus when subjects gave accurate probability estimate. The probabilities tested in Experiment 6 were the same as in Experiments 1 and 2. We designed the additional bonus based on the following rule: if subjects pressed a button whose interval overlapped with ±5% of the true reward probability, they would receive a 1 NTD bonus. For example, if the true reward probability was 50%, then subjects would receive 1 NTD bonus if she or he pressed either the 40% to 50% or 50% to 60% button. If probability was 70%, subjects would receive 1 NTD if she or he pressed either the 60% to 70% or 70% to 80% button. Subjects were told that they would not receive feedback on additional bonus during the experiment and that at the end of the experiment they would receive the sum of additional bonus they earned during the experiment.

General linear modeling of BOLD response

Model fitting and model comparison

Computational models.

A total of 9 different models—three versions of the URD model, four versions of the DN model, and two versions of the RN model—were fitted to the subjects’ average probability estimates and were compared using model comparison statistic (BIC). We also fitted data at the individual-subject level using the same models described below. In addition, we incorporated these models into the Rescorla–Wagner reinforcement-learning model framework and fitted them at the individual-subject level (S1 Text).

The URD model takes the following form, (5) as described in Eq 2, and we explicitly modeled τ by the estimated standard deviation of past reward outcomes . We also fitted versions of the same model with (S4 Fig). When fitting the group average data, we used the past 30 trials (the length of a block) to calculate . In general, BIC values tended to be very similar after 5 past trials (S3 Fig). When fitting individual-subject data, we fitted each model using different number of past trials (from 1 to 30), selected the number that best described the data and use it to compute the model fits on probability estimates and the BIC values.

To further model the overestimation of small probabilities and underestimation of large probabilities, which we observed in our data set and was also found in the work by Fox and Tversky [13], we used the following equation to model these estimation biases: (6) where w_p represents the distorted probability estimate and γ the free parameter that captures the nonlinear distortion of probability estimate. Note that when 0<γ<1, small f_S are overestimated and moderate to large f_S are underestimated. Then the probability estimate is computed by the same principles: (7) Eq 7 is referred to as URD-γ. Finally, in URD-γ−λ, we modeled loss aversion [1,11] by updating Eq 7: (8) where λ represents the loss-aversion parameter and is strictly greater than 0.

For divisive normalization models, we fit two forms. The first form (DN-1) [32] is (9) where f_OS is the reward frequency of the other stimulus (OS) present in the same context as f_S, and a and b are the free parameters. For example, in the [10%, 50%] context, if f_S corresponds to the stimulus carrying 50% chance of reward, then f_OS would be the stimulus carrying 10% reward. The second form (DN-2) [91] is (10) For both DN-1 and DN-2, we also considered probability weighting such that was transformed using Eq 6 to become the final probability estimate (DN-1-γ,DN-2-γ).

The RN model [32] takes the following form: (11) where a and b are the free parameters. We also considered probability weighting such that was transformed using Eq 6 to become the final probability estimate (RN-γ). In Table 1, we summarize the 9 different models and their respective free parameters. Note that the parameter σ_noise (Eq 12) is not considered a free parameter of these models. Rather, it is the free parameter for modeling the stochasticity of probability estimate once it is being computed by these models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Models and their free parameters.

https://doi.org/10.1371/journal.pbio.3000634.t001

These models weight past outcomes equally when computing the reward frequency and variance statistics and hence are different from the Rescorla–Wagner reinforcement-learning model framework. We therefore referred to them as the non-Rescorla–Wagner models. We used subjects’ probability-estimate data in the fMRI session when fitting non-Rescorla–Wagner models (for both group and individual-subject fitting). When fitting models in the Rescorla–Wagner model framework, we incorporated data from both pre-fMRI and fMRI sessions. We wish to point out that the conclusion of the model fits and model comparison would be the same regardless of whether to include data from pre-fMRI session. Note that for the individual-subject model fitting (both Rescorla–Wagner and non-Rescorla–Wagner model frameworks), in models that required computing either reward frequency statistic or variance statistic, or both, the results shown in Fig 8 and S4 Fig are based on the number of past trials, for each model separately, that produced the largest value of maximum likelihood across different numbers of past trials (from 1 to 30).