Computational imperfections in human visual search

Abstract


Introduction
Illustration of a trial in the discrimination task. A single oriented ellipse was presented and subjects reported its direction of tilt relative to vertical. The elongation of the stimulus could take two values; we refer to the most elongated type of ellipse as a "high reliability" stimulus and the less elongated type as a "low reliability" stimulus. Feedback was provided by briefly turning the fixation cross red (error) or green (correct) after the response was given. (B) The subject-averaged data (filled circles) and model fits (curves) reveal that sensitivity was higher for stimuli with high reliability (black) compared to those with low reliability (red). Error bars represent 1 s.e.m.

Discrimination task
On each trial, the subject was presented with a single ellipse (67 ms) and reported whether it 157 was tilted clockwise or counterclockwise with respect to vertical (Fig. 1A). Trial-to-trial 158 feedback was provided by briefly turning the fixation cross in the inter-trial screen green 159 (correct) or red (incorrect). The eccentricity of the stimulus was 0.80 on half of the trials ("low 160 reliability") and 0.94 on the other half ("high reliability"), randomly intermixed. On the first 20 161 trials, the orientation of the stimulus was drawn from a uniform distribution on the range −5˚ to 162 +5˚. In the remaining trials, a cumulative Gaussian was fitted to the data collected thus far and 163 the orientation for the next trial was then randomly drawn from the domain corresponding to 164 the 55-95% correct range. This adaptive procedure increased the information obtained from 165 each trial by reducing the number of extremely easy and difficult trials. Subjects completed 500 166 trials of this task.

168
Visual search without external uncertainty (condition A) 169 In this condition, subjects were on each trial presented with four oriented ellipses. On half of 170 the trials, all ellipses were distractors. On the other half, three ellipses were distractors and one 171 was a target. The task was to report whether a target was present. Targets were tilted μ degrees 172 in clockwise direction from vertical and distractors were tilted μ degrees in counterclockwise 173 direction. The value of µ was customized for each subject (Table 2) such that an optimal 174 observer with sensory-noise levels equal to the ones estimated from the subject's 175 discrimination-task data had a predicted accuracy of 85% correct. Stimulus display time was 176 67 ms and each stimulus was presented with an ellipse eccentricity of either 0.80 ("low 177 reliability") or 0.94 ("high reliability"). On each trial, the number of high-reliability stimuli was 178 drawn from a uniform distribution on integers 0 to 4 and reliability values were then randomly distributed across the four stimuli. Feedback was provided in the same way as in the 180 discrimination task. The task consisted of 1500 trials divided equally over 12 blocks with short 181 forced breaks between blocks.  . (3)

254
The optimal local decision variable is the special case in which the assumed level of noise is 255 identical to the true level of noise with which stimuli are encoded, ˆi i   .

256
Suboptimality 2: computational noise. The second kind of suboptimality that we 257 incorporate in the model is "computational noise" on the global decision variable, where η is a zero-mean Gaussian random variable with a standard deviation of σcomputational. One Based on this equation, we implement a 3×2 factorial set of models (Table 4). The first factor 269 determines how the model weights sensory cues: ii) Suboptimal weighting. The observer weights evidence differently for low-reliability and 274 high-reliability stimuli, but possibly using weights that deviate from the optimal ones.

Fig. 2. Four types of suboptimality that produce near-normally distributed errors in the global decision variable.
We simulated 1 million trials of the visual search task and computed for each trial the global decision variable in four suboptimal variants of the optimal model. The histograms (gray areas) show the distributions of the error in these suboptimal decision variables relative to the optimal one. In the first variant, local decision variables were corrupted by Gaussian noise (top left). In the second and third variants, local decision variables were computed using incorrect values for the mean (top right) or standard deviation (bottom left) of the stimulus distributions. In the last variant, local decision variables were computed using incorrect sensory weights (bottom right). All four distributions are reasonably well approximated by a Gaussian distribution (black curves). This suggests that the behavioral effects of these suboptimalities can be captured by a model in which the global optimal decision variable is corrupted by Gaussian noise. Constraining of estimated sensory noise levels 311 We use the estimates of low  and high  from the discrimination task (  Fig. 9 in [36]). Therefore, we expect that 317 sensory noise levels in the visual search task were between a factor 1 and 2 larger as in the 318 discrimination task. We denote this factor by α. Instead of fitting σlow and σhigh in the visual To estimate the effect of ellipse elongation on the sensory precision with which subjects 328 encoded the stimulus orientations, we fit two models to each subject's data in the discrimination 329 task. In both models, stimulus observations are assumed to be corrupted by Gaussian noise.

330
Under this assumption, the predicted proportion of "clockwise" responses is a cumulative 331 Gaussian as a function of stimulus orientation. We refer to the standard deviation of this 332 Gaussian as the sensory noise level. In the first model, the noise level is independent of ellipse 333 elongation and fitted as a single free parameter. In the second model, the sensory noise levels 334 are fitted as separate parameters for the low-and high-reliability stimuli, which we denote by 8.6±1.3 3 ). Moreover, for all subjects the estimated noise level is higher for the low-reliability 338 stimulus than for the high-reliability stimulus (Table 2). Hence, the stimulus-reliability 339 manipulation works as intended. As described in Methods, we use the estimates of low  and 340 high  to customize the target and distractor distributions in the visual search experiment (Table   341 2) and to constrain the models fitted to the data from that experiment.  (Fig. 3A, red lines). In all three 351 conditions, subjects clearly deviate from this prediction (Fig. 3A, black circles), which suggests 352 that they performed suboptimally. 353 We hypothesize that the suboptimality is caused by the kind of inferential imperfections 354 that we intend to capture with the computational noise parameter. Consistent with this 355 hypothesis, we find that the data are well accounted for by the model that uses Eq. (7) to 356 compute the decision variable (Fig. 3A, black curve).
where psubject is the subject's proportion of correct responses, pguess is chance-level accuracy 365 (0.50 in all our tasks), and poptimal is the accuracy expected from an optimal observer. When that the suboptimality is also not caused by attentional lapses, which would be expected to affect 389 optimality similarly in easy and difficult trials. In addition, if subjects had been guessing on a 390 significant proportion of trials, then the asymptotes in their response curves (Fig. 3A) would 391 have deviated from 0 and 1, which does not seem to be the case. Hence, neither a lack of learning 392 nor guessing due to attentional lapses seems to be a plausible explanation of the suboptimality.

393
Validating the assumption that sensory noise was negligible. So far, we have assumed 394 that there is no sensory noise in these conditions, σi=0. However, despite the unlimited display 395 time, it is unlikely that subjects encoded stimulus orientations without any error at all. To obtain 396 an estimate of σi in these conditions, we conduct a control experiment that is identical to the 397 discrimination experiment (Fig. 1A), except that the stimulus has an ellipse eccentricity of 0.97  imperfections (model factor 2). To quantify evidence for these hypotheses, we fit a factorial set of six models and compute the relative support for each of them ( slightly worse than that of the best-fitting model, except in the condition with the highest level 431 of uncertainty (Fig. 4A). However, suboptimal behavior at the level of individual subjects may 432 appear optimal when viewed at the level of the group [23]. Indeed, when analyzed at the level of individual subjects 7 , the combined root mean squared error of the optimal model is almost 434 twice as large as that of the best-fitting suboptimal model (0.0979 vs. 0.0571). The model fits 435 in Fig. 4A also show that there are aspects of the data that neither of the models explain well.

436
In particular, both models consistently underestimate the hit rate when the target is the only 437 high-reliability item. This suggests that there may be additional suboptimalities in the data that  Optimality index. We next estimate how much subject performance deviates from 441 optimal performance in these conditions. Because of the presence of sensory noise, we now 442 make a distinction between absolute and relative optimality [3]. An observer is defined as 443 optimal in the absolute sense when their accuracy equals that of a noiseless optimal observer.

444
In the condition without external uncertainty, this corresponds to an accuracy level of 100% indicates that there are also suboptimalities beyond the effects of sensory noise.

459
Decomposing sources of optimality loss. We assess the relative impact separately for 460 each of the three hypothesized sources of optimality loss: sensory noise, suboptimal weighting 461 of sensory cues, and computational noise. We estimate the optimality loss due to sensory noise 462 as the difference between Iabsolute (the expected accuracy for a noiseless, optimal observer) and is 0.195±0.016. To compute the optimality loss from the other two sources, we take for each 466 subject the maximum-likelihood fit of model M4 and use simulations to compute how much 467 the optimality index increases when "turning off" either type of suboptimality. We find that 468 replacing the suboptimal weights with the optimal ones increases the optimality index with there is no evidence that the degree of suboptimality depends on the level of external indicates clear evidence for a suboptimality beyond sensory noise. Decomposition of the 512 optimality loss suggests that 59% is caused by sensory noise, 38% by suboptimal cue weighting, 513 and 3% by computational noise. Hence, as expected, our methods identify sensory noise and 514 suboptimal cue weighting as factors that strongly affected accuracy in this dataset, but not 515 computational noise. This means that even though suboptimal weighting appears as 516 computational noise on the optimal decision variable (Fig. 2, bottom  found strong evidence against models that give equal weight to stimuli with low and high 602 reliability. Although we found evidence for a discrepancy between the estimated weights used 603 by the subjects and the optimal weights, the optimality loss caused by this discrepancy was 604 small (8%). Therefore, our findings suggest thatdespite our evidence for inferential 605 suboptimalitiessubjects weighted sensory cues near-optimally by their reliability.

607
Unlike most previous studies on visual search, we added external noise to the stimuli (however, 608 see [54] for a similar manipulation). We believe that this approach has two advantages over 609 using deterministic stimuli that could make it valuable in other perception studies too. First, it 610 allows the experimenter to include task conditions without sensory noise. As demonstrated here, 611 such conditions allow the experimenter to assess deviations from optimality without the need 612 to fit free parameters. However, a second advantage is that tasks with external uncertainty may 613 be more consistent with naturalistic conditions, where errors in judgment often arise not only 614 due to sensory uncertainty but also due to external ambiguities, for example caused by imperfect 615 correlations between features in the environment. Due to such ambiguities, naturalistic stimuli 616 are often probabilistic rather than deterministic [32], which prevents even a noiseless optimal 617 observer from reaching 100% correct performance.

618
Our results regarding the effect of external uncertainty on optimality are inconclusive: 619 an effect was found in the conditions without sensory noise, but not in the conditions with 620 sensory noise. One possibility is that the identified effect was a statistical fluke. However, an 621 alternative possibility is that an effect is simply harder to establish in the presence of sensory 622 noise. As explained above, computing the optimality index then requires estimating the 623 subject's level of sensory noise. Imprecisions in these estimates will increase the variance of 624 the optimality-index estimates which, in turn, will reduce the likelihood of finding statistically 625 significant effects. Consistent with this reasoning, we found that the variance in the optimality 626 index estimates was indeed more than 7 times larger in the conditions with sensory noise.

628
Further decomposition of sources of suboptimality 629 In the conditions with sensory noise, we decomposed suboptimalities into three sources: 630 sensory noise, suboptimal cue weighting, and computational noise. In our experiment, these sources accounted for about 47%, 9%, and 44%, respectively, of the optimality loss. The first 632 two sources have quite a specific interpretation, but effects captured by the computational noise 633 parameter could stem from a number of different sources, such as random variability in the 634 neurons that represent the local and global decision variables, imprecise knowledge of 635 experimental parameterssuch as the stimulus distributionsand the use of a suboptimal cue 636 integration rule. We tried to further decompose suboptimalities into these more specific sources, 637 but were unable to do so. The problem is that different types of suboptimalities have near-638 identical effects on the response data, due to which we were unable to reliably distinguish 639 models that implemented more specific kinds of suboptimalities. Future studies may try to solve 640 this model-identifiability problem by using experimental paradigms that provide a richer kind 641 of behavior data to further constrain the models (e.g., by collecting confidence ratings [55]).

642
Our experimental design also did not allow us to distinguish between systematic suboptimalities