Shared visual illusions between humans and artificial neural networks

Any information processing system should allocate resources where it matters: it should process frequent variable values with higher accuracy than less frequent ones. While this strategy minimizes average error, it also introduces an estimation bias. For example, human subjects perceive local visual orientation with a bias away from the orientations that occur most frequently in the natural world. Here, using an information theoretic measure, we show that pretrained neural networks, like humans, have internal representations that overrepresent frequent variable values at the expense of certainty for less common values. Furthermore, we demonstrate that optimized readouts of local visual orientation from these networks’ internal representations show similar orientation biases and geometric illusions as human subjects. This surprising similarity illustrates that when performing the same perceptual task, similar characteristic illusions and biases emerge for any optimal information processing system that is resource limited.


Introduction
When inferring certain features about the world from sensory inputs, one inevitably must deal with any uncertainty introduced by sensory noise or external ambiguity. If one has a fixed budget of resources with which to minimize uncertainty, it is clear that one should preferentially minimize uncertainty about important aspects of the world, such as those that occur very frequently or that are important for behavior. This general principle, often referred to as Barlows efficiency principle (Barlow et al., 1961), should apply to any efficient system, either biological or artificial.
In the study of human perception, our uncertainty about the world is often quantified by discrimination thresholds and by perceptual errors. This vein of psychophysical studies has revealed a great deal about the processes underlying human perception (Fechner, 1860;Körding & Wolpert, 2004). Many of these findings conform to the predictions of efficiency given physical constraints and behavioral needs. Recently, such ideas have helped to explain a class of illusions in which the perceived value of a feature can be counterintuitively biased away from more probable values under the prior distribution (Wei & Stocker, 2015. This effect is explained * Denotes equal contribution as deriving, ultimately, from having greater uncertainty on less probable values of a feature, as expected from efficiently allocating resources with which to minimize uncertainty. This means that many potential low-probability stimuli may be compatible with the same activity pattern. An open debate in neuroscience is the precise sense in which (artificial) deep neural networks (DNNs) reproduce aspects of human vision. While many surprising similarities have been noted (Glaser, Benjamin, Farhoodi, & Kording, 2019;Hassabis, Kumaran, Summerfield, & Botvinick, 2017), it is not obvious that DNNs trained on image classification tasks should share perceptual biases with humans. On the one hand, neural networks may be different from brains, e.g. by having no internal noise. On the other hand, neural networks do have noise related to learning, must represent countless potentially useful variables with a finite number of nodes, and must resolve unavoidable ambiguity. As such, their behavior may be similar.
Here we ask whether deep networks have greater uncertainty for rarer variables, as in humans. We also test whether this results in human-like perceptual biases and illusions when   such variables are decoded from network representations.

Results
We examined the uncertainty of the internal representations and decoding biases in pretrained deep neural networks (DNNs) for two simple tasks: estimating the absolute orientation of image patches with sinusoidal gratings or noise textures, and estimating the relative orientation between two lines. Humans are known to exhibit characteristic, nonuniform patterns of discrimination thresholds and biases indicating higher sensitivity to a certain range of orientations in both tasks.
We first quantified the representation uncertainty using Fisher information. We focused on early layers of the pretrained network, which show selectivity to the low-level visual variables we investigate here. A pretrained neural network defines a deterministic mapping from the stimulus to a population response vector, r = f (θ) where θ is a scalar variable that controls some aspect of the stimulus. We can use Fisher information, J(θ) to measure the uncertainty of r as one changes θ (Seriès, Stocker, & Simoncelli, 2009;Berardino, Laparra, Ball, & Simoncelli, 2017). Specifically, assuming in-dependent Gaussian noise, Fisher information simplifies to J(θ) = ∂ f /∂θ 2 2 . Thus, in the presence of either noise or nuisance variability, the sensitivity of the internal representation determines how accurately information can be read out.

Perceived orientation in neural networks
It has been shown that human observers are better in detecting small changes around cardinal (i.e. vertical and horizontal) rather than oblique orientations. Since cardinal orientations are more prevalent than oblique ones, this indicates that representations of orientation in the human visual system are adapted to the orientation statistics of our natural environment (Girshick, Landy, & Simoncelli, 2011). In particular, the visual system minimizes uncertainty for the most frequent feature values, Assuming efficient encoding, this leads to systematic bias in perceived orientations (Wei & Stocker, 2015. Here, we probed whether a convolutional neural network pretrained on image classification shares the same encoding-decoding properties. Representation sensitivity To calculate Fisher information J(θ), we used sinusoidal grating patches where θ controls the overall orientation in the patch (Fig. 2a). We found that Fisher information measured in early layers of the pretrained AlexNet clearly reflects the distribution of local visual orientation in natural images (Fig. 2b). In particular it is proportional to the square of the prior distribution, which is expected when the representation is learned with the constraint of finite resources, e.g. in efficient coding (Wei & Stocker, 2015).

Perceived orientation bias
For the network to "report" its perceived orientation, we added an additional decoding layer with two fully connected output units that we trained to output orientation in the form of a vector with coordinates x and y. The training and test images are bandpass-filtered, noisy orientations textures (Fig. 2d). Figures 2e,f show variance and relative bias between high noise and low noise orientation patches of the network. Smaller variance corresponds to larger Fisher information. The network also shows bias patterns similar to those of human observers where, with larger repulsive biases for larger noise levels (de Gardelle, Kouider, & Sackur, 2010;Wei & Stocker, 2015).

Relative Orientation and Geometric Illusions
Humans consistently perceive acute angles as wider than they are (Carpenter & Blakemore, 1973;Heywood & Chessell, 1977). This effect is closely related to various geometric illusions consisting of intersecting lines, such as the Zollner and Hering illusions, in which measurably straight and parallel lines appear to bend, converge, or diverge, always in manners consistent with acute angles appearing erroneously wide. It has been noted that the distribution of angles in typical scenes is peaked at small angles, which means that the overestimation of an angle is in the direction away from more probable values (Nundy, Lotto, Coppola, Shimpi, & Purves, 2000;Howe & Purves, 2005). This raises the possibility that DNNs would also overrepresent small angles at the expense of larger ones, leading to decoded orientations consistent with common geometric illusions.

Representation sensitivity
We first examined if DNNs trained on ImageNet are sensitive to relative angle in a manner consistent with the natural distribution of angles. We built stimuli consisting of two crossed lines, fed them to a pretrained VGG16 network (Simonyan & Zisserman, 2015), and examined the activations of an early convolutional layer (before the 1st maxpool). We calculated the derivative of the activations with respect to relative angle with a finite-difference method; by holding one line fixed and perturbing the orientation of the other. Since here we were interested in the Fisher information with respect to the relative angle, and not the absolute angle of the line (which also affects the Fisher information), we subtracted the Fisher information with respect to the same rotating line but without the fixed crossed line, and furthermore marginalized over the orientation of the fixed line. The result is the sensitivity to the relative angle. We found that the Fisher information was larger for smaller relative angles, consistent with the distribution previously found to exist in the natural world.

Geometric illusions
We then trained a decoder to estimate the local orientation of every pixel in an input image from the DNN activations, and asked if this would show effects consistent with Zollner and Hering illusions. In order to train this decoder, we generated tens of thousands of images of many crossed, black lines of various curvatures, and then minimized the mean-squared error of the pixel-wise orientation angle and magnitude (as output by a convolved quadrature filter with kernel of approximately 10 pixels). In Fig. 3c, we show the output for the tested illusions, and Fig. 3d shows the error on the angle outputs along the key slices. The errors were in the direction consistent with the visual percept.

Conclusion
We have found that when DNNs are trained to classify images in ImageNet, they learn representations that reflect the statistics of the natural world in a similar way as humans. Uncertainty about low-level visual features (here, orientation and angle) is decreased in proportion to that feature's occurrence in the world. When decoding this information either with a readout network, as shown here, or perhaps internally by downstream layers this results in perceptual biases. It will be interesting to investigate the degree to which this result reflects the distribution of the labels in ImageNet. One can imagine a task (i.e. an allocation of labels) for which a DNN requires only a subset of all low-level features of the world. If, say, the task required knowledge of which locations had angles near 45 • , the DNN would learn to overrepresent that angle despite its relative under-occurence in the data. Thus, the extent to which the Fisher information is proportional to a prior distribution of a low-level feature indicates how equally the information about the label is distributed across that feature. For complex features closer to object identities, we expect the Fisher to diverge from the prior distribution.