Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, ‘ surprise ’ and ‘ novelty ’ refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.


Introduction
An unexpected video interruption strengthens human memory of the video's content [1], mismatches between visual flow and locomotion facilitate synaptic changes in the mouse visual cortex [2], monkeys show faster saccades to unseen objects than to familiar ones [3], and mice have a higher breathing frequency when sniffing new odors than those already known [4].
What these four statements have in common is that they all concern situations where words like 'surprise' and 'novelty' seem applicable: The first two statements assess neural responses to violation of expectations, potentially caused by a feeling of surprise, whereas the second two statements assess behavioral responses to unfamiliar stimuli, potentially triggered by novelty of the stimuli. It hence feels tempting to rephrase the first two statements to 'surprise strengthens memory and modulates learning' and the second two to 'novelty attracts attention and drives curiosity'. However, the rephrased statements imply notably more than the original statements: They suggest common mechanisms for different experimental phenomena across different species. Such generalisations are important for moving towards a unified understanding of the brain, but they can be misleading if not justified.
Intuitive usage of 'surprise' and 'novelty' is common practice in neuroscience [5], psychology [6], and machine learning [7]. However, it has remained a mystery how humans' self-reported degree of 'surprise' when entering a new and unexpected room [8] relates to the brain activity of monkeys seeing 'surprising' fractals [9]. This is particularly worrisome as the words 'surprise' and 'novelty', sometimes used interchangeably, refer to different measurable variables in different studies [10,11]. Moreover, neural and behavioral signatures of several novelty-or surprise-related variables have been found simultaneously in single experiments [12e16].
If there are indeed common principles of how 'surprise' and 'novelty' contribute to different brain functions across brain areas and species, then we need systematic studies that enable neuroscientists to distinguish between different 'aspects' of surprise and novelty. In this paper, we argue that computational modeling and quantifiable definitions are necessary first steps for such systematic studies.

A unifying computational framework
In experimental paradigms for studying surprise and novelty, experimental subjects (human participants or animals) are presented with unlikely or infrequent observations [17,18], observations violating repeating patterns [19e22], or, in general, any observation that can intuitively be called 'novel' or 'surprising' (e.g., Figure  1a1). The goal of these experiments is to study how novel or surprising observations influence physiological brain signals [13,23] and action choices [16,24] ( Figure 1a2).
In an example of human multi-step decision-making [16], participants see an image on a computer screen and are instructed to select an action by clicking on one of the disks below the image (Figure 1a1). The next image to appear on the screen depends on the current image and the selected action and is determined by some underlying rules that are unknown to the participants. After several trials, participants associate a particular action with a particular outcome, e.g., clicking on the right action below the coffee cup yields the light bulb as the next image (t = 31 and t = 32 in Figure 1a1).
Participants will feel surprised if they see a different image than the expected one (e.g., the thumb at t = 35 in Figure 1a1). The experimental design is based on the idea that measurable changes in, e.g., EEG, pupil dilation, or reaction time after seeing the unexpected image can be attributed to surprise.
Computational models and quantifiable definitions allow us to go beyond mere ideas. A computational model consists of two parts: (i) an abstract description of the experimental paradigm (from the perspective of experimental subjects; Figure 1b1) and (ii) a formal description of subjects' perception and behavior ( Figure  1b2). We can describe most existing experiments on surprise and novelty by using only three variables at time t þ 1 (Figure 1b1): The observation y tþ1 , a potential cue x tþ1 , and a set of hidden parameters q tþ1 (Box 1) [11]. Computational modeling of experimental paradigms studying surprise and novelty. a. The goal of experiments on surprise and novelty is to study the influence of 'novel' or 'surprising' observations (a1) on various behavioral and physiological measurements (a2). The example in a1 shows a simplified version of the task of [16]: In each trial, human participants see an image on a computer screen and select one of the two available actions (i.e., disks below the image; selected actions are shown in blue). The next image depends on the current image and the selected action and is determined by the underlying rules of the experiment that are unknown to participants (i.e., the graph on the left side; the black arrows correspond to available actions and the red one to a potentially surprising transition after an unannounced change of rules). Assuming all transitions have been experienced in the first 30 trials, observing the 'light bulb' at t = 32 is expected, whereas observing the 'thumb' at t = 35 is unexpected and potentially surprising (after taking action 'right' when seeing the 'coffee cup'). See Figure 2a and [11] for other examples. b. A computational model of an experiment consists of an abstract description of the experimental paradigm (b1) and a formal description of the subjects' behavior (b2). b1. The great majority of experiments can be described using three variables for the trial at time t + 1: The observation y t+1 , the cue x t+1 , and the parameter set q t+1 [11]. For the example in a1, y t+1 is the image at time t + 1, x t+1 = (y t , a t ) is the pair of the last image y t and action a t , and q t+1 models the transitions according to the rules imagined by the subject. b2. A subject is modeled by an algorithm that receives a cue x t+1 and an observation y t+1 as inputs and gives an inferred surprise value s t+1 , an inferred novelty value n t+1 , and, when required, an action a t+1 as outputs. The algorithm has an internal state that is iteratively updated according to some internal dynamics by using the past cues and observations (x 1 , y 1 ; …; x t , y t ). In general, the internal state includes a belief p (t) (q t+1 ) about the parameter set q t+1 , a predictive model p (t) (y t+1 |x t+1 ) to summarise the subject's expectations (e.g., Equation (1)), and a familiarity measure p f The cue x tþ1 summarises all information in time step t þ 1 that subjects may consider for predicting y tþ1 , e.g., the pair (y t , a t ) of observation y t and action a t (Figure 1b1). We always include the action a t in the cue variable x tþ1 ; this allows us to use the same mathematical formulation for experiments with or without the possibility of selecting actions. The set of parameters q tþ1 summarises the hidden rules (for example action-dependent transitions in Figure 1b1) that subjects, potentially unconsciously, imagine to explain the observation y tþ1 given x tþ1 . The imagined rules are estimates of the 'real' rules of the experiment.
Defining novelty and surprise for the observation y tþ1 needs a formal model of how experimental subjects perceive y tþ1 , which is described by the second part of a computational model. All modeling studies on surprise and novelty assume that subjects use their past experiences (x 1 , y 1 ; .; x t , y t ) and some internal update dynamics to make a prediction of the next observation b y tþ1 (Box 1) and, if required, select an action a t accordingly ( Figure 1b2) [26e29]. Depending on the model assumptions, the internal dynamics can have different levels of abstractions [30], ranging from algorithmic implementations of Bayesian inference [31e34] to detailed models of biological neural networks [35e38]. In the most general setting, the model describes (i) the belief p (t) (q tþ1 ) of the subject about the unknown set of parameters q tþ1 and (ii) a predictive distribution of the next observation p (t) (y tþ1 |x tþ1 ) based on that belief (Box 1). The belief p (t) (q tþ1 ) indicates the probability of q tþ1 to be the 'real' rule of the experiment at time t þ 1 according to the subjects' past experience up to time t. The predictive distribution p (t) (y tþ1 |x tþ1 ) summarises subjects' expectations of what they might observe next (Box 1). For example, in a simple case where x tþ1 and y tþ1 take discrete values, we can define the predictive distribution as [29,39] (1) where C (t) (x tþ1 ) is the count of how many times a subject has received cue x tþ1 until time t, C (t) (y tþ1 |x tþ1 ) is the count of those trials that were followed by observation y tþ1 , and constants are added to avoid having zero probabilities.

Novelty is not surprise
Homann et al. (2022) [22] identify a population of neurons in the mouse primary visual cortex that shows strong responses to novel stimuli but not to familiar stimuli even if the latter violate highly predictable observation patterns (Figure 2a1 versus Figure 2a2; Box 1). In the computational framework described above, this means that the physiological variables studied by [22] do not depend on the unexpectedness of y tþ1 given the cue x tþ1 (i.e., preceding stimuli in this case) but only on the unfamiliarity of y tþ1 independently of any inferred regularities in the sequence of observations (Box 1). When describing an experiment -Cue refers to information that subjects use to predict the next observation. The previously selected action ( Figure 1a1) or the previous observation ( Figure 2a) can be used as cues. -Hidden parameters describe the rules that generate experimental observations. A rule may imply that observation B always comes after observation A (Figure 2a). The rules are called hidden because they are not known by the subject but need to be inferred from observations. The rule in the mind of a subject may not be the same as the 'real' rule of the experiment. -A Volatile experiment is an experiment where the 'real' rule changes at unknown moments in time, e.g., [19,24]. When describing an experimental subject -The Belief summarises the subject's guess about the hidden rules, based on past observations. Belief forms a probability distribution over all possible rules of the experiment. -Expectations summarise a subject's guess about possible next observations, based on the current cue and the current belief. Expectations form a probability distribution over all possible next observations. -A Prediction condenses a subject's expectations into a single guess for the next observation.
-Confidence quantifies the certainty of a subject about either (i) the hidden rule or (ii) the next observation.
-Familiarity quantifies how often a specific observation has occurred or how similar it is to other frequent observations. Familiarity does not depend on cues. When describing an observation -Predictable observations can in principle (i.e., if experimental rules are known) be predicted with high probability from cues. surprise and novelty based on their relation to unexpectedness and familiarity: Surprising stimuli violate expectations; hence, surprise is a measure of the unexpectedness of y tþ1 according to the predictive model p (t) (y tþ1 | x tþ1 ). Novel stimuli, however, violate familiarity; hence, novelty is a measure of the unfamiliarity of y tþ1 according to the familiarity p ðtÞ f ðy tþ1 Þ (Box 1 and Figure 2). The familiarity p ðtÞ f ðy tþ1 Þ quantifies how frequent y tþ1 (e.g., a specific image) has been up to time t independently of the cue x tþ1 and potential regularities in observations (see [40] for similar ideas in machine learning). For example, in cases where x tþ1 and y tþ1 take discrete values (same assumption as in Equation (1)), one can define familiarity as the observation frequency where C (t) (y tþ1 ) is the count of how many times a subject has observed y tþ1 until time t, and constants are added to avoid having zero frequencies. Novelty of observation y tþ1 defined as n tþ1 = À log p f (t) (y tþ1 ) ('frequency-based where N (t) is a general function that (i) takes y tþ1 as its argument, (ii) is independent of the cue x tþ1 , and (iii) depends on the subject's current internal state at time t (Figure 1b2).
The central criterion proposed by Xu et al. is that definitions of surprise quantify the unexpectedness of y tþ1 and must be conditioned on x tþ1 , whereas definitions of novelty quantify the unfamiliarity of y tþ1 and must be independent of x tþ1 . Almost all existing definitions of novelty in neuroscience and psychology meet this criterion and can be written as in Equation (3) [5,10]. For example, two alternative approaches to defining novelty are to (i) consider only the first encounter of a specific observation as novel ('absolute novelty'; Figure 2b) [41,42] or (ii) define the novelty of y tþ1 as a decreasing function of the count C (t) (y tþ1 ) ('always decreasing novelty'; Figure  2b) [43,44]. Note that according to novelty definitions based on observation frequency (e.g., Equation (2)), the novelty of the observation y tþ1 increases if it has not been observed for some time. Finally, the perceived novelty of a stimulus does not only depend on how often the exact same stimulus has been experienced. For example, a familiar image with an altered contrast level is a novel stimulus per se, but it may be perceived as a familiar one if the subject cares only about the image identity [46]; similarly, some novel stimuli may be perceived less novel than others if they look similar to familiar stimuli. Many experimental studies support such feature-dependency in novelty responses in the brain [9, 22,47]. Novelty definitions based on the simple observation frequency in Equation (2) can be generalised to account for feature-dependent novelty estimation as the familiarity measure p f (t) (y tþ1 ) can be an arbitrary (non-negative and normalised) function of the stimulus. Analogously, count-based novelty definitions can account for feature-dependent novelty estimation by turning to frequency-based pseudo-counts [40,48].

A taxonomy of surprise definitions
Surprise is caused by a violation of expectations. However, even if we agree that surprise quantifies the unexpectedness of y tþ1 conditioned on x tþ1 , there are multiple possibilities for quantifying unexpectedness [10,12,31e33,49e51]. In general, surprise of y tþ1 can be written as s tþ1 ¼ S ðtÞ ðy tþ1 jx tþ1 Þ; (4) where S (t) is a general function that (i) takes both y tþ1 and x tþ1 as arguments (in contrast to Equation (3)) and (ii) depends on the subject's current internal state at time t (Figure 1b2) [11]. A recent systematic taxonomy of commonly used definitions of surprise proposes two classification schemes for these definitions [52] (Figure 2c).
The first classification is based on the minimal information, about the subject's internal state, that is needed for computing surprise with a given definition (columns in Figure 2c): 1. Observation-mismatch surprise is defined based on the assumption that, at each time t, an experimental subject makes a prediction b y tþ1 of the upcoming observation y tþ1 . Observation-mismatch surprise quantifies surprise as a mismatch between y tþ1 and b y tþ1 ; an example is the absolute difference s tþ1 ¼ jy tþ1 À b y tþ1 j, where b y tþ1 is, e.g., the mean of the predictive distribution [53]. 2. Probabilistic mismatch surprise depends on the full distribution p (t) (y tþ1 |x tþ1 ) of possible outcomes and, hence, requires more information than a single prediction b y tþ1 ; an example is the Shannon surprise or surprisal s tþ1 = À log p (t) (y tþ1 |x tþ1 ) [10]. 3. Belief-mismatch surprise can be evaluated only by having access to the full belief p (t) (q tþ1 ) about the hidden parameter set q tþ1 and requires even more information than the full distribution p (t) (y tþ1 |x tþ1 ); an example is the Bayesian surprise s tþ1 = D KL (p (t) , p (tþ1) ), where D KL denotes Kullback-Leibler divergence [31,32].
The second classification is a conceptual one (rows in Figure 2c): 1. Prediction surprise defines surprising events as those that violate predictions, e.g., the Shannon surprise s tþ1 = À log p (t) (y tþ1 |x tþ1 ). 2. Changedetection surprise also defines surprising events as those that violate predictions but only in comparison with an alternative predictive model; an example is the difference in the Shannon surprise s tþ1 ¼ log½p ðtÞ ðy tþ1 jx tþ1 Þ = p ðalt:Þ ðy tþ1 jx tþ1 Þ, where p (alt.) (y tþ1 | x tþ1 ) is a prior or naive predictive model [33]. According to change-detection surprise definitions, if the observation y tþ1 is unlikely according to both the predictive model p (t) and its alternative, then it is not perceived as surprising. Hence, change-detection surprise can be interpreted as a measure of relative surprise. Importantly, change-detection surprise is optimal to modulate learning in volatile environments [11,33] (Box 1), in agreement with experimental observations [19,24,63].
3. Information-gain surprise defines surprising events as those that change a subject's belief about the world, e.g., the Bayesian surprise s tþ1 = D KL (p (t) , p (tþ1) ). We note, however, that only a handful of information-gain measures [64] have been previously interpreted as measures of surprise [12, 31,32]. 4. Confidence-corrected surprise is defined based on the argument that a given error in prediction should feel more surprising if it is made with higher confidence (Box 1); examples have been suggested both in neuroscience [50] and psychology [51].
The two classifications together propose a refined terminology necessary for a systematic study of surprise in the brain. The first classification is important to judge whether surprise computation based on different definitions can be biologically plausible. For example, evaluating observation-mismatch surprise in a recurrent network of spiking neurons might be simpler than evaluating probabilistic mismatch and belief-mismatch surprise under similar biological constraints [35,36,38]; see [65,66] for different views on the neural implementation of probabilistic inference. The first classification can thus help studies to bridge the gap between algorithmic and mechanistic neural models of 'surprise-driven' attention [67], exploration [68], and learning [28].
The second classification is important as it suggests that observations that intuitively feel surprising can do so because of different aspects of surprise. Importantly, experimental studies of surprise have found separate neural signatures for different definitions (Table 1) showed in an earlier study that even different definitions in the same surprise category (e.g., information-gain surprise) can have different neural signatures. These results suggest that the experimental phenomena previously attributed to a single broad notion of 'surprise' might relate to very different but precise definitions of surprise.
The proposed taxonomy can also provide new interpretations of existing experiments: Beyond the comparison of trial types (e.g., expected versus unexpected trials), mathematical definitions of surprise and novelty enable trial-by-trial data analysis (Table 1). For example, Zhang et al. (2022) [9] observe in monkeys that neural responses to an unexpected stimulus are different depending on whether the stimulus appears in a random unpredictable sequence or in a regular predictable sequence (Box 1). The observed difference may be an indication that surprise signals in different brain areas relate to different surprise categories rather than a single notion of surprise. Such a hypothesis can be tested by trial-by-trial data analysis combined with computational modeling.
Finally, surprise can also quantify the unexpectedness of a scalar (or low dimensional) summary signal extracted from the (high dimensional) observation y tþ1 instead of y tþ1 itself. For example, the unsigned reward prediction error (uRPE) [69,70] measures the mismatch between the reward r(y tþ1 ) associated with stimulus y tþ1 and a prediction b r tþ1 thereof (see [11]). Similarly, an unsigned novelty prediction error (uNPE) measures the unexpectedness of the novelty value N (t) (y tþ1 ) of an observation y tþ1 [16,71]. We can think of uRPE and uNPE as secondary surprise signals since they are derived from a scalar summary signal. When interpreting neural responses to 'novel' stimuli, it is hence important to consider that responses correlated with novelty may in fact be caused by errors in novelty prediction [16,71]. Moreover, subjects may assume potential associations between novelty (or similarly between surprise) and threats or rewards [43,60], which can lead to confounding effects of threats and rewards on neural responses to novelty (or surprise); hence, ideal experimental paradigms for studying neural and behavioral signatures of novelty and surprise require a dissociation of these signals from threats and rewards.
In addition, there can be multiple forms of neural responses to surprise and novelty of an abstract observation y tþ1 depending on how it is neurally represented regarding, for example, sensory modality (e.g., auditory versus visual [59]) or the hierarchy of representations (e.g., image identity [16,46] versus primary visual features [2,22]). For example, a repeating sequence of binary observation as in Figure 2a can be presented as either a sequence of tones or a sequence of images (i.e., different modalities); Grundei et al. (2023) [59] found separate modality-specific and modality-independent EEG signatures of surprise in an experimental paradigm using somatosensory, auditory, and visual roving stimuli. Moreover, a sequence of images could consist of meaningless fractals, sketches of meaningful objects, or different visual drawing styles of always the same object, which results in the same temporal sequence of stimuli in the visual domain but at different levels of abstraction.

Towards a systematic study of surprise and novelty
Different computational roles in learning [34,72] and decision-making [73e75], broadly attributed to 'surprise' and 'novelty, may correspond to different but mathematically precise definitions of novelty and surprise and ultimately also to distinct physiological signals. This leaves us with two main questions: 1. How many fundamentally distinct physiological signals are involved in brain computations related to surprise and novelty? 2. Table 1 Example experimental papers with more than one signal related to surprise and novelty. 'T-by-T' indicates whether trial-by-trial data analysis is performed. 'Compared signals' lists precise mathematical definitions (for trial-by-trial analysis) or the description of trial types (otherwise) that are compared. Animal studies with trial-by-trial analysis exist (e.g., [54,55]) but none with more than one definition of surprise or novelty. What is the role of each physiological signal in each brain function? Addressing these questions requires interactions of theory and experiments.
Recent years have seen an increasing interest in this line of research. For example, Akiti et al. (2021) [60] show that mice exhibit different behavioral patterns when inspecting novel versus surprising objects and that striatal dopamine release modulates the inspection of novel objects differently from the inspection of surprising ones. Dubey and Griffiths (2019) [44] show that seeking novelty and information-gain (i.e., two distinct curiosity-related behavioral patterns) can be considered special cases of seeking a single 'curiosity signal' that is 'optimal' for exploration and depends on experimental conditions. Another study on exploratory behavior, on the other hand, shows that novelty-driven algorithms explain the human search for rewarding states better than algorithms driven by prediction surprise or information-gain, even when novelty-seeking is suboptimal [52]. Similar approaches can be applied to studying the influence of different aspects of surprise and novelty on learning, memory, and attention.
In conclusion, different aspects of surprise and novelty can be captured and quantified by precise definitions and well-designed experiments. The classifications in Figure 2 offer a foundation for future experimental and theoretical studies on surprise and novelty.

Declaration of competing interest
The authors declare no competing interests.

Data availability
No data was used for the research described in the article.