Markov decision process

The simulations reported below build upon the notion of active inference. This is a ‘first-principles’ approach to understanding (Bayes) optimal behaviour. Simply put, active inference treats the brain as using an internal (generative) model of the world to explain exteroceptive, proprioceptive, and interoceptive sensory data. By optimizing beliefs about variables in this model (perceptual inference), or by changing their internal or external environment (action), creatures can ensure their sensations and prior beliefs are consistent. The term ‘belief’ here is used in the technical sense of a Bayesian belief, or probability distribution, typically considered to be sub-personal. A Markov decision process (MDP) is a form of probabilistic generative model that describes the sequential dynamics of unobserved (hidden) variables (e.g., the current state of the cardiac cycle) and the sensations they cause (e.g., baroreceptor signals). In this paper, we deal with a special case of a Markov decision process known as a partially observed Markov decision process (or POMDP). The difference between the two is that the latter include unobserved variables, while other forms of MDP assume all information is directly available to the agent employing that model. The hidden variables of an MDP are hidden states (s_τ) and sequences of actions or policies (π). Within this model there are four kinds of precision; namely, sensory precision in the visual (α) and cardiac (β) domains and the precision of state transitions among interoceptive and exteroceptive states (ζ). For clarity, we will refer to the precision of transitions as (inverse) volatility and use precision to refer to sensory (i.e., likelihood) precision. The generative model then embodies the conditional dependencies between these variables, as expressed graphically in Fig 1. While we provide a brief overview here, we refer readers to [49,51] for more technical detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A Markov decision process generative model: the factor graph on the left illustrates the conditional dependencies, and independencies, between the variables in the generative model (see the main text for a description of the variables).

The variables are shown in circles (with filled circles showing observable variables). An arrow from one variable to another indicates that the latter depends upon the former. The square nodes each represent probability distributions. The panels on the right give the forms of the distributions (associated with each square node) in the generative model, in addition to defining the expected free energy, and specifying the factorization of the approximate posterior (variational) distributions the agent possesses. In this figure, the tilde symbol (~) means a sequence over time. The σ is a softmax (normalized exponential) distribution. H is a vector of the likelihood entropy (where each element of the vector corresponds to an alternative state). The bold o is a predicted observation, while the italicized o represents realized observations. Generally, bold refers to the parameters of the generative model, while italicized symbols represent the random variables described by the model.

https://doi.org/10.1371/journal.pcbi.1010490.g001

It is important to be clear about what is meant by precision, which can be a confusing concept when first encountered. Technically, precision is a feature of a probability distribution, and can be thought of in several complementary ways: the negentropy of that distribution, its inverse temperature, or its inverse (co)variance. All of these quantify the uncertainty in a given distribution, where uncertainty grows as precision declines. This means precision is only meaningful when we ask which distribution it is applied to. In this paper, the precisions we deal with primarily relate to the likelihood distributions (i.e., the probability of sensory data given their latent or hidden causes). In this context, precision becomes the quality or reliability of sensory data (c.f., that can be read as a sensory signal-to-noise ratio).

Hidden states generate observable sensory data with probabilities expressed in a likelihood matrix A. The states evolve through time according to a transition probability matrix, B and depend only on the state at the previous time, and on the policy, π,. Finally, we equip the generative model with preferences (C), prior beliefs about initial states (D), and prior beliefs about policies. Beliefs about policies have two parts. The first of these is a fixed bias (E). This may be thought of as a habit; i.e., ‘what I expect to do’ a priori. The second is a belief that the most probable policies are those that have the lowest expected free energy (G); i.e., ‘what I expect to do’ after considering the consequences of action. A simple intuition for the latter is to think of the selection between alternative courses of action as we might think of Bayesian hypothesis testing (i.e. model comparison); namely, planning as inference [52,53]. Here, each policy can be thought of as an alternative hypothesis about ‘how I am going to behave’. These are evaluated in terms of prior beliefs (E), and the (predicted) evidence future data affords (G). Just as free energy is used to approximate the evidence data affords a hypothesis, expected free energy evaluates the expected evidence, under beliefs about how data are actively generated. As expressed in Fig 1, expected free energy can be separated into two parts. ‘Risk’ quantifies how far predicted observations deviate from preferred outcomes. Minimizing this ensures maintenance of homeostasis. ‘Ambiguity’ quantifies the uncertainty in the mapping from states to outcomes. Minimizing this component ensures that salient, uncertainty-resolving data are sought (leading to epistemic, information gathering, behavior).

Synthetic cardiac arousal

Using the MDP scheme detailed above, we set out to simulate a cardiac arousal response to threatening stimuli (e.g., a vicious looking spider), in comparison to non-arousing stimuli (e.g., some flowers). To do this, we had to define ‘arousal’ and its interoceptive correlates. To keep things as simple as possible, we assumed the subject’s generative model included two sorts of hidden states (interoceptive and exteroceptive–and that she could adopt two modes of engagement with the world (relaxed and aroused). These sorts of generative models are generally cast as Markov decision processes, whereby transitions among (hidden) states generate observable outcomes in one or more modalities. The modalities considered here were exteroceptive (non-arousing versus arousing visual stimuli) and interoceptive (the cardiac phase; diastolic or systolic). Having defined the nature of the state space generating outcomes, this model can then be parameterised in a relatively straightforward fashion as outlined above. For any set of A,B,C,D, and E parameters, one can then simulate active inference using standard marginal message passing schemes [54] to optimize expectations about hidden states of the world–and the action or policy currently being pursued (technically, a policy is a sequence of actions. In what follows, we only consider policies with one action) [49,55].

Crucially, inference about policies rests upon prior beliefs that the policies will minimise expected free energy in the future. This expected free energy has both epistemic and instrumental terms; namely; the ability of any particular course of action to resolve uncertainty about hidden states (known as salience, Bayesian surprise, information gain, etc.) [56–59] and the pragmatic affordance (known as expected value, utility, reward, etc.) as specified by the prior preferences [60].

To capture the fundaments of multimodal integration of interoceptive and exteroceptive modalities, we assumed the following, reasonably plausible, form for the model. The synthetic subject had to infer which of two policies she was pursuing: a relaxed policy or an aroused policy. These are defined operationally in terms of transitions among interoceptive states. Here, we model this in terms of two distinct forms of cardiac cycling among diastolic and systolic bodily states. When relaxed, the probability transitions among cardiac states meant that there were two phases of diastole and one of systole. Conversely, when aroused, the first diastolic state jumped immediately to systole. In brief, this means that being aroused causes cardiac acceleration and increases the average amount of time spent in systole. The outcomes generated by these states were isomorphic; in other words, there was a simple likelihood mapping from states to sensations; such that the subject received a precise or imprecise interoceptive cue about the current cardiac status (i.e., diastole or systole).

Here, we modelled interoceptive and exteroceptive modalities using a joint factorial approach. Probably the simplest way to understand the joint modelling of exteroceptive and interoceptive modalities is with reference to Fig 2. The important message from this figure is that the same hidden states generate outcomes in both modalities. In practice, this means that combinations of exteroceptive and interoceptive observations are generated by the hidden states (and therefore combinations of multimodal sensations inform inferences about hidden states). The joint modelling is reinforced by the specification of preferences over all combinations of observations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The generative model.

This schematic illustrates how hidden states cause each other and sensory outcomes in the interoceptive and exteroceptive domain. The upper row describes the probability transitions among hidden states, while the lower row specifies the outcomes that would be generated by combinations of hidden states that are inferred on the basis of outcomes. The green panel specifies the model’s prior preferences; namely, the sorts of outcomes it expects to encounter. Please see main text for a full explanation. Although this figure portrays interoceptive and exteroceptive outcomes as separate modalities, they were in fact modelled as combinations–so that the prior preferences could be evaluated (this is necessary because the preferred physiological outcome depends upon the visual cue). In this model, the precisions are denoted by Greek letters and control the fidelity of various probabilistic mapping is (i.e., the likelihood or A matrices and the transition or B matrices).

https://doi.org/10.1371/journal.pcbi.1010490.g002

On the exteroceptive side, we considered two states of the visual world; namely, the subject was confronting an arousing or non-arousing visual object. The corresponding visual modality again had two levels (arousing versus non-arousing picture). Crucially, the fidelity or precision of this mapping depended upon the interoceptive state. When the subject was in systole, this mapping became very imprecise. In other words, all outcomes were equally plausible under each hidden state of the visual world. Conversely, during diastole, there was a relatively precise likelihood mapping. This is the crucial part of our model that links the state of the body to the way that it samples the world. Put simply, precise visual information is only available during certain parts of the cardiac cycle, which itself depends upon the state of arousal (i.e., the policy currently inferred and selected). This can be thought of as a simple approximation of cardiac and other bodily timing effects, expressed as a momentary occlusion or attenuation of sensory input by (for example) afferent inhibitory baroreceptor effects [3,7,8,12,17,32,61–63], or by the brief flooding of the retina during cardiac contraction. We note however that this linkage of interoceptive and exteroceptive precision is but one operational hypothesis regarding how frequently observed effects on cardiac, respiratory, and gastric cycles could be defined—and should accordingly be treated more as a proof of principle than a definitive empirical model of brain-body coupling. Nevertheless, this minimal model evinces some of the curious and familiar results in empirical studies of visceral-perceptual coupling.

This simple structure produced some remarkable results that speak to the intimate relationship between interoception and exteroception. These phenomena (see below) rest upon the final set of beliefs; namely, preferred outcomes. Here, the subject believed that she would be, on average, in a systolic state when confronted with an arousing picture and in a diastolic state otherwise. These minimal prior preferences then present the subject with an interesting problem. She has to choose between extending periods of precise evidence accumulation (i.e., a relaxed state with more diastolic episodes) and sacrificing precise information, via cardio-acceleration, should she infer there is something arousing ‘out there’. However, to infer what is ‘out there’, she has to resolve her uncertainty, through epistemic foraging; i.e., maintaining a relaxed state. More precisely, the generative model ensures a preference for a combination of ‘systole’ and the ‘arousing picture’ outcomes and a combination of ‘diastole’ and the ‘non-arousing picture’ outcome. We therefore hypothesized that at the beginning of each trial or exposure to a picture, subjects would be preferentially in a relaxed state until they had accumulated sufficient evidence to confidently infer the visual object was arousing or not. If arousing, she would then infer herself to be aroused and enter into a period of cardio-acceleration (illustrated in Fig 3). Every simulation started off with a weak prior over hidden states that the picture was not arousing–and a weaker prior in favor of the relaxed policy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Simulated Physiology and Perceptual Inference.

To establish the face validity of our model, we first set out to reproduce some basic psychophysiological phenomenology and establish how these phenomena change under ‘healthy’ (i.e., normative) versus ‘visceral lesion’ parameter settings. To do so, we fed agents a fixed sequence of cardiac and exteroceptive stimuli, such that the first 14 trials constituted a ‘baseline’ period of cardiac quescience (i.e., a steady heart rate), in the absence of arousing stimuli. On the 15th trial, an unexpected arousing stimulus (a ‘spider’) is presented and a further 85 trials simulated. This simulation was repeated for 60 simulated participants, each with randomized starting values, half of which had ‘lesioned’ interoceptive precision (β = 0.5, blue lines). Under these conditions our synthetic subjects exhibit a clear ‘startle’ or ‘defense’ reflex (Graham and Clifton, 1966; Sokolov, 1963), characterized by an immediate cardio-acceleration (left panel) and a dramatic shift in the posterior expectation of encountering another threatening stimulus. Interestingly, during the baseline period the posterior expectation of encountering a threat stimulus oscillates with the heartbeat; i.e., the lesioned subjects show both an attenuation of the cardiac response and a blunted belief update. Note that for the right panel, only trials 1–40 are shown. On the left, blue lines show summed heartbeats (time spent in systole) for 15-trial bins; on the right, lines depict the median posterior probability that the agent will see a spider on the next trial. See Methods and Results for more details.

https://doi.org/10.1371/journal.pcbi.1010490.g003

By carefully adjusting the precision of sensory evidence (through adjusting the A matrix), we could trade-off the evidence accumulation against these imperatives to simulate the elaboration of an arousing response to, and only to, arousing stimuli. Furthermore, we anticipated that a failure to implement a selected policy of arousal would both confound inference about the policy being pursued (i.e., an aroused state of mind) and–importantly–confidence about the exteroceptive state of affairs. The latter can be measured quantitatively in terms of the entropy or average uncertainty over hidden exteroceptive states (after taking a Bayesian model average over policies). This leads to the prediction that confidence in perceptual categorisation would not only evolve over time but would depend upon interoceptive inference, with implications for metacognitive inferences at higher levels of the hierarchy not here implemented. We tested this hypothesis in silico through various lesion experiments reported in the subsequent sections (Figs 3–5). In what follows, we illustrate the belief updating and arousal responses under ‘normal’ priors (i.e. precisions) based upon the generative model above (summarized graphically in Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Simulating the influence of interoceptive and exteroceptive precision on uncertainty.

To explore how interoceptive inference may shape metacognition, i.e., inference over inferred sensory uncertainty, we measured the summed entropy of beliefs for both exteroceptive (top panels) and interoceptive (bottom panels) states. By simulating the full range of sensory precision values, from lesioned precision (α or β = 0.5) to extremely high precision (α or β = 1), the predominant pattern of interactions is revealed. A) For exteroceptive inferences (i.e., the agent’s belief that a spider or flower is present), the principle entropy gradient is characterized by reductions in exteroceptive precision. This effect is modulated in part by interoceptive precision; for example, the lowest uncertainty is obtained when interoceptive and exteroceptive precision are maximal. B) Separating exteroceptive uncertainty by each phase of the cardiac cycle reveals a clear effect of the heartbeat on belief entropy, which is modulated most strongly by lesioning the precision of exteroceptive predictions. Lesioning interoceptive uncertainty does raise the overall level of exteroceptive uncertainty, but to a lesser degree. Note that altering exteroceptive precision only affects the diastolic phases (as precision is already attenuated during systole). Interoceptive lesions preclude precise inferences about the cardiac phase, so reduce the discrepancy in uncertainty between these phases. C) Similar to exteroceptive belief, interoceptive metacognition is predominately influenced by interoceptive precision. D) The cardiac cycle also modulates the overall uncertainty of interoceptive beliefs; this effect is greatly increased when interoceptive precision is lesioned. Interestingly, exteroceptive lesions primarily reduce the differentiation between cardiac states.

https://doi.org/10.1371/journal.pcbi.1010490.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Synthetic Heart-Rate Variability (HRV) and Interoceptive Computational Phenotyping.

To illustrate the potential of our approach as a generative model of physiological reactivity, we produced synthetic heartbeat traces and analyzed these with a standard time-frequency approach under various canonical parameter settings. A) Synthetic ECG traces produced by convolving a standard QRS-wave function with systole events generated by our model. B) These where then transformed into RR-intervals by assuming an 350ms sampling rate, C) Power spectra of RR-intervals were calculated using Welch’s method and categorized as ultra-low (ULF), low, (LF), high (HF), and very high frequency (VHF) bands for each simulated agent. Physiological responses were then summarized in terms of beats-per-minute (BMP) and sympathovagal balance (ratio of area under curve for each frequency band, aLF/aHF) [74]. To illustrate the potential of our approach for interoceptive computational phenotyping, we simulated three different agents–one with healthy interoceptive inference (bottom left), another with extremely precise visceral sensations (bottom middle), and another with overly precise priors for the aroused (sympathetic) policy (bottom right). These each produce unique interoceptive inference ‘fingerprints’; i.e., the individual patterns of heart-rate variability produced by these parameter settings. In this example, extremely precise visceral sensations reduce heart-rate and shift overall peak frequency to the high-frequency domain, whereas overly precise arousal priors induce strong heart-rate acceleration coupled with attenuated ultra-low and ultra-fast oscillations. In the future, these idiosyncratic patterns could be used to identify maladaptive interoceptive inference from heart-rate data.

https://doi.org/10.1371/journal.pcbi.1010490.g005

For the purposes of this paper, the generative model and process are assumed to be identical i.e., that the generative model is a good model of its environment, in the spirit of the good regulator theorem [64]. Conceptually, this assumes an important connection between brain, body, and external environment: namely that a particular kind of body and a particular kind of environment implies a particular kind of brain. Given that the brain regulates the body and environment, this implication is reciprocal. An alternative approach would have been to use a fixed generative process, and to assume alternative models that deviate from this process. This may be more interesting in the context of autonomic pathologies, in which the brain fails to regulate its environment in the way it anticipates.

Simulations

We implemented a minimal model of interoceptive inference: in the sense that one’s state of active engagement with the world may be inferred from its interoceptive and exteroceptive consequences. In this minimal model, the two domains of perception are coupled by–and only by–sensory attenuation: i.e., attenuation of sensory precision in the visual domain during (inferred) systole. Precision refers to the reliability or confidence ascribed to a given probabilistic belief. Within this model there are four kinds of precision; namely, sensory precision in the visual (α) and cardiac (β) domains and the precision of state transitions among interoceptive and exteroceptive states (ζ). For clarity, we will refer to the precision of transitions as (inverse) volatility and use precision to refer to sensory (i.e., likelihood) precision. To see how these precisions manifest in the equations of the generative model, please see Fig 2. In this example, because there are only two states, the corresponding parameters of the generative model control both the expected contingencies and their precision. In other words, when α (or β) decreases to 1/2, sensory signals become imprecise and completely ambiguous. In what follows, we will focus on manipulations of precision under a canonical volatility of ζ = 0.9. In other words, we will assume that our synthetic subject believes state transitions among phases of the cardiac cycle follow each other fairly reliably with a 90% probability. Similarly, if there is a flower ‘out there’, then there is a 90% probability that it will remain there at the next sample. Cardiac and visual stimuli were generated by the same precisions and volatilities as assumed by the subject’s generative model.

We conducted three sets of simulations to illustrate the sorts of behaviours that emerge under this active inference scheme–and to establish the construct validity of the model in relation to empirical phenomena that speak to the influence of interoception on exteroception and vice versa. This enabled us to illustrate the basic phenomenology of our agent–in terms of simulated perceptual inference and cardiac physiology–under some differing levels of sensory precision.

In the first set of simulations (Fig 3), we focused on the physiological and psychological response to arousing stimuli. To do so, we tested the hypothesis that the unexpected presentation of a ‘spider’ would induce an aroused state–as reflected in an increased heart rate–and a greater posterior expectation of encountering an arousing spider stimulus on the next trial. To evaluate this hypothesis, we supplied the subject with a fixed sequence of 15 stimuli–in both the cardiac and visual domains–and examined the posterior beliefs about the next exteroceptive state following a period of relaxed cardiac input. Note that this is possible precisely because our generative model includes beliefs about the future–including the next hidden state and subsequent sensory sample. Here, we used as outcome measures the agent’s evoked cardiac acceleration response (calculated by binning the number of systole events across the experiment) and the agent’s posterior belief that the next stimulus would be threatening. These simulations were repeated 60 times with randomized starting values, such that the first thirty ‘healthy’ agents were compared to an ‘interoceptive lesion’ group for whom interoceptive precision had been attenuated (β = 0.5). This enabled us to not only establish the interaction of arousal-based expectations and cardiac arousal, but also to demonstrate how these responses change when interoceptive sensory precision is ablated.

In the second set of simulations (Fig 4), our focus moved from perceptual to metacognitive signals. Here, we examined the interaction between exteroceptive and interoceptive sensory precision on the one hand and their coupling to cardiac timing and metacognitive uncertainty (posterior confidence) on the other. Our goal here was to illustrate how both interoceptive and exteroceptive precision interact to influence the uncertainty signals that form the basis of metacognitive inference, and to link these to empirical findings showing that cardiac arousal biases confidence [1,65]. We note however that our model does not constitute a full generative or hierarchical model of metacognitive inference itself: rather, we here examine the influence of first-order interactions in interoceptive and exteroceptive precision on uncertainty, which implies a difference of input to a putative, higher-order metacognitive system [66–70]. Metacognitive, in this context, is used in a technical sense to refer to the updating of beliefs about beliefs. In this setting, this translates into predictions of predictability or precision; where precision is itself an attribute of a (Bayesian) belief about the causes of sensations. There are arguments that this kind of metacognitive belief updating underwrites sensory attenuation, selective attention and perhaps sentience per se [71–73].

For these simulations, we used the uncertainty about inferred exteroceptive and interoceptive states (as quantified by the summed entropy of posterior beliefs for each state) as outcome measures, simulated under a range of cardiac and visual precision settings (Fig 4A). To further illustrate how these effects oscillate with the cardiac rhythm, we separated these measures for each phase of the cardiac cycle (early diastole, late diastole, systole). We then repeated these analyses comparing ‘healthy’ interoceptive inference agents (α & β = 0.9), to agents for whom either exteroceptive or interoceptive precision was lesioned (α or β = 0.5, respectively). Note that there is nothing special about these numerical values, which were chosen simply to demonstrate substantive changes in behaviour. Their particular values become more meaningful when active inference models are fit to empirical choice behaviour: e.g., [50]. In virtue of our coupling of exteroceptive sensory precision to the cardiac cycle, we anticipated that metacognitive uncertainty (scored by the negative entropy of posterior beliefs) would depend on the precision of both interoceptive and exteroceptive states, and that this effect would clearly oscillate with the cardiac cycle. Further, we expected in the extreme case of our ‘lesioned’ subjects, these effects would be further exacerbated such that interoceptive and exteroceptive uncertainty would increase dramatically, under their respective lesion conditions.

Finally, to complement these simulations we modelled the response of first and second order statistics of the physiological responses to changes in sensory precision. These were based upon simulated heart rate (frequency of systole) and the heart rate variability (HRV) assessed over multiple trials or heartbeats (Fig 5). Our objectives here were; 1) to test the hypothesis that fluctuations in both low-and high- frequency synthetic heart rate variability can be produced by altering the balance of interoceptive sensory precision versus the prior precision for the aroused sympathetic policy, and 2) to illustrate how generative modelling of interoceptive active inference can be used to phenotype maladaptive inference parameters from observed heart-rate data (i.e., interoceptive inference phenotyping). For this analysis, we simulated 1000 trials under three canonical parameter settings designed to resemble potential neuropsychiatric phenotypes of interest: healthy interoception (α = 0.8, β = 0.8, prior probability of parasympathetic policy = 55%), extremely precise interoceptive sensation (α = 0.8, β = 1, prior probability of parasympathetic policy = 65%), and hyper-precise arousal priors (α = 0.8, β = 0.8, prior probability of sympathetic policy = 75%). This prior probability is specified in the E-vector, which provides a prior probability for the two alternative policies. The values used here were MDP.E = [11/20, 9/20] for the baseline policy (slight parasympathetic preference), MDP.E = [13/20, 7/20] for the parasympathetic policy and MDP.E = [5/20, 15/20] for the sympathetic policy.

The resulting time-series of systole events from each agent were then convolved with a canonical QRS-wave response function and transformed into normalized beat-to-beat RR-intervals. To normalize the (arbitrary) sampling rate of each time-series, we assigned a 350ms repetition time (TR) for each state of the MDP simulation, such that the healthy agent had a heart rate of approximately 60 BPM. The time intervals between successive synthetic R-peaks were then calculated. As the RR interval data is unevenly sampled, the time series was linearly interpolated. The power spectrum was then estimated using Welch’s method. In line with conventional HRV analysis, the power spectra were then categorized into four frequency bands corresponding to ultra-low (0–0.04 Hz), low (0.04–0.15 Hz), high (0.15–0.4 Hz), and very-high (> 0.4 Hz) frequency categories. Finally, to summarize the physiological response of each agent, we calculated the beats per minute (BPM) and the ratio between low and high frequency components (LF/HF), i.e., sympathovagal gain or balance. Sympathovagal balance is thought to index the balance of sympathetic and vagal outflows and is frequently implicated in stress and other psychophysiological and clinical disorders [74,75] but see [76–78] for critique. Sympathovagal balance was calculated as the ratio of area under the curve (AUC) for low and high-frequency HRV; AUC^LF/ AUC^HF.

Simulated physiology and perceptual active inference

To explore the face validity of our model, we simulated the basic psychophysiological behavior of our active inference agent. This involved simulating a fixed series of stimuli (states) in which the heartbeat was forced to remain relaxed–and only non-arousing (flower) stimuli were presented. On the 15th trial, an unexpected spider stimulus was presented, and the simulation continued for a further 85 trials. Thus, by evaluating the evolution of the agent’s synthetic interoceptive physiology and exteroceptive beliefs, before and after the quiescent baseline period, we hoped to reproduce and illuminate well-known psychophysiological phenomena such as the defensive startle reflex [79,80] in which the heart-rate accelerates greatly following an unexpected, threatening stimulus.

This analysis, illustrated in Fig 3, revealed several interesting aspects of interoceptive active inference. Over 60 simulations there was a clear and robust increase in heart-rate acceleration, following the presentation of the unexpected or novel threat stimulus. During subsequent experiences of its own heartbeat and arousing versus non-arousing stimuli, this response habituates, resulting in a gradual heart-rate deceleration from the evoked cardiac response. This robust modulation of heart-rate was accompanied by a jump from an expected probability of encountering a spider of about 25% to almost 65% following the spider presentation. This combined response of both the heartbeat and fear-expectations is further underscored by the curious oscillation of cardiac states and the expected probability of observing a spider; note the uptick in expectations of approximately 5% on each systole event (denoted by the pink dotted line on Fig 3, right panel). In sum, it is as if the agent “hallucinates” or misperceives a threatening affective outcome both when baseline arousal is high, and when the linkage between interoceptive states and outcomes is severed.

A simple explanation for this result is that, during presentation of a stream of flowers, we can confidently infer a safe external environment. This accounts for the relatively low probability of inferred spiders in the earlier part of the plot. However, during systole, attenuated integration of exteroceptive data leads to greater uncertainty. Going from a confident inference in the absence of a spider to a more uncertain inference necessarily increases the probability of a scary environment during this cardiac phase. This offers a simple perspective on previous experimental work suggesting that fear-stimuli are potentiated when presented in synchrony with the heart [5,32]; namely, that a mechanism underlying this effect can be found in the link between cardiac active inference and affective expectations. In short, under generative models of an embodied world–in which sensory sampling depends upon fast fluctuations in bodily states–there is a necessary dependency of Bayesian belief updating (i.e., perceptual inference) across all modalities on interoception.

When comparing these effects in the healthy agent to our sample of ‘lesion patients’, a few sensible but counter-intuitive consequences ensue. In the physiological domain, when presented with the unexpected arousal stimulus, the lesioned agent shows a blunted cardiac acceleration response, which remains diminished throughout the simulated trials. This blunting effect is mirrored for negative affective expectations in the immediate post-stimulus (e.g., trials 15–20) period, further underlining the close link between visceral and exteroceptive inference in our agent. The reason for this blunting is found in the variable exteroceptive precision anticipated during different cardiac phases (see also Fig 4B). A visual impression–consistent with a spider–is highly informative during diastole but must be treated with suspicion during the sensory-attenuated systolic phase. This implies a blunting of belief-updating in response to a spider, when we are unsure of cardiac phase (compared to when we are confident of a diastolic phase). A further interesting result is found when examining the controlled baseline period (trials 0–15); baseline fear expectations in the interoceptive lesion group are actually slightly enhanced by about 5–10% posterior probability. This lends an interesting, embodied twist to the literature on ‘circular inference’, psychosis and hallucinations [81,82], suggesting that the disruption of interoceptive precision may be one mechanism underlying hallucinations, particularly those that are affective and/or somatic in nature.

Simulating the influence of sensory precision on belief uncertainty

We next performed a series of simulations to tease apart how interoceptive and exteroceptive precision (and their disruption) influence variables important for metacognition such as the uncertainty in our agent’s beliefs. To do so, we first measured the Shannon entropy for interoceptive and exteroceptive inferences (summed across both factors of posterior beliefs) under a full range of precision settings from 0.5–1. To highlight the oscillatory nature of cardiac effects, we then calculated the same entropy measure separately for each cardiac state (early diastole, late diastole, systole). Finally, we compared these ‘healthy’ simulations to extreme degradations in sensory precision (exteroceptive and interoceptive ‘lesions’), to better understand how disruptions of each modality are integrated in perceptual uncertainty, a key first-order input for metacognitive inference. These lesions can be read as a simulation of total cardiac deafferentation through vagotomy (as was sometimes performed for stomach ulcers) [83], or in the case of perturbed higher-order arousal priors, lesions of the insular cortex [84,85].

This analysis revealed first of all that, in our simplified model, uncertainty is largely influenced by the unimodal precision of each domain. For both exteroceptive and interoceptive inferences, the slope of the uncertainty gradient (Fig 4A and 4C) was predominantly characterized by degradations in the precision of the corresponding modality. However, this modularity is not complete; exteroceptive uncertainty is at its lowest when interoceptive and exteroceptive precision are maximal. Similarly, although interoceptive uncertainty is largely driven by interoceptive precision, small interactions with exteroceptive precision can be observed in the plotted uncertainty gradient. One interesting isomorphism, however, is that overall interoceptive uncertainty is less affected by exteroceptive precision. This is likely due to that fact that in our model, the cardiac cycle directly modulates exteroceptive precision, whereas exteroceptive states only indirectly modulate interoceptive responses, via policy selection.

This intricate relationship of the cardiac cycle and uncertainty is further teased apart in Fig 4B, which shows clearly that exteroceptive confidence oscillates with each phase of the heartbeat, being highest at diastole. This is an unsurprising feature of our model: on each diastole, exteroceptive sensory precision drops effectively to null. Interestingly however, average exteroceptive uncertainty is modulated in a fairly linear fashion by visceral and exteroceptive lesions: average entropy is increased modestly by lesioning interoceptive precision and more robustly by exteroceptive lesions. Whereas interoceptive lesions caused the greatest increase in interoceptive entropy, exteroceptive lesions seem to exert a specific effect of unbinding entropy from the individual cardiac state, again mirroring the isomorphic representation of these states in uncertainty. This is a sensible finding, as the manipulation leads to relatively high uncertainty in the mapping between hidden states and outcomes during all cardiac phases, not just during the previously attenuated systolic phase. This sort of chronic hypo-arousal–as a consequence of a failure to contextually modulate precision–is not unlike that which may underwrite the negative symptoms of schizophrenia or depression [86–89].

Synthetic Heart-Rate Variability (HRV) and embodied computational phenotyping

In our final set of simulations, we illustrated how the interoceptive inference approach developed here offers a new means for analyzing and interpreting fluctuations in observed physiological data. Our goal here was to demonstrate the potential for generative modelling and ‘embodied computational phenotyping’; i.e., the identification of specific parameters of brain-body interaction underlying maladaptive interoceptive inference in psychiatric and other health-harming disorders; e.g., [90]. This approach could for example, be expanded to model either brain-body interaction during explicit interoceptive decisions [91,92], or to cases of passive brain-body interaction, i.e., during experiments presenting stimuli at specific phases of the heart, breath, or stomach (see, e.g., [93]).

To this end, we generated synthetic cardiac data by convolving our train of cardiac events with an ECG response waveform. Following standard methods, we then calculated the normalized beat-to-beat intervals and performed a time-frequency analysis of the resulting RR-interval data. By repeating this analysis for a ‘healthy’ agent under normative values, an agent with overly precise interoceptive signals (i.e., β = 1), and an agent with overly precise prior beliefs about its own arousal, we illustrate how individual HRV fingerprints are linked to unique patterns of interoceptive active inference.

This analysis showed that, despite the exceedingly simple (biomechanically speaking) conditions of our model, sensible and interesting patterns of heart-rate variability emerge for different combinations of interoceptive sensory and prior precision. Specifically, we found that whereas the healthy agent exhibited a relatively relaxed profile in terms of heart rate and sympathovagal balance (BMP = 63.5, peak frequency = 0.14 Hz)–predominated by low versus high frequency oscillations (aLF/aHF = 0.52)–an agent with extremely precise visceral sensations exhibited a mild downshift in heart-rate (BPM = 59.5) coupled with an overall increase in high-frequency oscillations (aLF/aHF = 0.51, peak frequency = 0.25). In contrast, the agent with overly precise arousal priors showed a strong bimodal modulation of both ultra-low and ultra-high frequencies HRV (peak frequencies = 0.04 Hz & 0.34 Hz, respectively), coupled with a strong increase in heart-rate (BPM = 84) and high versus low-frequency outflow (aLF/aHF = 0.47). These results speak to the unique role of different active inference parameters in producing highly idiosyncratic patterns of HRV. In the future, our model may be enhanced to subserve computational phenotyping of individual differences and/or patient subgroups categorized by the balance of visceral precision and arousal policy priors from raw HRV data alone.

The computational neuroanatomy of interoceptive inference

Having addressed the construct validity of our model, we now speculate as to some likely neuronal substrates of the message passing implied by variational inference. Interoceptive inference can be broken down into four core functional domains: basic sensory-motor control, conscious interoceptive (perceptual) awareness, metacognitive monitoring, and hedonic (intrinsic) value. In our model, we focused primarily on the simplest possible implementation of interoceptive inference, corresponding to the sensory-motor domain (i.e., ascending and descending cardiac pathways) and their low-level interaction with exteroceptive inference, via neuromodulatory gain control. Future work will benefit from expanding upon our representation of uncertainty to include the computation of epistemic and/or intrinsic value as proxies for these higher-order interoceptive and metacognitive systems [112,113].

Accordingly, in our sketch of the putative neuroanatomy underlying cardiac active inference (Fig 6), we focus primarily on the neuronal substrates that inscribe low-level viscerosensory and visceromotor control, as well as some hierarchically superior regions related to emotional salience and interoceptive awareness. For simplicity, our model depicts only the minimal neuronal message passing scheme implied by our generative model; as such, we have omitted many of the intermediary relay nodes; e.g., in the thalamus and ventral visual stream. Afferent baroreceptor signals are transmitted along the ascending vagus to the rostrum of the nucleus tractus solitarus [115,116]. From here, ascending viscerosensory signal are projected via brainstem and midbrain nuclei to the thalamus, somatosensory cortex, and posterior insula [117,118]; ascending cardio-sensory outcomes are thus encoded in the NTS and then passed to the posterior insular cortex (PIC) as inferred interoceptive states. The PIC has a well-known role as primary viscerosensory cortex; electrical stimulation of this area elicits phantom visceral sensations (e.g., pain, heart-rate acceleration) [119,120] and bolus isoproterenol infusions increase the intensity of cardiorespiratory sensations and concomitant PIC activations [121,122]. In parallel, visual sensory outcomes are passed via the second cranial nerve to the superior colliculus, where they inform exteroceptive inference in the amygdala, which is well-situated to process salient emotional stimuli [123,124]. These interoceptive and exteroceptive expectations then converge in the anterior insular cortex (AIC), where they inform the selection of the appropriate autonomic policy. Finally, the selected policy is passed down the hierarchy via descending pathways (e.g., by von Economo neurons), to eventually engage the rostral nucleus ambiguus and descending vagus, decelerating the heart-rate when the relaxed policy is selected. Collectively, the scheme represents a multimodal reflex arc interlinking exteroceptive and interoceptive domains to specific patterns of cardio-ballistic responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Computational neuroanatomy of interoception.

The schematic above shows the form of the neuronal message passing implied by active inference for the generative model depicted in Fig 2. We have related this to the anatomical networks that could implement these inferences. The sensory observations in our simulations are visual and interoceptive (cardiac). These sensations are carried by cranial nerves II and IX respectively. Cranial nerve II targets the superior colliculus in the midbrain. This structure sends short latency visual data to the amygdala, which is well placed to make inferences about emotionally salient stimuli. The amygdala additionally receives downstream visual data from the ventral visual stream in the temporal lobe. Cranial nerve IX carries information from the carotid sinus baroreceptors to the nucleus tractus solitarus in the brainstem. This nucleus communicates with the posterior insula (via thalamic and PAG relays); the anterior cingulate monitors and controls the precision of this ascending visceral information via neuromodulation, possibly via feedback through noradrenergic pathways (not shown). The posterior insula and amygdala interact with one another but also project to the anterior insula. This targets the nucleus ambiguus (via brainstem relays such as the periaqueductal gray), which gives rise to the vagus (X) nerve. The vagus nerve targets neurons in the cardiac plexus that project to both the sinoatrial node and the atrioventricular node of the heart, slowing its rhythm. The nucleus tractus solitarus additionally participates in a reflex loop implicating the sympathetic control of the cardiac cycle, but this is omitted for simplicity. The functional anatomy suggested here implies the anterior insula might play a similar computational role in autonomic policy selection to the basal ganglia in selection of policies involving the skeletal muscles [114]. Note that inscribed directed influences (blue arrows), are not assumed to be monosynaptic–for simplicity, many intermediary relay nodes have been omitted.

https://doi.org/10.1371/journal.pcbi.1010490.g006

Within this scheme, we suggest that the rostral anterior cingulate (ACC) controls the precision of ascending visceral outcomes and inferred interoceptive states via neuromodulatory gain control [125,126]. Further, interoceptive and exteroceptive state precisions here interact indirectly through global neuromodulatory influences, possibly through regulation of noradrenaline by the ACC (via descending influence on the locus coeruleus). Neurobiologically and functionally speaking, the AIC and ACC share similar profiles; both are densely populated with Von Economo neurons (VENs), which are well-suited for the long-range modulation of neural activity across the cortex [127], and also contain diverse populations of noradrenergic, dopaminergic, and opioidergic neurons. Both regions further share an integrative connectivity structure, with projections to both lower-level visceral-motor brainstem nuclei and higher-order regions implicated in decision-making, metacognition, and self-awareness, such as the ventromedial and dorsomedial prefrontal cortices [128–132]. However, the AIC is more densely interconnected with the PIC whereas the ACC is more closely related to uncertainty and decision-making. On this basis, we propose that whereas the AIC integrates the visceral and exteroceptive states required for the regulation of arousal policies, the ACC is likely to regulate the gain or precision of these interactions. It is worth noting that this model may explain the widespread, seemingly unspecific activation profiles of these areas [133,134], as the generative model specified here suggests both form part of an integrated hierarchical circuit by which interoceptive and exteroceptive states interact: e.g., either through the regulation of arousal policies or through the modulation of ascending viscero-sensory precision.

What about metacognitive or reward-related interoceptive processes? Although we do not model these higher-order functions, the model can be expanded to include the explicit representation of policy uncertainty and epistemic value as the mechanisms underlying metacognitive self-inference; i.e., the integrative self-model that combines exteroceptive and interoceptive predictions into a conscious schema [39,68]. In this case, we would expect that the VMPFC and DLPFC are likely to be engaged in inferences about variables (e.g., those derived from expected free energy such as epistemic and intrinsic value) that contextualize the inferences performed by the AIC and ACC over longer timescales [55,60].

Limitations and future directions

The model and simulations presented here represent a minimal proof-of-principle demonstrating how cyclic interactions of interoceptive and exteroceptive perception could arise directly from the principles of active inference. Here, our primary goal was to move the literature beyond purely conceptual analyses of ‘interoceptive inference’, to provide a formal model sub-serving direct hypothesis testing. As such, we focus primarily on reproducing commonly reported phenomena, rather than empirical cross-validation or biological plausibility. The model as presented is thus a demonstration of formal theory, and substantive further work will be needed to develop an observation model of empirical heart-brain interactions or associated physiological data: c.f., [46,135].

Along these lines, we imagine a variety of interesting further applications of the modelling developed herein. For example, the present MDP scheme could be expanded to include more biologically realistic cardiac parameters, or to include other visceral modalities such as gastric or respiratory fluctuations. Similarly, the exteroceptive states modelled here could be adapted to a variety of experimental tasks to capture embodied influences on, for example, active spatial navigation [136–138], active reward learning [139,140], interaction between the cardiac cycle and ballistic saccades [15,17,50], or metacognitive inferences [1,65,112]. These and other future directions will hopefully guide a newly embodied approach to computational psychiatry, enabling the detailed phenotyping of clinical populations in terms of aberrant interoceptive inference.