Active Inference and Psychology of Expectations: A Study of Formalizing ViolEx

Abstract

Expectations play a critical role in human perception, cognition, and decision-making. There has been a recent surge in modelling such expectations and the resulting behaviour when they are violated. One recent psychological proposal is the ViolEx model. To move the model forward, we identified three areas of concern and addressed two in this study - Lack of formalization and implementation. Specifically, we provide the first implementation of ViolEx using the Active Inference formalism (ActInf) and successfully simulate all expectation violation coping strategies modelled in ViolEx. Furthermore, through this interdisciplinary exchange, we identify a novel connection between AIF and Piaget’s psychology, engendering a convergence argument for improvement in the former’s structure/schema learning. Thus, this is the first step in developing a formal research framework to study expectation violations and hopes to serve as a base for future ViolEx studies while yielding reciprocal insights into ActInf.

A Appendix

1.1 A.1 Detailed Decomposition of VFE and EFE

The first line of the equation below starts with the classical definition of VFE, as mentioned in the main text. The intermediate lines show how to derive the immunization-accommodation or complexity-accuracy decomposition step-by-step.

$$\begin{aligned} \begin{gathered} \begin{aligned} VFE&= E_{q(s|o)} [\ln {q(s|o)} - \ln {p(s,o)}] \\&= E_{q(s|o)} [\ln {q(s|o)} - \ln {(p(s)*p(o|s))}] \\&= E_{q(s|o)} [\ln {q(s|o)} - \ln {p(s)} -\ln {p(o|s)}] \\&= E_{q(s|o)} [\ln {q(s|o)} - \ln {p(s)}] -E_{q(s|o)}[\ln {p(o|s)}] \\&= E_{q(s|o)}\left[ \ln { \frac{q(s|o)}{p(s)}}\right] -E_{q(s|o)}[\ln {p(o|s)}] \\&= D_{KL} [q(s|o)||p(s)]- E_{q(s|o)} [\ln {p(o|s)}] \end{aligned} \end{gathered} \end{aligned}$$

(4)

Starting with the second line, we use the Product rule of probability to factorize the generative model into the generative prior on states and likelihood of observations, p(s) and p(o|s) respectively. Then we use the property of logarithms, $\ln {(A*B)} = \ln {A} + \ln {B}$ to split the factorised generative model in the third line. Finally, we gather up the variational posterior (q(s|o)) and generative prior (p(s)) together and once again use the property of logarithms, $\ln {A} - \ln {B} = \ln {(A/B)}$ to yield the $D_{KL}$ term (see main text). Thus, the decomposition of immunization-accommodation, as pointed out in the main text, is achieved.

Like VFE, we start with the definition of EFE mentioned in the main text - a direct extension of the VFE into the future, calculated under the expectation of future observations and conditioned on a policy. We then derive the Experimentation-Assimilation or Epistemic-Pragmatic decomposition.

$$\begin{aligned} \begin{gathered} \begin{aligned} EFE&= E_{q(o,s|\pi )} [\ln {q(s|\pi )} - \ln {p(s,o)}] \\&= E_{q(o,s|\pi )} [\ln {q(s|\pi )} - \ln {(p(o)*p(s|o))}] \\&= E_{q(o,s|\pi )} [\ln {q(s|\pi )} - \ln {(p(o)*q(s|o))}] \\&= E_{q(o,s|\pi )} [\ln {q(s|\pi )} - \ln {p(o)} -\ln {q(s|o)}] \\&= E_{q(o,s|\pi )} [\ln {q(s|\pi )} - \ln {q(s|o)}] - E_{q(o,s|\pi )}[\ln {p(o)}] \\&= -E_{q(o,s|\pi )} [\ln {q(s|o)} - \ln {q(s|\pi )}] - E_{q(o,s|\pi )}[\ln {p(o)}] \end{aligned} \end{gathered} \end{aligned}$$

(5)

In the second line, we factorize the generative model just like in VFE decomposition using the Product rule. However, unlike VFE, we factorize it into a generative prior on observations and posterior on states, p(o) and p(s|o), respectively. Next, we make an assumption that the variational posterior can approximate the generative posterior well enough, i.e.: $q(s|o) \approx p(s|o)$. These are the two key moves in EFE decomposition, and the remaining steps involve the property of logarithms and gatherings illustrated above to yield the experimentation-assimilation form.

To calculate the values of $q(o,s|\pi )$, q(s|o) and $q(s|\pi )$, further assumptions and approximations are made. For example, q(o|s) and p(o|s) are assumed to be the same distribution encoded by the A matrix. This yields the divergence between variational posterior and variational prior,

$$\begin{aligned} \begin{gathered} \begin{aligned} \left[ \ln { \frac{q(s|o)}{q(s|\pi )}}\right]&= \left[ \ln { \frac{p(o|s)*q(s|\pi )}{q(o|\pi )*q(s|\pi )}}\right] \\&= \left[ \ln { \frac{p(o|s)}{q(o|\pi )}}\right] \\ q(o|\pi )&= \sum _{s} q(o,s|\pi ) \\&= \sum _{s} p(o|s)*q(s|\pi ) \\&= A*s_{\pi } \\ \end{aligned} \end{gathered} \end{aligned}$$

(6)

These equations highlight the close relation between minimizing Free energy functionals, optimally inferring the policies/states and the generative model parameters like A, B, and D, as mentioned in the main text.

1.2 A.2 Learning and Parameter Uncertainty

As briefly mentioned in the main text, learning the parameters of the Generative Process, A (for example), is possible only if ‘a’ is defined in the Generative model. This is because a, b, and d are the Dirichlet concentration parameters that act as priors on the categorical distributions A, B, and D, respectively. Figure 1.a depicts this by equations, $P(A) = Dir(a), P(D) = Dir(d)$ and the choice of choosing Dirichlet distribution as the prior is because of its conjugacy with Categorical distributions [28]. By being a conjugate prior to the parameters of categorical distributions, updating the model parameters amounts to the simple addition of counts to the vector/matrix.

The learning equation used to update the model parameters is given by,

$$\begin{aligned} \begin{gathered} \begin{aligned} d_{trial+1} = \omega *d_{trial} + \eta *s_{\tau =1} \\ \end{aligned} \end{gathered} \end{aligned}$$

(7)

where, d is the initial prior on states given by, $d = p(s_{\tau =1}) = {\begin{bmatrix} d_{1} &{} d_2 \\ \end{bmatrix}}^T$, $\omega$ forgetting rate, and, $\eta$ learning rate. Learning of model parameters mean just updating the concentration parameters of d, namely $d_1$ and $d_2$. An intuitive understanding of learning can be given using an example. Suppose that in the task described in the main text, the agent initially did not know which of the two slots was the ‘correct’ slot and so had an initial prior of $d_{trial = 1} = {\begin{bmatrix} 0.5 &{} 0.5 \\ \end{bmatrix}}^T$. However, at the end of the trial, the agent observes that left slot is the ‘correct’ slot with probability, $s_{\tau =1} = {\begin{bmatrix} 1 &{} 0 \\ \end{bmatrix}}^T$. Assuming that both $\omega$, and $\eta$ is equal to 1, then the agent would have an updated initial prior at the next trial, $d_{trial = 2} = {\begin{bmatrix} 1.5 &{} 0.5 \\ \end{bmatrix}}^T$. If the agent observes left being the ‘correct’ slot for 8 consecutive trials, then $d_{trial = 8} = {\begin{bmatrix} 8.5 &{} 0.5 \\ \end{bmatrix}}^T$. This is what learning through adding counts means in Active Inference.

Note that with a higher initial concentration, the impact of additional count is way lower than when the concentration is lower. Upon normalising, even though ${\begin{bmatrix} 5 &{} 5 \\ \end{bmatrix}}^T$, and ${\begin{bmatrix} 50 &{} 50 \\ \end{bmatrix}}^T$ represent the same probability distribution, the addition of a count in the former changes the distribution more impactfully than in the latter. This is why initializing a model parameter with high concentration prevents learning [28]. This state of having a high initial concentration is called the ‘saturated’ state, where the agent has nothing to learn. Meanwhile, the state of low concentration is called the ‘uncertain’ state because these Dirichlet parameters are a kind of confidence estimate. The lower the Dirichlet counts, the less confident the agent is in its belief and vice versa. Thus, if the Dirichlet count for one state factor, say $d_{1}$ is low compared to the Dirichlet count for another, $d_{2}$, the agent actively seeks out observations that increase the count of $d_1$.

In order to actively seek out observations, this learning process has to be reflected in the EFE functional. This is precisely what happens when learning is activated in an agent,

$$\begin{aligned} \begin{gathered} \begin{aligned} EFE&= E_{q(o,s,\theta |\pi )} [\ln {q(s,\theta |\pi )} - \ln {p(s,o,\theta )}] \\&= -E_{q(o,s,\theta |\pi )} [\ln {q(s|o)} - \ln {q(s|\pi )}] - E_{q(o,s,\theta |\pi )}[\ln {p(o)}] \\&-E_{q(o,s,\theta |\pi )} [\ln {q(\theta |s,o,\pi )} - \ln {q(\theta )}]\\ \end{aligned} \end{gathered} \end{aligned}$$

(8)

where $\theta$ could be either of a, d, b or all of them. Depending on the number of learning parameters, the number of terms in EFE increases. The connection between learning and EFE/experimentation is hinted at in the main text. Having elaborated on it, it becomes clear why learning and information-seeking behaviour are closely linked and why we chose to run our accommodation-immunization simulation without it.

1.3 A.3 Experimentation-Assimilation

The relevant question for experimentation-assimilation is which parts of the task mentioned in the main text parameterize uncertainty and risk-seeking. Any action that reduces uncertainty about the situation (by generating valid information) is regarded as experimentation in ViolEx [21]. There are several uncertainties for the agent here. 1) The uncertainty of finding the ‘correct’ slot (slot uncertainty) in the trial. 2) Uncertainty of not getting a reward even if one chooses the ‘correct’ slot (reward-rate = 90%). 3) Uncertain whether the hint-giver gives a correct hint (hint accuracy = 90%). The simulated agent does not know all these values (because a & d are defined in the GM, but A and D are random variables) but can learn by interacting with the environment and gathering information (experimentation). One can also simulate a lack of this experimentation by making the agent strongly prefer getting rewards w/o hints rather than with hints. To that extent, we construe taking a hint over directly choosing the slot machine as experimentation.

Fig. 3.

The top and bottom rows correspond to Risk-averse and Risk-seeking conditions. From left to right, the uncertainties involved are 1) slot in a stationary environment, 2) slot in a non-stationary environment, and 3) slot and hint in a non-stationary environment. Labels of ‘Action’ and ‘Free Energy’ subplots in each panel have the same interpretation as in Fig. 2.

We used three different parameterizations of the above task in two conditions (Risk-seeking, Risk-averse). The agents in the first and second parameterization have only the slot uncertainty but operate in different (stationary/non-stationary) environments while having all other information. In the third one, the agents do not have information about hint accuracy but should experiment to learn about the uncertainty to perform well. This gradual increase of uncertainty should allow us to analyze the impact of uncertainty and disposition to risk in experimentation and assimilation.

As mentioned above, we ran six simulations to explore how uncertainty and risk mediate experimentation-assimilation. Starting with a risk-averse agent with just the slot uncertainty (Fig. 3.a), we could see that the agent initially explores once and considers exploring briefly right after a loss. Since this is the least uncertain of the three environments, it quickly gains enough information about its environment to start assimilating.

However, when there is higher uncertainty due to a non-stationary setting, it has to be in experimentation mode for a more extended period before starting to assimilate. Experimentation mode here is hint action, or higher probability of taking hint action to collect information but not “random selection of behavioural alternatives” [21] as in RL. The qualitative difference in behaviour observed in the two cases shows that uncertainty, as hypothesized from ViolEx, does play a critical role in Experimentative behaviour. Finally, to drive home the point, we simulated hint uncertainty in the non-stationary environment, and the plot shows that experimentation increases even further (Fig. 3.c). Only in the last 3–4 trials does the agent have enough information to choose right (highlighted through the action probability).

To test our second hypothesis of risk-seeking as a potential mediator of assimilative behaviour, we simulated the same three conditions as above but with a higher difference between prior preference ($\ln {p(o)}$) for winning with and without hints.⁵ This difference, in turn, affects the EFE value (see Eq. (2)), which changes the agent’s policy $\pi$ (Fig. 1 equations). We could garner from Fig. 3.a and 3.d that there is not a vast difference in observed behaviour between an RS and an RA agent when there is relatively small uncertainty. However, the difference in behaviour and performance becomes striking when uncertainty gradually increases, as depicted in Figs. 3.b, 3.c, 3.e, 3.f. Even when observing more losses than wins, the inherent risk-seeking tendency makes experimentation less likely than the risk-averse agent. These qualitative results show that while increased uncertainty leads to increased experimentation, strong individual traits like risk-seeking can counteract the effects of uncertainty to make assimilative strategies more or less likely.