Quantitative and Qualitative Insights into Emotional Interaction Patterns in Families with Clinically Depressed and Non-Depressed Adolescents

Emotional interactions between parents and their children are known to have a significant effect on the development and recurrence of clinical depression in children. While speech models used by existing conversation modeling algorithms can provide information about frequency of speech-silence states, the modeling process itself does not provide qualitative insights into the nature of the emotional process that underlies the speakers’ behavior. To address this issue, a recently proposed higher order emotional influence model (HOEIM) was applied to determine the extent to which “emotional influences” (interpreted as the values of the model coefficients) differed between families with depressed and non-depressed adolescents. The analysis was based on four speaker states: positive emotion, negative emotion, neutral emotion, and silence. The HOEIM estimated the conditional probabilities of these states in parent-child conversations in 29 families with clinically depressed adolescents (14-18 years old) and in 31 families with non-depressed adolescents. The trajectories of the model coefficients displayed across model orders increasing from 1 to 5 (corresponding to the memory time of past emotional states ranging from 1 to 5 seconds) indicated that parent-child interactions were significantly different between these two types of family environments, and the nature of these interactions clearly depended on the topic of conversation.


Introduction Background
Existing research evidence shows strong connections between family processes and adolescent depression [1][2][3][4]. Therefore, the family environment is an important setting to investigate the manner in which depressive symptoms are associated with adolescents' ability to manage interpersonal situations. A negative reciprocal parent-child process, characterized by parents communicating in a negative, critical, and hostile manner, has been linked to an increase in children's depressive behaviours [2][3][4][5]. In many post-therapy cases when depression remits, chronic maladaptive family-based patterns of emotional interactions persist, thereby potentiating relapse or reoccurrence of depressive episodes [6,7]. Knowledge of family based emotional interactions is therefore a key to the prevention of adolescent depression.

Study aim
The primary aim of this study was to provide both a quantitative and descriptive (qualitative) analysis of differences in emotional interactions between depressed and non-depressed family environments.
To achieve this aim, the study applied a recently introduced computational modelling technique, known as the higher order emotional influence model (HOEIM), to determine the nature and extent of differences in parent-child emotional interactions between families with clinically depressed and non-depressed adolescents. The modelling technique and its validation were described in detail in [8,9]. This paper shows for the first time an example of a clinical application of this modelling technique.

Related work
One of the most commonly used models for dyadic conversations is the actor partner independence model (APIM) [10][11][12]. The APIM measures inter-and intra-dependencies between speakers using statistical techniques integrated with the model. The estimated amount of self-dependency is based on subjective observation scores for a number of predictor variables that describe a person's emotion, cognition, and behaviour.
The methodology applied in the current study is closely related to the Influence Model (IM) [25,26]. The IM represented individual speakers in terms of first order coupled Markov chains of speakers' states. The complex joint conditional probabilities of speakers' states were estimated as weighted linear combinations of simpler pair-wise intra-and inter-speaker conditional probabilities, where the weights represented the amount of influence the speakers had on themselves and on other speakers. The IM concept was discussed from the theoretical point of view in and experiments based on the "ground truth" synthetic data showed that the model was computationally tractable [25,26]. of speaker or being in a state S t at time t given a set of previous N states for all speakers participating in the conversation. These estimates were given as [9]: Where p  ()denoted posterior (given by the model) and P() denoted prior (calculated from the data) conditional probabilities, 1<i, j<M, and M was the number of speakers (M=2 for dyadic conversations). The model coefficients θ ij , called the "influence coefficients" (ICs), were normalized such that 1 ij j θ = ∑ and for , These constrains were to ensure that the model posterior probabilities were kept within range. The speaker's states S i t belonged to a set of four meta-states (positive, negative, neutral and silence) defined in Table  1. Three of these meta-states were created by combining a few basic emotional categories and the fourth state denoted silence. Reduction of the number of speaker states to four facilitated easier modelling, and reduced a potential data sparseness problem (i.e. lack of sufficient data in some states for accurate modelling). It also simplified the descriptive interpretation of the modelling outcomes.
In comparison with the more common first order IMs Asavathiratham et al. and Stolar et al. [8,25] the HOEIM used a longer memory of states containing information corresponding not only to a single value of time delay t-1, but also a wider range of time delays t-1,t-2,…,t-N. The model coefficients θ ij were assumed to represent either intra-speaker emotional self-influences (for i=j) or inter-speaker emotional cross-influences . It is important to emphasize that the term "influence" used in this study refers strictly to the objective values of the model coefficients. This is in contrast with subjective qualities related to human feelings more commonly denoted by this term.

How to interpret the HOEIM?
The HOEIM introduces a concept of analysing conversations in terms of emotional (exchange of emotional states) rather than semantic interactions (exchange of particular words or sentences).  In our experiments, the nodes were given as chains of emotional state labels made by external observers at 1 second intervals, when simultaneously listening to speech recordings and observing facial expressions. This means that the labels did not reflect "felt emotions", but emotions conveyed to external observers.
The arrows in Figure 1 indicate the sources (previous states) and directions of emotional influences. Each speaker was assumed to be influenced by his or her own previous states (self-influences) and by the previous states of his or her conversational partner. The common disadvantage of these techniques is that the modelling was limited to two states (silence and speech), and therefore the modelling process could not provide significant insights into the nature of the emotional process that underlies the speaker's behaviour. This issue was addressed in [8], where the IM concept was extended to the first-order analysis of emotional interactions between speakers in dyadic conversations by the dynamic influence model (DIM). The speakers' states were annotated using four categorical emotion labels (positive, negative, neutral, and silence).
In Stolar et al. [9] the first order IM Asavathiratham et al. [25,26] extended to the higher order emotional influence model (HOEIM) combined with four types of speakers' states proposed in [8]. Compared with the first order DIM, the HOEIM used the same number of speakers' states, but offered a longer memory of speakers' states (over 1-5 seconds). As shown in Stolar et al. [9], the HOEIM coefficients can be used to very accurately discriminate between depressed and non-depressed adolescents. This indicated the existence of strong differences between the models for depressed and non-depressed family environments. These differences were investigated in more detail in this study. Namely, the influence coefficients were interpreted as indicators of intra and inter-speaker emotional influences. This allowed a quantitative analysis and qualitative (descriptive) interpretation of interaction differences between family environments with depressed and non-depressed adolescents.

Conversational data
The data used to validate the proposed method was part of a larger collection of audio-visual recordings obtained from the Oregon Research Institute (ORI). The validation set included recordings of 63 different dyadic conversations between a single parent (mother or father) and their adolescent child (son or daughter). The adolescents were between 14 and 18 years of age. All conversations had a natural, unscripted character and were conducted in the same sequential order on three different topics of family interaction: family event planning (EPI), family consensus (FCI), and problem solving (PSI).
The average time duration of the conversations was about 20 minutes/dyad/topic. The average ratio of the adolescents' to parents' speech duration across all conversations was 0.73. Based on clinical interviews, 29 adolescent participants (24 female and 5 male) met the diagnostic criteria (DSM-4) for a current episode of major depressive disorder [1]. The remaining 34 adolescent participants (24 female and 8 male) did not meet diagnostic criteria for any current psychiatric disorders, and had no history of mental health treatment. The psychiatric status of the parents was not assessed. All participants were English speaking and matched in cultural and socio-economic background. The speech recordings were synchronized with second-bysecond manual annotation made by three independent observers using 10 categorical emotion labels listed in Table 1. The labelling process was based on the observed speech, facial expressions, body movements, and gestures. The coding results were positively assessed for inter-observer agreement with the kappa statistics value κ=82 indicating a good strength of agreement [27].

Higher order emotional influence model (HOEIM)
The HOEIM of order N estimated the joint conditional probabilities   For each speaker, the model given in (1) provided estimates of complex conditional probabilities of a speaker i (parent or child) being in a particular emotional state i t S (positive, negative, neutral or silent) at current time t. For example a N th order model could estimate the probability that "the mother is currently negative given that she was positive one second ago and her son was silent one second ago, and that she was neutral two seconds ago and her son was positive two seconds ago, and ...., and that she was negative N seconds ago and her son was neutral N seconds ago". These probabilities were estimated as weighted sums of two components: probability conditioned by the speaker's own previous states (e.g. the mother is currently negative given that she was positive one second ago and ...., and she was negative N seconds ago) and probability conditioned by the partner's previous emotional states (e.g. the mother is currently negative given that her son was silent one second ago and ..., and her son was neutral N seconds ago). The strength of each of these components was indicated by the corresponding value of the influence coefficient ij θ : intra-speaker influence (for i=j) and inter-speaker influence (for i≠j). Values of the influence coefficients were determined experimentally by fitting the model to the emotional state labels, which were assigned by the independent observers of the audio-visual recordings of parent-child conversations.
Note that due to constraints imposed on the influence coefficients to ensure that the model posterior probabilities were kept within (0,1) range, the equal strength of intra-and inter-speaker influences was indicted by both coefficients being equal to 0.5.

Fitting the HOEIM to conversational data
The prior values of the conditional probabilities of the HOEIM in equation 1 were estimated from the annotation labels of the audio recordings. For the first order model (N=1) this was done by using the Maximum Likelihood (ML) method (counting the occurrences of specific events and dividing the result by the total number of events).
For higher order models (N>1), the N-gram method was applied to improve the efficiency of calculations [28,29]. The Nth-order N-gram model estimated the higher order conditional probabilities using the N most recent state transitions. When concatenating the last N states into a single meta-state, the N-gram represented an N th order Markov model in terms of the first order Markov model.
For each of the first-order components of the HOEIM, the conditional probabilities were then calculated using the ML approach. To compensate for sparse data Kneser et al. and Jeline et al. [30,31] the N-gram method was combined with the modified Kneser-Ney (KN) [28,24] data smoothing technique. Given the higher order prior probabilities estimated from the data using the combined N-gram method and the KN data smoothing method, the best fitting (optimal) ICs Of the HOEIM in (equation 1) were determined using the expectation maximization (EM) algorithm [25,32].
The model fitting procedure was applied separately to the data representing each of the 63 dyads, leading to 63 sets of four different model coefficients ( for , = 1,2) opt ij i j θ for each of the three topics of conversation. This procedure was then repeated for each model order ranging from 1 to 5. Since the data were emotionally labelled at 1 second intervals, the time delay between the current and the previous states was set to 1 second to preserve this resolution.
The models were generated for orders ranging from 1 to 5. Extending the model order up to 5 was motivated by research reports observing so called "emotional inertia" in depressed adolescents, who had a tendency to remain in the same emotional state (up to 4 seconds), whereas their non-depressed counterparts would change their emotional state within 1-2 seconds [33].
For analysis of the results, the model coefficients θ ij were averaged across all 63 dyads for each combination of conversation topic (EPI, FCI, and PSI) and family environment (depressed and non-depressed), and displayed as functions of model order (time memory) as illustrated in Figure 2. Table 2 explains the directions of the influences indicated by the values of the θ ij indexes.

Descriptive Analysis of the Influence Coefficients θ ij
A comparison between the emotional interactions within depressed and non-depressed families was based on the average values of the influence coefficients θ ij , and their evolution trajectories across increasing model orders (time memories) illustrated in Figure 2. Statistical analysis based on the MANOVA Wilks lambda test without alpha correction (SPSS software) confirmed that the differences between θ ij values were significantly different (p<0.05) between the depressed and non-depressed groups for each model order and for each topic of conversation (Table 3). This means that the overall emotional interaction patterns differed significantly between depressed and nondepressed family environments.
Differences between depressed and non-depressed family environments observed in the θ ij trajectories in Figure 2 2. Self-influences θ 11 of the depressed adolescents were lower than self-influences of the non-depressed adolescents for all model orders.
3. Cross-influences θ 12 of the parents on their depressed children were higher than the cross influences of the parents on their nondepressed children for all model orders.

Cross-influences θ 21 of the depressed adolescents on their parents
were higher than the cross influences of the non-depressed adolescents on their parents for all model orders.
These results indicate that during EPIs, both the depressed adolescents and their parents were more strongly driven by reactions to the emotional states of their conversational partners than the non- 3. In contrast to the EPI, the cross-influences θ 12 of the parents on their depressed children were lower than the cross influences of the parents on their non-depressed children for all model orders. 4. In contrast to the EPI, the cross-influences θ 21 of the depressed adolescents on their parents were lower than the cross influences of the non-depressed adolescents on their parents for all model orders except order 1.
These results indicate that during FCIs, differences between the depressed and non-depressed groups were generally reversed compared with the EPI. This means that compared with the non-depressed group, the depressed adolescents and their parents were more strongly driven by reactions to their own previous emotional states, and less strongly by reactions to the emotional states of their conversational partners. However, the adolescents and parents remained predominantly driven by reactions to their own previous emotional states for both groups.
Differences between depressed and non-depressed family environments observed in the θ ij trajectories in Figure 2(c) during problem solving interactions (PSI): 1. As in the EPI, the self -influences θ 22 of the parents of depressed adolescents were lower than self-influences of the parents of non-depressed adolescents for all model orders (time memories).
2. Similarly to the FCI, the self -influences θ 11 of the depressed adolescents were higher than the self -influences of the non-depressed adolescents for all model orders except order 1 (1 second memory).
3. Similarly to the FCI, the cross-influences θ 12 of the parents on their depressed children were lower than the cross influences of the parents on their non-depressed children for all model orders except order 1.

As in the EPI, the cross-influences θ 21 of the depressed
adolescents on their parents were higher than the cross influences of the non-depressed adolescents on their parents for all model orders.
These results indicate that during PSI, when the interaction was likely to be more conflictual than during EPI and FCI, the patterns were mixed. This means that in comparison with the non-depressed group, the depressed adolescents were more strongly driven by self-influences, and more strongly influenced by the emotional states of their parents. As indicated in Cox et al. [34] this particular type of behaviour could be a sign of a much stronger desire or struggle by the depressed adolescents to achieve emotional autonomy from their parents [35]. As for the other types of interaction, in both groups the adolescents and parents were predominantly driven by reactions to their own previous emotional states.

General observations based on Figure 2:
Both the parents and the adolescents were predominantly driven by their own influences (intra-speaker influences were in all cases higher than the inter-speaker influences). However, as the model order increased and longer-time memory of previous states was taken into account, the cross influences increased and the self-influence decreased with both values coming closer to the value of 0.5, indicating equal importance of intra-and inter-speaker influences. This could be interpreted as the speakers becoming more attuned to the emotions conveyed by their conversational partners with increasing time delay. Therefore, a certain amount of time was needed to observe speakers' reactions to each other's states. These observations show consistency with previously reported existence of "emotional inertia" [33,36].

Influence Coefficient
Meaning and Direction of Influence θ 11 Self-influence of adolescents on themselves θ 22 Self-influence of parents on themselves θ 21 Cross-influence of adolescent on parent θ 12 Cross-influence of parents on adolescents  depressed group. For both groups, the adolescents and parents were predominantly driven by reactions to their own previous emotional states.
Differences between depressed and non-depressed family environments observed in the θ ij trajectories in Figure 2 1. In contrast to the EPI, the self-influences θ 22 of the parents of depressed adolescents were higher than the self-influences of the parents of non-depressed adolescents for all model orders (time memories) except order 1 (1 second memory).
2. In contrast to the EPI, the self-influences θ 11 of the depressed adolescents were higher than the self-influences of the non-depressed adolescents for all model orders.

Discussion
This paper described the first ever application of the HOEIM to the analysis of emotional interactions in clinical settings. As demonstrated in the previous section, application of the HOEIM to emotion labels can reveal the nature of interactions in family environments with depressed and non-depressed adolescents. Most importantly, this is given not only in quantitative terms (values of θ ij ) but can be also expressed in qualitative (descriptive) terms (observational description of θ ij time evolution trajectories illustrated in Figure 2).
To determine the plausibility of the presented results, their consistency with previous, closely related studies will be discussed [8,33,36]. Generally both studies (current and Stolar et al. [8]) observed that the speakers are mostly self-influenced by their immediate past emotional state, whereas, the inter-speaker emotional influences have a lagged characteristic due to the speaker's tendency to remain in the same state for some time.
Disregarding the differences between depressed and non-depressed categories analysed in the current study but not in Stolar et al. [8], and taking into account only the PSI interaction in Figure 2(c) (analysed in both studies), other consistencies include: 1. The self-and cross-influences for both parents and adolescent children show changes from 1 to 5 seconds into the past.

For both the parents and adolescents, as the time delay
between the current and the previous state increases, the cross influences become larger, whereas the self-influence become smaller.
Both the parents and the adolescents are predominantly driven by their own self-influences While maintaining these general agreements, the current study provided new information regarding relative values of self-and crossinfluences of parents versus children, and their differences between depressed and non-depressed family environments. Namely, the values of parents' self-influences versus children's self-influences θ 22 vs θ 11 , as well as the parent influence on child versus the child influence on parent (θ 12 vs θ 21 ), were shown to be dependent on the type of family environment. In addition, the current study showed that these relations were topic-dependent.
The HOEIM revealed characteristics of extrinsic (cross-influences) and intrinsic (self-influences) processes responsible for monitoring, evaluating, and modifying emotional reactions in parent-child conversations. The intensity of these processes was expressed through the values of the ICs, and their temporal characteristics were described by the shape of the IC trajectories. This is consistent with Thompson et al. [37] definition of "emotional regulation" and therefore the proposed HOEIM could offer a new tool for studying emotion regulation processes and the ways in which people accomplish their goals in social interactions.
Other studies that can be regarded as, to some degree, related to the current work have been described in [33,36]. These works examined and compared the degree of "emotional inertia"-a tendency to remain in the same emotional state for some short period of time, in depressed, "at risk" for depression, and non-depressed adolescents.
In Kuppens et al. [33] it was shown that high inertia is an important characteristic of the emotion dynamics observed in depression, and in the follow up study Kuppens et al. [36], examined whether emotional inertia prospectively predicts the onset of depression during adolescence. The assessment was based on speech recordings of early adolescents engaged in naturalistic conversations with their parents. The recordings were labelled second-by-second using behavioural coding indicating positive, negative, and neutral emotional states. The emotional inertia was operationalized as the autocorrelation of emotional behaviour. In other words, it was assumed that higher levels of autocorrelation corresponded to higher levels of emotional inertia. The results showed that greater inertia of both negative and positive emotional behaviours was able to efficiently predict the onset of clinical depression 2.5 years later.
These results appear to be consistent with the values of selfinfluence coefficients θ 11 of the depressed adolescents changing more slowly across model order (time delay) than the θ 11 slopes of their nondepressed counterparts in Figure 2. It could imply lower emotional adaptation rates in depressed adolescents and thus indicating higher "emotional inertia".
In the light of emerging theories, the observations described here relating to parent-child conversations are of particular importance, since the social maladjustment has been linked to difficulties in regulating affect [38,39]. Emotional experience is especially intense in adolescence and the prevalence of affective and behavioural disorders increases dramatically during this phase of life. A better understanding of emotion regulation during adolescence may help us understand individual differences in mental health and adjustment during this period of increased risk [40,41].
The proposed modelling process could have potentially important clinical applications. Firstly, the qualitative terms could lead directly to the design of family based intervention therapies for depression. For example, based on the HOEIM analysis of their conversations, the parents of depressed adolescents (as well as the adolescents themselves) can be instructed by health practitioners on how to adjust their emotional behaviour to prevent relapses or recurrence of depression episodes in their children. Secondly, the quantitative outcomes of the HOEIM can be used to monitor outcomes of the prescribed intervention therapy. This can be achieved by applying the modelling and analysis procedures described in this study before therapy, and then regularly (e.g. monthly) during the course of therapy to observe and monitor the progress and effectiveness of prescribed behavioural changes.

Conclusion
The study applied a recently introduced conversation modelling technique (HOEIM) to model emotional interactions between parents and their children in two types of family environments: families with clinically depressed adolescents and families with non-depressed adolescents.
The modeling outcomes have been confirmed to provide statistically significant differences between these two types of family environments. Analysis of the HOEIM coefficients, representing in a quantitative way the amount of intra-and inter-speaker influences, indicated that the patterns of emotional influences changed depending on the topic of conversation.
Descriptive analysis revealed behavioural characteristics that could provide psychologically important information for the development of intervention therapies for depression and monitoring of outcomes of these therapies.
Future research directions include validation of the proposed methodology on larger data sets, providing balanced gender representation, larger range of conversational topics as well as different demographic and cultural groups of participants.