What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis

doi:10.1016/j.inffus.2020.09.005

Information Fusion

Volume 66, February 2021, Pages 184-197

https://doi.org/10.1016/j.inffus.2020.09.005 Get rights and content

Highlights

•
Cross-modal interactions spread across a multimodal sequence.
•
Multimodal models struggle with positive utterances.
•
Very deep representations do not always result in better multimodal embeddings.
•
Linguistic modality should work a pivot for the nonverbal modalities.
•
The integration of modality context yields improved performance.

Abstract

Multimodal video sentiment analysis is a rapidly growing area. It combines verbal (i.e., linguistic) and non-verbal modalities (i.e., visual, acoustic) to predict the sentiment of utterances. A recent trend has been geared towards different modality fusion models utilizing various attention, memory and recurrent components. However, there lacks a systematic investigation on how these different components contribute to solving the problem as well as their limitations. This paper aims to fill the gap, marking the following key innovations. We present the first large-scale and comprehensive empirical comparison of eleven state-of-the-art (SOTA) modality fusion approaches in two video sentiment analysis tasks, with three SOTA benchmark corpora. An in-depth analysis of the results shows that the attention mechanisms are the most effective for modelling crossmodal interactions, yet they are computationally expensive. Second, additional levels of crossmodal interaction decrease performance. Third, positive sentiment utterances are the most challenging cases for all approaches. Finally, integrating context and utilizing the linguistic modality as a pivot for non-verbal modalities improve performance. We expect that the findings would provide helpful insights and guidance to the development of more effective modality fusion models.

Introduction

Human language is inherently multimodal and is manifested via words (i.e., linguistic modality), gestures (i.e., visual modality), and vocal intonations (i.e., acoustic modality). Consequently, we need to process both verbal (e.g., linguistic utterances) and nonverbal signals (e.g., visual, acoustic utterances) to better understand human language. Verbal signals often vary dynamically in different nonverbal contexts. Even though for humans, comprehending human language is an easy task, this is a non-trivial challenge for machines. Giving machines the capability to understand human language effectively opens new horizons for human–machine conversation systems [1], tutoring systems [2], and health care [3], to name a few applications.

The challenge of modelling human language lies in coordinating time-variant modalities. At its core, this research area focuses on modelling intramodal and crossmodal dynamics [4]. Intramodal dynamics refer to interactions within a specific modality, independent of other modalities. An example is word interactions in a sentence. Crossmodal dynamics refer to interactions across several modalities, for example, a simultaneous presence of a negative word, a frown, and a soft voice. Such interactions, occurring at the same time step, are called synchronous crossmodal interactions. Crossmodal interactions might span over a long-range multimodal sequence and are called asynchronous crossmodal interactions. For example, the negative word with the soft voice at the time step $t$ might interact with the frown at the time step $t + 1$ .

Early approaches for learning multimodal representations have widely utilized conventional natural language processing (NLP) techniques in multimodal settings [5], [6], [7], [8]. A recent trend in multimodal embedding learning research is to build more complex models utilizing attention, memory, and recurrent components [9], [10], [11], [12], [13], [14], [15]. Various review papers have surveyed the advancements in multimodal machine learning [16], [17], [18], [19], [20]. In particular, they mostly provide an insightful organization of modality fusion strategies. They also identify broader challenges faced by multimodal representation learning, such as synchronization across different modalities, confidence level, contextual information, etc. However, none of them has conducted a comprehensive empirical study across different state-of-the-art (SOTA) fusion approaches to multimodal language analysis, intending to provide critical and experimental analysis. Such an extensive empirical evaluation would be useful to find out which aspects in the SOTA approaches are the most effective in solving the problem of multimodal language analysis. This paper aims to fill the gap. In particular, we replicate and evaluate the most recent SOTA fusion approaches for modelling human language on three widely used benchmark corpora for multimodal sentiment and emotion analysis [21], [22], [23], and investigate the following Research Questions (RQ).

•
RQ1 How effective are the current machine learning-based multimodal fusion strategies for the sentiment analysis and emotion recognition tasks?
•
RQ2 How efficient are the SOTA multimodal fusion strategies, and how could the effectiveness affect efficiency, in the context of the multimodal sentiment and emotion analysis tasks?
•
RQ3 Which components/aspects in the multimodal language models and fusion strategies are the most effective?

The rest of the paper is organized as follows: Section 2 briefly reviews the related work. Section 3 describes the experiments in detail. The experimental results are shown and discussed in Sections 4 Results, 5 Discussion on key findings, respectively. Finally, Section 6 concludes the paper.

Section snippets

Related work

In this section, we provide a review of multimodal representation learning and multimodal time series for video sentiment analysis and emotion recognition.

Methodology

This section details the methodology we used for our empirical study of the most recent SOTA multimodal language fusion approaches, in the context of video sentiment and emotion analysis tasks. We first formulated the task on which our study was carried out. Sentiment analysis was a binary multimodal classification task inferring either positive or negation emotions. Emotion recognition was a multimodal multilabel classification task inferring one or more emotions, e.g., happy and joyful.

Effectiveness

In Table 1, we see that attention mechanism-based approaches, namely, MulT, MMUU-BA, and RAVEN, exhibit the highest binary accuracy (between 78.2% and 78.7%) on MOSI. MulT reports just 0.1% higher accuracy than RAVEN. Yet, for $A c c_{7}$ , RAVEN reports an increased performance of 34.6% as compared to 33.8% for MMUU-BA and 33.6% for MulT. TFN attained the highest accuracy of 34.9% for $A c c_{7}$ . Raven and MMUU-BA report the highest correlation ( $C o r r$ ). Despite the low accuracy, MCTN exhibits the lowest mean

Discussion on key findings

In this paper, we replicated the most recent SOTA models for multimodal language analysis. We evaluated their effectiveness through comprehensive comparative studies, error analyses and series of ablation studies. The efficiency of the models was also compared in terms of three evaluation metrics, namely, parameters, training time, and validation set convergence. The results associated with ablation studies helped us determine which components and methodologies contribute most to solving the

Conclusions

We have replicated and proposed a large-scale empirical comparison among SOTA approaches for multimodal human language analysis. We thoroughly investigated both their effectiveness and efficiency on two human multimodal affection recognition tasks and determined important components in multimodal language models. The results showed that attention mechanism approaches are the most effective for both sentiment analysis and emotion recognition tasks, even though they are not computationally cheap.

CRediT authorship contribution statement

Dimitris Gkoumas: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Writing - original draft, Visualization. Qiuchi Li: Software. Christina Lioma: Writing - review & editing, Supervision. Yijun Yu: Writing - review & editing, Supervision. Dawei Song: Writing - review & editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This study is supported by the Quantum Information Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321, and Natural Science Foundation of China (grant No.: U1636203).

References (77)

PoriaS. et al.
A review of affective computing: From unimodal analysis to multimodal fusion
Inf. Fusion
(2017)
JamesA.P. et al.
Medical image fusion: A survey of the state of the art
Inf. Fusion
(2014)
ChaturvediI. et al.
Fuzzy commonsense reasoning for multimodal sentiment analysis
Pattern Recognit. Lett.
(2019)
ZhangY. et al.
Deep fusion: An attention guided factorized bilinear pooling for audio-video emotion recognition
YoungT. et al.
Dialogue systems with audio context
Neurocomputing
(2020)
WuW. et al.
Proactive human-machine conversation with explicit conversation goal
PhamP. et al.
Predicting learners’ emotions in mobile MOOC learning via a multimodal intelligent tutor
PrangeA. et al.
Multimodal speech-based dialogue for the mini-mental state examination
RajagopalanS.S. et al.
Extending long short-term memory for multi-view structured learning
PoriaS. et al.
Convolutional MKL based multimodal emotion recognition and sentiment analysis

PoriaS. et al.

Context-dependent sentiment analysis in user-generated videos

WangH. et al.

Select-additive learning: Improving generalization in multimodal sentiment analysis

ZadehA. et al.

Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages

IEEE Intell. Syst.

(2016)

TsaiY.H. et al.

Multimodal transformer for unaligned multimodal language sequences

GhosalD. et al.

Contextual inter-modal attention for multi-modal sentiment analysis

GuY. et al.

Multimodal affective analysis using hierarchical attention strategy with word-level alignment

LiangP.P. et al.

Multimodal language analysis with recurrent multistage fusion

ZadehA. et al.

Memory fusion network for multi-view sequential learning

PhamH. et al.

Found in translation: Learning robust joint representations by cyclic translations between modalities

MaiS. et al.

Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion

AtreyP.K. et al.

Multimodal fusion for multimedia analysis: a survey

Multimedia Syst.

(2010)

SunS.

A survey of multi-view machine learning

Neural Comput. Appl.

(2013)

RamachandramD. et al.

Deep multimodal learning: A survey on recent advances and trends

IEEE Signal Process. Mag.

(2017)

BaltrusaitisT. et al.

Multimodal machine learning: A survey and taxonomy

IEEE Trans. Pattern Anal. Mach. Intell.

(2019)

ZadehA. et al.

MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos

(2016)

ZadehA. et al.

Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph

BussoC. et al.

IEMOCAP: interactive emotional dyadic motion capture database

Lang. Resour. Eval.

(2008)

DonahueJ. et al.

Long-term recurrent convolutional networks for visual recognition and description

MorencyL. et al.

Towards multimodal sentiment analysis: harvesting opinions from the web

GhoshS. et al.

Representation learning for speech emotion recognition

AntolS. et al.

VQA: Visual question answering

BokhariM.U. et al.

Multimodal information retrieval: Challenges and future trends

Int. J. Comput. Appl.

(2013)

PoriaS. et al.

MELD: A multimodal multi-party dataset for emotion recognition in conversations

NorooziF. et al.

Survey on emotional body gesture recognition

IEEE Trans. Affect. Comput.

(2018)

D’melloS.K. et al.

A review and meta-analysis of multimodal affect detection systems

ACM Comput. Surv.

(2015)

MorvantE. et al.

Majority vote of diverse classifiers for late fusion

ShutovaE. et al.

Black holes and white rabbits: Metaphor identification with visual features

GlodekM. et al.

Multiple classifier systems for the classification of audio-visual emotional states

Cited by (57)

DJMF: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamics
2024, Expert Systems with Applications
Multimodal sentiment analysis (MSA) is a considerable research area within the domain of social networks where users continuously share a vast amount of content. This content includes text, images, and audio, all of which are accompanied with sentiment or emotion expressions. MSA encompasses multiple tasks, such as sentiment prediction and emotion recognition, which have multi-task dynamic relationships owing to their close correlation and complementary nature. However, previous works using a single-task framework have been limited in exploring these dynamics. In this work, we propose a discriminative joint multi-task framework (DJMF) realizing sentiment prediction and emotion recognition synchronously. The discriminative learning strategy takes into account intra-task dynamics, and customized fusion schemes are applied to the subtasks. Furthermore, we utilize a joint training strategy to capture the inter-task dynamics by leveraging their interrelatedness, enhancing the performance of each task. Additionally, in the fusion process, we incorporate two categories of regularizers to enable the model to obtain distinct fusion results by reducing redundancy and task-irrelevant information. We conducted extensive evaluations on three well-known multimodal baseline datasets: CMU-MOSI, CMU-MOSEI, and UR-FUNNY. The results demonstrate that our approach offers better reliability and stability compared to the single-task framework and outperforms state-of-the-art methods.
Adapt and explore: Multimodal mixup for representation learning
2024, Information Fusion
Research on general multimodal systems has gained significant attention due to the proliferation of multimodal data in the real world. Despite the remarkable performance achieved by existing multimodal representation learning schemes, missing modalities remain a persistent issue, thereby limiting the overall applicability of multimodal systems. Intending to address the issue, we propose a novel approach named M $^{3}$ ixup (Multi-Modal Mixup), which leverages the mixup strategy to improve unimodal and multimodal representation learning while simultaneously increasing robustness against missing modalities. First, we adopt productive multimodal learning scheme to model representations with modality-specific and joint-modality encoders. The general scheme ensuring the proposed approach transferable for various multimodal learning scenarios, including supervised, unsupervised, and reinforcement learning. Then, the unimodal input and manifold mixup is used to enhance the modality-specific encoders to capture intra-modal dynamics. Next, we present multimodal mixup to mix different modalities and generate mixed multimodal representations in adapting and exploring steps. The former step aims at bridging the huge information gaps between unimodal and multimodal representations in the joint space in the alignment, while the latter step further captures the inter-modal dynamics and exploits the non-linear relationships among different modalities. After that, the mixed views are aligned with the original multimodal representations by contrastive learning. Additionally, we innovatively extend the mixup strategy to the loss function of multimodal contrastive learning in two steps to improve the alignment between mixed and original views. Extensive experiments on public datasets in various multimodal learning scenarios demonstrate the superiority of the proposed M $^{3}$ ixup. The codes are available at https://github.com/RH-Lin/m3ixup.
Capsule network-based deep ensemble transfer learning for multimodal sentiment analysis
2024, Expert Systems with Applications
Understanding the attitudes of users about different topics through their huge amount of comments and opinions on social networks is an emerging hot issue. Analyzing the sentiments of these posts and comments provides useful information utilized for many applications. Multimodal sentiments often contain more information than single-modal sentiments, including text, image, audio and video, which leads to better performance of multimodal sentiment analysis (MSA) compared to single-modal sentiment analysis (SSA). In this paper, a capsule network-based deep ensemble transfer learning approach, called DSY-ETL-MSA, is proposed for MSA on images and texts, the results of which are fused using Yager theory. Capsule networks and ensemble learning methods improve classification performance, and the deep transfer learning approach reduces the training time. A hybrid deep architecture is used for automatic feature extraction in the proposed method. For analyzing the sentiment of image modality, a pre-trained VGG16 model, fine-tuned on the datasets, is used to extract high-level features for image classification. A capsule convolutional neural network (CNN) is also separately used for extracting image features and classifying them. In the text modality, the pre-trained GloVe model is exploited to embed words and 2 separate classifiers are employed for text classifications. The Yager fusion rules are finally used for early and late fusions of the results of the classifiers. The results of the text and image classifiers are combined separately as the early fusion and the final results of them are fused at the decision level as the late fusion. The proposed model is empirically evaluated on the MVSA and T4SA datasets. The significant performance improvement compared to a variety of former methods is shown in the experimental results. The final accuracies obtained by the proposed method on the MVSA and T4SA datasets are 0.9866 and 0.9996, respectively.
Progress, achievements, and challenges in multimodal sentiment analysis using deep learning: A survey
2024, Applied Soft Computing
Sentiment analysis is a computational technique that analyses the subjective information conveyed within a given expression. This encompasses appraisals, opinions, attitudes or emotions towards a particular subject, individual, or entity. Conventional sentiment analysis solely considers the text modality and derives sentiment by identifying the semantic relationship between words within a sentence. Despite this, certain expressions, such as exaggeration, sarcasm and humor, pose a challenge for automated detection when conveyed only through text. Multimodal sentiment analysis incorporates various forms of data, such as visual and acoustic cues, in addition to text. By utilizing fusion analysis, this approach can more precisely determine the implied sentiment polarity, which includes positive, neutral, and negative sentiments. Thus, the recent advancements in deep learning have boosted the domain of multimodal sentiment analysis to new heights. The research community has also shown significant interest in this topic due to its potential for both practical application and educational research. In light of this fact, this paper aims to present a thorough analysis of recent ground-breaking research studies conducted in multimodal sentiment analysis, which employs deep learning models across various modalities such as text, audio, image, and video. Furthermore, the article dives into a discussion of the multiple categories of multimodal data, diverse domains in which multimodal sentiment analysis can be applied, a range of operations that are integral to multimodal sentiment analysis, deep learning architectures, a variety of fusion methods, challenges associated with multimodal sentiment analysis, and the benchmark datasets in addition to the state-of-the-art approaches. The ultimate goal of this survey is to indicate the success of deep learning architectures in tackling the complexities associated with multimodal sentiment analysis.
Co-space Representation Interaction Network for multimodal sentiment analysis
2024, Knowledge-Based Systems
Developing an effective multimodal representation has always been the crux of multimodal sentiment analysis. Different modalities possess distinct sentiment attributes between modality-invariant and modality-specific representation spaces. Prior studies have concentrated on utilizing intricate networks to directly generate joint representations of three modalities and lack exploiting relationships of the two representation spaces. To mitigate this, (1) we introduce a novel framework Co-space Representation Interaction Network (CRNet) that leverages different acoustic and visual representation subspaces to interact with linguistic modality. (2) To construct a joint representation through coordinated acoustic and visual spaces with linguistic modality, a novel module named Gradient-based Representation Enhancement (GRE) is proposed which is effective for extracting significant variation of representation matrices. (3) we design a novel multi-task strategy to optimize the training process to improve the performance of different representation combinations that come from the two spaces. Experimental results demonstrate that our suggested framework achieves state-of-the-art (SOTA) performance on CMU-MOSI, CMU-MOSEI and CH-SIMS datasets.
Learning from the global view: Supervised contrastive learning of multimodal representation
2023, Information Fusion
The development of technology enables the availability of abundant multimodal data, which can be utilized in many representation learning tasks. However, most methods ignore the rich modality correlation information stored in each multimodal object and fail to fully exploit the potential of multimodal data. To address the aforementioned issue, cross-modal contrastive learning methods are proposed to learn the similarity score of each modality pair in a self-/weakly-supervised manner and improve the model robustness. Though effective, contrastive learning based on unimodal representations might be, in some cases, inaccurate as unimodal representations fail to reveal the global information of multimodal objects. To this end, we propose a contrastive learning pipeline based on multimodal representations to learn from the global view, and devise multiple techniques to generate negative and positive samples for each anchor. To generate positive samples, we apply the mix-up operation to mix two multimodal representations of different objects that have the maximal label similarity. Moreover, we devise a permutation-invariant fusion mechanism to define the positive samples by permuting the input order of modalities for fusion and sampling various contrastive fusion networks. In this way, we force the multimodal representation to be invariant regarding the order of modalities and the structures of fusion networks, so that the model can capture high-level semantic information of multimodal objects. To define negative samples, for each modality, we randomly replace the unimodal representation with that from another dissimilar object when synthesizing the multimodal representation. By this means, the model is led to capture the high-level concurrence information and correspondence relationship between modalities within each object. We also directly define the multimodal representation from another object as a negative sample, where the chosen object shares the minimal label similarity with the anchor. The label information is leveraged in the proposed framework to learn a more discriminative multimodal embedding space for downstream tasks. Extensive experiments demonstrate that our method outperforms previous state-of-the-art baselines on the tasks of multimodal sentiment analysis and humor detection.

View all citing articles on Scopus

View full text

Full length articleWhat makes the difference? An empirical comparison of fusion strategies for multimodal language analysis

Highlights

Abstract

Introduction

Section snippets

Related work

Methodology

Effectiveness

Discussion on key findings

Conclusions

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgements

Inf. Fusion

Inf. Fusion

Pattern Recognit. Lett.

Neurocomputing

Proactive human-machine conversation with explicit conversation goal

Predicting learners’ emotions in mobile MOOC learning via a multimodal intelligent tutor

Multimodal speech-based dialogue for the mini-mental state examination

Extending long short-term memory for multi-view structured learning

Convolutional MKL based multimodal emotion recognition and sentiment analysis

Context-dependent sentiment analysis in user-generated videos

Select-additive learning: Improving generalization in multimodal sentiment analysis

Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages

IEEE Intell. Syst.

Multimodal transformer for unaligned multimodal language sequences

Contextual inter-modal attention for multi-modal sentiment analysis

Multimodal affective analysis using hierarchical attention strategy with word-level alignment

Multimodal language analysis with recurrent multistage fusion

Memory fusion network for multi-view sequential learning

Found in translation: Learning robust joint representations by cyclic translations between modalities

Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion

Multimodal fusion for multimedia analysis: a survey

Multimedia Syst.

A survey of multi-view machine learning

Neural Comput. Appl.

Deep multimodal learning: A survey on recent advances and trends

IEEE Signal Process. Mag.

Multimodal machine learning: A survey and taxonomy

IEEE Trans. Pattern Anal. Mach. Intell.

MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos

Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph

IEMOCAP: interactive emotional dyadic motion capture database

Lang. Resour. Eval.

Long-term recurrent convolutional networks for visual recognition and description

Towards multimodal sentiment analysis: harvesting opinions from the web

Representation learning for speech emotion recognition

VQA: Visual question answering

Multimodal information retrieval: Challenges and future trends

Int. J. Comput. Appl.

MELD: A multimodal multi-party dataset for emotion recognition in conversations

Survey on emotional body gesture recognition

IEEE Trans. Affect. Comput.

A review and meta-analysis of multimodal affect detection systems

ACM Comput. Surv.

Majority vote of diverse classifiers for late fusion

Black holes and white rabbits: Metaphor identification with visual features

Multiple classifier systems for the classification of audio-visual emotional states

Full length article
What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis