Exploring Multimodal Features to Understand Cultural Context for Spontaneous Humor Prediction

Abstract

This study aims to predict humor in a binary label, i.e., the presence or absence of humor in video recordings. The challenge here is to predict the variable in a cross-cultural manner, where the training data is in German, and the testing is done on the recordings of English-language-speaking football coaches. The novelty of this paper lies in exploring audio and textual features to predict humor in a cross-cultural setting. It is interesting to study audio and text-based features due to the cross-cultural nature of the problem, which remains largely unexplored when studying pose and facial features. The paper explores several audio (mms-lid, wav2vec 2.0) and textual (LaBSE, multilingual-e5-base) features and then uses them to train both RNN and Transformer encoder architectures. Experiments have been performed on the transformer encoder architectures with and without position encoding to study the effects of the absence of positional encoding in those features for humor detection. Late fusion has also been studied with combinations of all three modalities. We achieved our best AUC Score of 0.9251 and 0.8245 for the development and test set, respectively, out of five given submissions.

This research was supported by the Ministry of Science and ICT (MSIT) Korea under the National Research Foundation (NRF) Korea (NRF-2022R1A2C4001270), by the MSIT Korea Korea under the India-Korea Joint Programme of Cooperation in Science & Technology (NRF-2020K1A3A1A68093469), and by the ITRC (Information Technology Research Center) support program (IITP-2022-2020-0-01602) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).