Audio-Visual Source Separation Based Fusion Techniques

In this work we introduced a hybrid deep learning method for audio-visual speech separation. Our approach is speaker-independent and leverages a multi-stream architecture to analyze visual streams containing detected faces alongside with auditory inputs. By integrating both visual and audio information, our model aims to effectively separate speech signals from complex audio-visual mixtures involving two speakers. A combination of Short-Time Fourier Transform (STFT) and Mel-Frequency Cepstral Coefficients (MFCCs) is used for audio feature extraction. The STFT is a technique for analyzing a signal's frequency content throughout short, overlapping time intervals it provides a time-localized frequency information when the frequency components of a signal fluctuate over time [26], while MFCCs possess the capability to effectively capture significant components of the spectral composition of audio signals, namely within the frequency ranges that hold the utmost relevance to human auditory perception [27]. Principal Component Analysis (PCA) is a dimension reduction technique with the primary objective of effectively reducing the dimensionality of a given dataset and identifying its most important elements by transforming it into a new, uncorrelated space defined by principle components [28]. It reduces computing complexity, training time, and overfitting risks for high-dimensional data. STFT and MFCC help extract robust characteristics, especially in speech and speaker recognition. STFT and MFCC are vital to identify speaker features and adjust to different acoustic settings. High dimensionality in short-time Fourier transform (STFT) analysis is addressed by MFCC. A logarithmic compression and mel-scale filterbank are applied to the STFT magnitude spectrum to compress and efficiently represent the audio signal. This combination also allows multimodal fusion, which integrates visual signals to improve performance. Lossy compression is used to convert the MFCC's two-dimensional output into a one-dimensional vector representation using vector quantization. As we are dealing with large dataset Principal Component Analysis (PCA) is used for visual feature extraction before the deep learning model. This method reduces computing complexity, training time, and overfitting risks for high-dimensional data. PCA reduces noise and speeds training convergence by prioritizing signal over noise. By reducing the correlation between highly correlated features, it successfully addresses multicollinearity. The audio and visual feature vectors that are first extracted are subsequently combined and passed to the deep model. For audio-visual source separation, the proposed deep modal is made up of several layers and is mainly based on combining one-dimensional convolutional neural networks (1D CNNs) and long short-term memory (LSTMs). It is more reliable to use 1D CNNs for feature extraction because they can perform translation invariance and multimodal integration. This is because they are better at capturing localized patterns and temporal hierarchies in sequential input. The integration of LSTM enhances the model's ability to capture long-term dependencies and understand sequential context, which is of utmost importance in source separation tasks. LSTM memory cells store and utilize information effectively over long sequences, which make it easier for 1D CNNs to learn local patterns. The output of the deep model is a set of complex spectrogram masks, each of which corresponds to a detected face in the video (the proposed model is illustrated in Figure 1) We can isolate the speech signals of each speaker while efficiently suppressing any interfering signals from other sources by multiplying the noisy input spectrograms with these masks. This method allows for a distinct separation of each speaker’s voices.

Figure 1. Proposed framework

The mixture audio signal in the time domain is given by:

$x(t)=s_1(t)+s_2(t)$        (1)

The spectrogram of the audio signal in the frequency domain at frequency bin $f$ and time frame $\tau$ is given by:

$X(f, \tau)=S_1(f, \tau)+S_2(f, \tau)$                  (2)

We predict complex ratio masks (CRM) to accomplish separation, separating the corresponding speakers' clean speech from the mixture x.

The estimated sound sources in the time domain at time t are given by:

$\begin{aligned} & \hat{s}_1(t)=x(t) * m_1(t) \\ & \hat{s}_2(t)=x(t) * m_2(t)\end{aligned}$                  (3)

The estimated sound sources in the frequency domain are given by:

$\begin{aligned} & \hat{S}_1(f, \tau)=X(f, \tau) * M_1(f, \tau) \\ & \hat{S}_2(f, \tau)=X(f, \tau) * M_2(f, \tau)\end{aligned}$                 (4)

3.1 Audio and video data representation

Our model takes both audio and visual data as input. We calculate the short-time Fourier transform (STFT) for 3-second audio segments to derive audio features.

After the STFT we calculate Mel-frequency cepstral coefficients (MFCC) for further audio spectral feature extraction, it captures sound signal spectral features by converting the audio spectrum into a compact, perceptually useful form. Each time-frequency (TF) bin comprises both the real and imaginary components of a complex number, both of which are utilized as input. Additionally, we apply power-law compression, which involves applying a non-linear transformation to the audio signal in order to reduce amplitude variations between loud and quiet parts, hence preventing distortion and clipping while retaining perceived quality to prevent loud noise from overpowering faint audio signals [29]. Furthermore, to ensure consistency, the same processing is done to both the noisy signal and the clean reference signal.

Given a video recording that may contain multiple speakers, first we convert the video into 75 frames per speaker, and then convert to grayscale images with a size of 128 × 720. The conversion to gray scale is highly sufficient since we are dealing with large number of image frames thus less processing cost [30].

For the face detection part, we use the multitask cascaded convolutional neural network (MTCNN) [31], which is designed to accurately detect faces and localize facial landmarks, such as the eyes, nose, and mouth in the images, we perform face detection in each frame (75 frames for each speaker in a 3-second video at 25 FPS) and extract face embedding for each face detected per frame. After face detection the frames are resized to a size of 160 × 160. The reasoning is that these embeddings preserve crucial information for recognizing countless faces while eliminating unimportant variations, like lighting differences between photos. For visual features extraction we used Principal component analysis (PCA) to characterize the pattern with the fewest number of features and to lower the dimensionality of the feature space without losing the most critical information for discriminating.

Our method generates a multiplicative spectrogram mask that represents the correlations between clean speech and background noise in both time and frequency. To achieve this, we employ a complex ratio mask (CRM), which quantifies the relationship between the complex clean and noisy spectrograms. The CRM consists of real and imaginary components, which are assessed independently in the real domain. This allows us to accurately separate the desired speech from unwanted interference, leading to improved audio quality [32]. The complex ratio mask is given by:

$\begin{gathered}M_{\text {real }}=\frac{X_{\text {real }} \cdot S_{\text {real }}+X_{\text {imag }} \cdot S_{\text {imag }}}{\left|X^2\right|+\epsilon} \\ M_{\text {imag }}=\frac{X_{\text {real }} \cdot S_{\text {imag }}-X_{\text {imag }} \cdot S_{\text {real }}}{\left|X^2\right|+\epsilon} \\ M=\left[M_{\text {real }}, M_{\text {imag }}\right]\end{gathered}$                           (5)

3.2 Network architecture

In our proposed model, both audio and visual features are processed jointly. After preprocessing the visual and auditory data, we fuse the extracted features using concatenation fusion, combining these features before passing them to the deep model (as depicted in Figure 2). Our deep learning model consists of a total 36 layers, 11 Convolutional layers with filter size of (16, 32, 64, 128, 256, 256, 512, 1024, 512, 128, 128) respectively for feature extraction, 11 MaxPooling layers with pool size 1 and stride of 1 to reduce the spatial size of the extracted features, 8 LeakyRelu layers with alpha of value 0.3, 2 LSTM layers to learn the long term dependencies between the sequenced data, 2 dense layers with linear activation functions which we used as a collector for strong and close to strong features, and one dense layer that works as fully connected layer with softmax as activation function and finally one flatten layer that converts the multidimensional data into a one dimensional so it can be passed to the fully connected layer. The model deep learning layers specifications and parameters are listed in Table 2.

Figure 2. Deep neural network architecture