Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

Abstract

Multimodal sentiment analysis models can determine users’ sentiments by utilizing rich information from various sources (e.g., textual, visual, and audio). However, there are two key challenges when deploying the model in real-world environments: (1) the limitations of relying on the performance of automatic speech recognition (ASR) models can lead to errors in recognizing sentiment words, which may mislead the sentiment analysis of the textual modality, and (2) variations in information density across modalities complicate the development of a high-quality fusion framework. To address these challenges, this paper proposes a novel Multimodal Sentiment Word Optimization Module and a heterogeneous hierarchical fusion (MSWOHHF) framework. Specifically, the proposed Multimodal Sentiment Word Optimization Module optimizes the sentiment words extracted from the textual modality by the ASR model, thereby reducing sentiment word recognition errors. In the multimodal fusion phase, a heterogeneous hierarchical fusion network architecture is introduced, which first utilizes a Transformer Aggregation Module to fuse the visual and audio modalities, enhancing the high-level semantic features of each modality. A Cross-Attention Fusion Module then integrates the textual modality with the audiovisual fusion. Next, a Feature-Based Attention Fusion Module is proposed that enables fusion by dynamically tuning the weights of both the combined and unimodal representations. It then predicts sentiment polarity using a nonlinear neural network. Finally, the experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets show that the MSWOHHF outperforms several baselines, demonstrating better performance.

1. Introduction

With the rapid growth of social media and short video content, the online information related to emotions has greatly increased [1]. This has attracted widespread attention to multimodal sentiment analysis (MSA) and its applications in many fields (e.g., sentiment recognition and emotion-intelligent robots [2]). Unlike the traditional unimodal sentiment analysis, MSA extracts users’ emotional features from various modal inputs (i.e., textual, visual, audio, and others) [3]. Typically, modality interactions can enrich emotional semantics by utilizing complementary information [4]. However, the different modal information used in MSA is often heterogeneous [5]. Therefore, a key issue in MSA is how to design a multimodal fusion scheme that efficiently integrates heterogeneous data [6], thereby learning multimodal representations containing more emotion-related information while maintaining the consistency and differentiation of each modality.

To enhance fusion representations, earlier studies like CATF-LSTM [7] emphasized designing novel LSTM architectures and attention mechanisms to capture the contextual information within each modality. With the emergence of Transformers, models such as CTFN [8] and MulT [9] introduced modality translation using Transformer-based structures, facilitating cross-modal encoding. Moreover, several representation-learning-based methods (e.g., MISA [5] and Self-MM [10]) have been developed to capture features by leveraging both consistent and divergent information across modalities, aiming to enhance the accuracy of MSA. However, these approaches often assign equal importance to all the modalities, neglecting the varying semantic depth each modality offers. This can lead to undervaluing dominant modalities while overemphasizing weaker ones. Given the distinctive features of each modality in sentiment analysis, the resulting fusion representations may lack critical emotion-related information, ultimately reducing the model performance. Previous studies (e.g., [11,12]) emphasize the dominant role of textual emotional features in MSA tasks, which provide highly structured and semantically rich features and further improve the text modeling techniques in natural language processing. However, in practical scenarios, a large amount of text from ASR systems may lead to errors in recognizing emotional words, affecting the accuracy of MSA models in determining emotional polarity. In summary, the main challenges of multimodal sentiment analysis in the real world include the following: (1) the incorrect recognition of emotional words and (2) the impact of different modes on the emotion analysis being imbalanced.

To address the above challenges, we propose a new MSWOHHF framework. This framework includes four core units: the Multimodal Sentiment Word Optimization Module, the Transformer Aggregation Module, the Cross-Attention Fusion Module, and the Feature-Based Attention Fusion Module.

In the Multimodal Sentiment Word Optimization Module, we mitigate the negative impact of ASR errors by employing a language model that captures both syntactic and semantic information. This allows us to predict the most probable positions of sentiment words. By dynamically reconstructing incomplete sentiment semantics, the model generates new word vectors using multimodal sentiment information.

The Transformer Aggregation Module is designed to elevate the low-level features of audio and visual modalities by converting them into higher-level representations. Initially, the module organizes and compresses information from both modalities. These integrated data are then processed through the aggregation module, enabling complementary feature learning and addressing disparities in the information density between the two modalities.

The Cross-Attention Fusion Module improves the effectiveness of fusion in multimodal sentiment analysis by referencing a learnable matrix. This helps the model to better capture interactions and emotional information across modalities. The attention-based interrelations facilitate the capture of semantic correlations between audiovisual fusion and textual features.

In the Feature-Based Attention Fusion Module, we integrate the representations from the three modalities through vector concatenation. This approach consolidates the features from each modality into a single unified representation. Next, we implement a feature-based attention mechanism that dynamically adjusts the weight of each dimension within these representations. This mechanism identifies and emphasizes the most significant features by establishing correlations between dimensions, thereby optimizing the fusion of multimodal data. The module enhances the overall fusion process by focusing on the key features and filtering out less relevant information. The refined joint representation, resulting from this attention mechanism, is then utilized to generate sentiment predictions, improving the accuracy and reliability of the model’s outputs.

The main contributions of this paper are as follows:

• We propose the MSWOHHF model, which enhances the robustness of sentiment prediction by dynamically completing the emotional semantics of damaged ASR textual modalities through a Multimodal Sentiment Word Optimization Module.
• We develop a novel heterogeneous hierarchical multimodal fusion network to facilitate effective interaction among the three modalities, each with varying information densities. This network incorporates a fusion module that utilizes attention aggregation and cross-modal attention mechanisms, enabling fair and efficient complementary learning across modalities.
• To evaluate each modality’s significance during the fusion process, we design a feature-based attention module that dynamically adjusts the weighting of its representation.

2. Related Work

In recent years, MSA has garnered increasing interest and has become a significant research field [13]. It primarily focuses on extracting human emotions from complex multimedia data comprising text, visual, and speech components. Before multimodal information fusion can occur, it is essential to represent heterogeneous multimodal data. Extensive research has been conducted on modality-specific representations, ranging from handcrafted designs tailored for specific applications to data-driven approaches [2]. For textual feature representation, the commonly used word embedding models like Google’s Word2Vec model [14] mainly rely on either Skip-grams or the CBOW models. For instance, Yang et al. [15] used the Word2Vec model for feature extraction in the textual modality. GloVe [16] word vectors utilize a co-occurrence matrix to capture global information, while ELMo [17] word vectors can capture context-related meanings in words as the linguistic context changes. In 2018, Google introduced the BERT [18] pre-training model, which has been widely adopted by scholars who use large-scale corpora for pre-training to learn semantic relationships before applying it to downstream tasks for word vector input. Chen et al. [19] used both GloVe word vectors and the BERT model for textual feature representation and compared their performance. For visual feature representation in MSA, researchers often use specialized tools like Facet and OpenFace [20] to capture facial emotions. Manually crafted acoustic features like Mel-frequency cepstral coefficients (MFCC) are frequently used in the audio domain. Multimodal representation, which builds upon individual modal features, can be achieved through either straightforward concatenation or more sophisticated deep neural networks (DNNs) [21,22,23,24]. Xu et al. [25] introduced the MultiSentiNet model, which employs LSTM and attention mechanisms guided by visual features to fuse multimodal data.

The early research on multimodal fusion often relied on either early fusion, where the raw features from multiple sources were combined directly [26,27], or late fusion, where the decisions from multiple sentiment classifiers were aggregated [28]. However, early fusion can result in redundant input vectors and increased computational complexity, while late fusion struggles to capture cross-modal correlations. Various deep fusion and multistage fusion strategies have been introduced to overcome these limitations for multimodal sentiment analysis. For instance, Zadeh et al. [29] proposed a Tensor Fusion Network (TFN) model that leverages tensor representations and operations to better capture dynamic properties and interactions across modalities. However, the computational demands of three-dimensional tensors grow exponentially with increasing feature dimensions. To address this, Liu et al. [30] developed Low-Rank Multimodal Fusion (LMF), which reduces the complexity of high-order tensors using low-rank decomposition, although it overlooks local interaction modeling. The previous methods often treated each utterance as independent, neglecting the dependencies between the utterances within a video. Advances have occurred regarding models that incorporate contextual information using recurrent neural networks (RNNs) [31,32]. Poria et al. [33] introduced the BC-LSTM model to capture the contextual dependencies between utterances in the same video, but this approach did not account for the varying importance of each utterance across different modalities. With the introduction of the Transformer architecture [34], multimodal applications have expanded. Tsai et al. [9] proposed MulT, a multimodal model that uses pairwise cross-modal attention in a directional manner, enhancing adaptability between modalities. Among the fusion techniques, the attention-based methods have proven to be highly efficient and effective [35]. Huddar et al. [36] calculated bimodal attention matrices separately and combined them into a trimodal attention matrix, enabling the fusion of interactive information across modalities. Similarly, Wu et al. [3] developed a bimodal-information-enhanced multi-head attention mechanism to study the relative importance of different modality pairs and to fuse multimodal data.

In addition, some researchers have explored the issues of noise reduction and missing modalities. Pham et al. [37] proposed the MCTN model to handle scenarios where there is a possibility of missing visual and acoustic data. Given the importance of textual modalities in multimodal sentiment analysis tasks, Lei et al. [38] introduced the TMRN model, which uses the textual modality as the primary thread, interacting with and enhancing the other two modalities to achieve low-redundancy and denoised feature representations. Chen et al. [11] proposed a gated multimodal embedding approach to filter out noise in acoustic and visual data. Zhu et al. [39] proposed the SKEAFN model, which constructs an emotion knowledge graph and performs graph computations for MSA tasks, designing an additional emotional knowledge generation module.

Although many excellent multimodal sentiment analysis models have been proposed, they often do not address textual acquisition in the current environments. Notably, Wu et al. [40] proposed the SWRM model, which utilizes an emotion word position detection module to determine the most likely locations of emotion words in the textual modality and dynamically completes the emotional semantics of word representations at these locations through a multimodal word refinement module. However, the model does not account for the differences in information density between modalities.

In this paper, we consider the differences in information density across different modalities and allow them to interact fairly with each other while also considering the impact of ASR models in real-world environments.

3. Methodology

Our proposed architecture can be divided into six sub-modules: (1) the modality feature extraction module uses BERT to extract textual features and $sLSTM$ to extract audio and visual features. BERT [18] can capture words’ complex relationships in context through bidirectional encoding. The LSTM network is the basic unit of the $sLSTM$ model structure. LSTM is often used for sequence data processing because it effectively leverages long-distance dependencies in sequential data without encountering the vanishing gradient problem that RNN faces. The $sLSTM$ model contains multiple hidden LSTM layers, each consisting of several LSTM units capable of handling long-term states. Audio features typically include pitch, intensity, speech rate, and tone, which can reflect the speaker’s emotional state. For example, a high pitch, fast speech rate, and louder volume may be associated with positive emotions such as excitement, while a low pitch and slow speech rate may be related to negative emotions such as sadness or disappointment. $sLSTM$ captures these changes by analyzing the temporal characteristics of the audio and extracting high-dimensional audio feature vectors for emotion recognition. Visual features include facial expressions, movements of the eyes and mouth, and head posture, all of which can directly or indirectly reflect emotions. $sLSTM$ processes the temporal data of the video to extract continuous facial expression changes, body posture, and other features, capturing visual sentiment cues. (2) Multimodal Sentiment Word Optimization Module: This module dynamically refines the sentiment word semantics using multimodal information. (3) Transformer Aggregation Module: This module enhances the low-level features of the audio and visual modalities into high-level features. (4) Cross-Attention Fusion Module: Based on cross-attention, this module helps to capture the semantic correlations between textual features and audiovisual fusion features. (5) Feature-Based Attention Fusion Module: This module integrates multi-source representations. (6) Prediction Module: This module generates sentiment regression labels. Figure 1 shows the overall architecture.

3.1. Multimodal Sentiment Word Optimization Module (MSWOM)

We use an MSWOM to improve the accuracy of ASR in recognizing emotional words. Figure 2 illustrates the process of this module. First, we sequentially mask each word $w_i$ in a sentence. The word is replaced with a special token [MASK]. Second, a Multimodal Gating Network (MGN) is employed to filter out irrelevant sentence information. At the same time, a Multimodal Sentiment Word Attention Network (MSWAN) is designed to incorporate valuable information from candidate words generated by the BERT model, resulting in optimized word embeddings. The unaligned representations of three modes—word embeddings, visual features, and audio features—are represented as $x^m=\{x_l^m:1\le l\le n_i,x_l^m\in {ℝ}^{d_x^m}\},m\in \{t,v,a\}$. To acquire the multimodal information corresponding to each word, a pseudo-alignment method is applied to align textual features with visual and audio features. The audio and visual features are then divided into non-overlapping segments of lengths $\lfloor n_a/n_t\rfloor$ and $\lfloor n_v/n_t\rfloor$, respectively. The average is taken for each group of features, yielding the pseudo-aligned feature representation ${\mathrm{u}}^i=\{u_l^i:1\le l\le n_l,u_l^i\in {ℝ}^{d_x^i}\},i\in \{v,a\}$. Next, BERT and $sLSTM$ are used to model the textual, visual, and audio features separately, resulting in $s^i=\{s_l^i:1\le l\le n_i,s_l^i\in {ℝ}^{d_s^i}\},i\in \{t,v,a\}$ in Equations (1)–(3). Additionally, we employ a Transformer network to fuse audio and visual features, enabling the capture of high-level sentiment semantics and obtaining $s^{va}=\{s_l^{va}:1\le l\le n_i,s_l^{va}\in {ℝ}^{d_s^{va}}\}$ in Equation (4). The specific equation is as follows:

$$s^t=BER{T}_t(x^t)$$

(1)

$$s^v=sLST{M}_v(u^v)$$

(2)

$$s^a=sLST{M}_a(u^a)$$

(3)

$$s^{va}=Transforme{r}_{va}([s^v;s^a])$$

(4)

Subsequently, we employ an MGN to remove irrelevant information from the input sentence. Specifically, we connect the unimodal context-aware representations $s_p^t$, $s_p^v$, $s_p^a$, and the bimodal representation $s_p^{va}$, and the detected sentiment word position $p$ from the language model. These are fed into a linear neural network and through a sigmoid-activated function to generate a gate value $g^m$ in Equation (5), which applies to filter out irrelevant information from the word embeddings. To ensure that the model ignores improbable signals, we obtain the representation information $r^m$ of the position in which sentiment words are located by a gate mask $k$ in Equation (6).

$$g^m=Sigmoid(W_1([s_p^t;s_p^v;s_p^a;s_p^{va}])+b_1)$$

(5)

$$r^m=(1-g^{m}k)x_p^t$$

(6)

where $W_1\in {ℝ}^{1\times {\Sigma }_{i\in \{t,v,a,va\}}{d}_s^i},b_1\in {ℝ}^1$.

Additionally, we use an MSWAN to extract sentiment-related information from the candidate words selected by BERT model. First, at as many time steps $p$ as possible, we concatenate the word embeddings $x^{c_l^p}$ of the candidate word $c_l^p$ with the multimodal representations $s_p^v$, $s_p^a$, and $s_p^{va}$, and then through a linear layer to obtain the attention score $g_l^e$ in Equation (7). The attention scores are processed through a $softmax$ function to derive the attention weights ${\tau }_t^e$ in Equation (8). These weights are then applied to the candidate word embeddings, resulting in the sentiment embeddings $r^e$ in Equation (9).

$$g_l^e=W_2([x^{c_l^p};s_p^v;s_p^a;s_p^{va}])+b_2$$

(7)

$${\tau }_t^e=softmax(g_l^e)$$

(8)

$$r^e={\displaystyle \sum _{t=1}^k{\tau }_t^e{x}^{c_l^p}}$$

(9)

where $W_2\in {ℝ}^{1\times (d_x^t+{\Sigma }_{i\in \{v,a,va\}}{d}_s^i)},b_2\in {ℝ}^1$.

Additionally, since the desired sentiment words may not be included in the candidate word list, we introduce the representation of [MASK], $x^{mask}$, during the optimization of sentiment word representations. This allows the BERT model to handle the issue based on the context. We then design an aggregation network to balance the contributions of the special word embedding $x^{mask}$ and the sentiment embedding $r^e$. Finally, we combine the above representations to obtain the optimized word embedding function for the target word $r^t$. The specific equation is as follows:

$$g^{mask}=Sigmoid(W_3([r^e;x^{mask}])+b_3)$$

(10)

$$r^t=(g^{m}p)(g^{mask}{r}^e+(1-g^{mask})x^{mask})+r^m$$

(11)

where $W_3\in {ℝ}^{1\times 2d_x^t},b_3\in {ℝ}^1$.

3.2. Unimodal Features Extraction

After obtaining the new word embeddings $z^t=\{x_1^t,x_2^t,\dots ,r^t,\dots ,x_{nl}^t\}$ in MSWOM, we perform feature extraction using the BERT model to obtain $h^t$, which is provided as

$$h^t=BER{T}_{textual}(z^t)$$

(12)

We use two separate $sLSTM$ networks to extract visual and audio features, resulting in $h^v$ and $h^a$, which are expressed as

$$h^v=sLST{M}_{visual}(x^v)$$

(13)

$$h^a=sLST{M}_{audio}(x^a)$$

(14)

3.3. Transformer Aggregation Module (TAM)

Considering that visual and audio modalities have low-level semantic features, while textual features are high-level semantic features, we designed a low-level feature fusion mechanism within the TAM. This mechanism is intended to compensate for the differences in information density between different modalities, allowing them to be fused more equitably.

First, since the feature dimensions of the visual and audio modalities are inconsistent after being extracted by the $sLSTM$ network, the extracted visual and audio features are processed through a temporal convolutional network to unify the dimensions and further extract features $X^v$ and $X^a$, which are denoted as

$$X^v=Conv1D(h^v,k^v)\in {ℝ}^{f_v*d}$$

(15)

$$X^a=Conv1D(h^a,k^a)\in {ℝ}^{f_a*d}$$

(16)

Then, we use a Transformer encoder to enable attention to flow independently within each modality for unimodal representation learning in Equations (17) and (18).

$$T^v=Transformer(X^v)$$

(17)

$$T^a=Transformer(X^a)$$

(18)

Finally, low-level feature fusion is performed through the Transformer aggregation network. This module forces each modality to organize and condense its information before sharing it with another modality. The core idea is to introduce $T^f=T_1^f,T_2^f,\dots ,T_B^f$ in the input sequence, where the dimension $B$ token is much smaller than the dimensions of the audio and visual modalities. The specific equation is as follows:

$$[\overline{T}\parallel \overline{T^f}]=Transformer([T^v\parallel T^f])$$

(19)

$$[\overline{T^a}\parallel \overline{T^{f+1}}]=Transformer([T^a\parallel \overline{T^f}])$$

(20)

$$T^{va}=[\overline{T^a}\parallel \overline{T^{f+1}}]$$

(21)

The aggregation module sequentially compresses the information of each modality using visual and audio information. Introducing the $B$ tokens to share only the necessary information improves the performance of multimodal fusion while reducing computational complexity.

3.4. Cross-Attention Fusion Module (CAFM)

We use a CAFM to enhance information complementarity and improve feature representation capability. Through the TAM module, we obtain high-level semantic features $T^{va}$ from audiovisual fusion. Cross-attention is used to achieve mutual enhancement between the textual modality and the audiovisual fusion.

First, a fully connected layer is used to align the dimensions of the textual modality feature vector with the audiovisual fusion modality feature vector, facilitating the attention calculation. The specific equations are as follows:

$$X_t=W_t{h}^t+b_t$$

(22)

$$X_{va}=W_{va}{T}^{va}+b_{va}$$

(23)

where $W_t\in {ℝ}^{d_m\times d_t},b_t\in {ℝ}^{d_m}$, $W_{va}\in {ℝ}^{d_m\times d_{va}},b_{va}\in {ℝ}^{d_m}$.

The correlation between textual modality features and audiovisual fusion features is assessed to capture their interrelation. To address modality differences, a learnable weight matrix $W\in {ℝ}^{K\times K}$ is employed, and the cross-correlation is expressed as follows:

$$Z=X_{va}^{T}W{X}_t$$

(24)

where $Z\in {ℝ}^{d_m\times d_m}$, $W$ represents the mutual weights between the textual and audiovisual fusion features, denotes the feature dimensions of the textual and audiovisual fusion features, and $T$ represents the transpose.

The $Z$ quantifies the relationship between textual and audiovisual fusion features. The higher the $Z$ value, the stronger the correlation between the text fusion features and the corresponding subsequences of the audiovisual fusion features. Therefore, the textual and audiovisual fusion features are computed by applying a column-wise $softmax$ to $Z$ and $Z^T$, where $A_t$ and $A_{va}$ represent the cross-attention weights, which are denoted as

$$A_{t_{i,j}}={\displaystyle \frac{e^{\raisebox{1ex}{$Z_{i,j}$}\!\left/ \!\raisebox{-1ex}{$T$}\right.}}{{\displaystyle \sum _{k=1}^K{e}^{\raisebox{1ex}{$Z_{k,j}$}\!\left/ \!\raisebox{-1ex}{$T$}\right.}}}}$$

(25)

$$A_{v{a}_{i,j}}={\displaystyle \frac{e^{\raisebox{1ex}{${Z^T}_{i,j}$}\!\left/ \!\raisebox{-1ex}{$T$}\right.}}{{\displaystyle \sum _{k=1}^K{e}^{\raisebox{1ex}{${Z^T}_{i,k}$}\!\left/ \!\raisebox{-1ex}{$T$}\right.}}}}$$

(26)

where $i$ and $j$ represent the $i$-th row and $j$-th column of the cross-correlation matrix, respectively, and $T$ denotes the $softmax$ temperature coefficient.

Since the weight matrix $W$ is learned based on the mutual correlations between textual and audiovisual fusion features, the attention weights for each modality are guided by the other modality. This effectively leverages the complementary nature of the textual and audiovisual fusion modalities. Subsequently, the textual and audiovisual fusion features are weighted using cross-attention weights, making the cross-attention weights more comprehensive. The specific equation is as follows:

$$\overline{X_t}=X_t{A}_t$$

(27)

$$\overline{X_{va}}=X_{va}{A}_{va}$$

(28)

where $A_t$ and $A_{va}$ represent the cross-attention weights for the textual and audiovisual fused features, respectively.

Then, the reweighted representation is added to the corresponding features to obtain the attended features $X_{att,t}$, $X_{att,va}$, which are represented as

$$X_{att,t}=tanh(X_t+\overline{X_t})$$

(29)

$$X_{att,va}=tanh(X_{va}+\overline{X_{va}})$$

(30)

Finally, $X_{att,t}$ and $X_{att,va}$ are concatenated to obtain the final representation $\overline{X}$ of the cross-attention fusion in Equation (31).

$$\overline{X}=[X_{att,t};X_{att,va}]$$

(31)

3.5. Feature-Based Attention Fusion Module (FBAFM)

We designed a feature-based attention mechanism fusion module to dynamically adjust the weights of each dimension of $X_t$, $X_{va}$, and $\overline{X}$ obtained from the above modules. This module identifies and emphasizes the most important features by establishing correlations between dimensions, thereby optimizing the fusion of multimodal data. This module enhances the fusion process by focusing on key features and filtering out less relevant information.

First, the obtained $X_t$, $X_{va}$, and $\overline{X}$ are concatenated to form a multimodal representation $H$ in Equation (32). Then, the weighted feature attention matrix $W_{att}$ is computed in Equation (33).

$$H=[X_t\parallel X_{va}\parallel \overline{X}]$$

(32)

$$W_{att}=Sigmoid(W_2(\mathrm{Re}LU(W_1*H)))$$

(33)

where $H\in {ℝ}^{3\times d_m}$, $W_1$, and $W_2$ are learnable parameters, with $W_1,W_2\in {ℝ}^{3\times d_m}$.

Additionally, to assess the importance of each feature channel, we use the sigmoid activation function to ensure that each value in $W_{att}$ is between 0 and 1. This enhances the contribution of key features to the multimodal fusion process. Finally, the concatenated representation is multiplied by $W_{att}$ to obtain the final output $H^{\prime }$ in Equation (34).

$$H^{\prime }=W_{att}*H$$

(34)

3.6. Regression Analysis

In the final regression analysis, we first apply a linear transformation to the multimodal features using a linear layer, followed by a $ReLU$ activation function to introduce nonlinearity. The specific equation is as follows:

$$score=ReLU(W_F{H}^{\prime }+b_F)$$

(35)

where $W_F\in {ℝ}^{1\times d_F},b_F\in {ℝ}^1$.

4. Experiments

4.1. Dataset

The three real-world datasets include MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek, constructed by Wu et al. [40]. Below is a brief introduction to these datasets:

These datasets are built upon the CMU-MOSI dataset [41], which contains 93 videos collected from YouTube, ranging in length from 2 to 5 min, and divided into 2199 short video clips annotated with emotion scores. The emotion scores range from −3 (strongly negative) to 3 (strongly positive). However, the text provided in the MOSI dataset was manually transcribed by experts, making it unsuitable for practical applications. We replaced manually transcribed text with ASR systems to enhance the model’s applicability in real-world scenarios. We used the SpeechBrain ASR [42] model to construct the MOSI-SpeechBrain dataset, as well as two commonly used commercial APIs, IBM and iFlytek, to create the MOSI-IBM and MOSI-iFlytek datasets, respectively.

Table 1. The performance of sentiment word replacement scores in the experimental datasets.

No.	Dataset	Score
Ds 1	MOSI-SpeechBrain	73.5
Ds 2	MOSI-IBM	82.4
Ds 3	MOSI-iFlytek	89.4

presents the sentiment word replacement accuracy scores for the experimental datasets using the three ASR models, with a maximum score of 100. In the following descriptions, dataset codes are used instead of dataset names. The MOSI-SpeechBrain dataset scores 73.5, indicating that only about 74 out of every 100 utterances are free of errors.

4.2. Implementation Details

We employed identical parameters for experiments across all three datasets to guarantee fairness. We trained the network utilizing mean square error (MSE) loss and the Adam optimizer with a consistent learning rate of $5\times {10}^{-5}$. The batch size was established at 64. The hyperparameters $d_x^t$, $d_s^v$, $d_s^a$, and $d_s^{va}$ applied in training were 768, 32, 64, and 96, correspondingly. All experiments were performed on an NVIDIA GeForce RTX 3090 GPU (manufactured by NVIDIA Corporation, Santa Clara, CA, USA).

4.3. Baseline

We will conduct a comparative analysis with the following representative multimodal sentiment analysis models:

MulT [9], which utilizes the Transformer architecture to model the interaction process of multimodal sequences.

MISA [5], which decomposes modalities into modality-invariant and modality-specific representations, performs multimodal fusion on these representations.

Self-MM [10], which is based on a self-supervised learning module for single-modal label generation, explores single-modal supervision.

SWRM [40], which leverages multimodal information to correct text with erroneous sentiment words generated by ASR models.

MMIM [4], which addresses the issues of cross-modal information interaction and multimodal feature fusion effectiveness by using multitask learning to maximize information gain.

TETA [43], which employs a label encoding technique to assist the Transformer network, covering cases of uncertain modality missing through supervised joint representation learning.

4.4. Results and Analysis

In this paper, we evaluate our model’s performance on the Ds 1, Ds 2, and Ds 3 datasets using accuracy (Acc2), F1 score (F1), mean absolute error (MAE), and correlation (Corr). We provide Acc2 and F1 scores for negative/non-negative (Non0-Acc, Non0-F1) and negative/positive (Has0-Acc, Has0-F1) classifications. Continuous predictions were converted to sentiment labels for classification. The results of the baseline models are shown in

Table 2. Performance comparison with baseline models. An upward arrow (↑) signifies that higher numerical values indicate better model performance, while a downward arrow (↓) denotes that lower numerical values represent better performance.

Dataset No.	Model	Evaluation Metrics
		Has0-Acc ↑	Has0-F1 ↑	Non0-Acc ↑	Non0-F1 ↑	MAE ↓	Corr ↑
Ds 1	MulT	71.78	71.70	72.74	72.75	109.00	54.69
	Self-MM	73.67	73.72	74.85	74.98	90.95	67.23
	MMIM	73.81	73.93	75.02	75.11	90.83	67.43
	TATE	74.41	74.48	75.63	75.69	90.48	67.52
	SWRM	74.58	74.62	75.70	75.82	90.56	67.47
	Ours	76.82	76.74	78.20	78.18	87.23	68.35
Ds 2	MulT	75.57	75.54	76.74	76.79	100.32	64.34
	Self-MM	77.32	77.37	78.60	78.72	85.65	73.23
	MMIM	78.28	78.30	79.02	79.08	83.04	73.68
	TATE	78.51	78.63	79.64	79.72	83.22	73.84
	SWRM	78.43	78.47	79.70	79.80	82.91	73.91
	Ours	80.52	80.63	82.05	82.31	79.12	75.78
Ds 3	MulT	77.32	77.05	78.75	78.56	89.84	68.14
	Self-MM	80.26	80.26	81.16	81.20	78.79	75.83
	MMIM	79.24	79.36	79.89	80.08	78.50	75.62
	TATE	81.32	81.38	82.06	82.10	76.98	76.60
	SWRM	80.47	80.47	81.28	81.34	78.39	75.97
	Ours	82.31	82.25	83.73	83.78	74.74	76.89

Bolded values indicate the optimal performance.

.

Based on the experimental results in Table 2, we can conclude that the MSWOHHF model outperforms several common multimodal models across multiple datasets. Compared to the MulT model, the MSWOHHF shows significant improvements regarding all the datasets. This is mainly because MulT overlooks the differences in information density between modalities, whereas the MSWOHHF more effectively addresses this issue, particularly excelling in the fusion of visual and audio features. Compared to the Self-MM and MMIM models, the MSWOHHF also demonstrates superior performance on most datasets. While multi-task learning methods can share information across modalities and create synergy, their performance may be impacted when the correlation between certain modalities is weak. The MSWOHHF overcomes this challenge by better handling the heterogeneity between modalities. Additionally, the comparison with the TATE model shows that, when the probability of replacing the emotional words in the text is low, the performance gap between the two models narrows. However, while TATE has advantages in handling missing modalities, it does not pay attention to the heterogeneity among different modes, resulting in insufficient fusion and redundant information in the fusion process. Lastly, the comparison between the MSWOHHF and SWRM models indicates that the MSWOHHF promotes more balanced interactions between modalities by more finely addressing the differences in information density, especially excelling in the fusion of both low-level and high-level features.

4.5. Ablation Study

4.5.1. Modal Ablation

Since the three datasets differ only in their textual content, this section performs modality ablation experiments using only the Ds 1 dataset.

Table 3. Ablation experiments from the modality perspective on the Ds 1 dataset.

Task	Has0-Acc ↑	Non0-Acc ↑	MAE ↓
Textual	73.32	75.09	94.22
Audio	44.75	45.10	144.32
Visual	55.24	56.30	140.30
Textual, Visual	73.92	75.36	91.43
Textual, Audio	73.56	74.70	92.46
Visual, Audio	58.23	59.86	136.75
Textual, Visual, Audio	74.20	75.12	89.10
MSWOHHF	76.82	78.20	87.23

presents the experimental results for the unimodal, bimodal, and trimodal combinations. The results show that the textual modality performs best among the unimodal configurations, trailing the MSWOHHF model by only 3.50% on the Has0-Acc metric. This highlights the decisive role of the textual modality. The trimodal fusion outperforms both the bimodal and unimodal configurations. The performance is slightly lower when only the bimodal information is fused, and it is the worst when the model only includes a unimodal branch. The results indicate that both the consistency and divergence between the modalities are captured through multimodal information fusion, and the accuracy of the sentiment prediction in the model is enhanced by providing the global feature information.

4.5.2. Model Ablation

This section presents an ablation study to evaluate the contribution of each model component, as shown in

Table 4. Performance comparison from the model perspective.

Dataset No.	Model	Has0-Acc ↑	Non0-Acc ↑	MAE ↓
Ds 1	MSWOHHF	76.82	78.20	87.23
	w/o MSWOM	75.88	77.49	89.52
	w/o TAM	74.46	75.87	90.23
	w/o CAFM	75.64	77.56	88.80
	w/o FBAFM	75.37	76.62	89.69
Ds 2	MSWOHHF	80.52	82.05	79.12
	w/o MSWOM	79.72	81.16	80.76
	w/o TAM	78.23	79.88	82.02
	w/o CAFM	80.12	81.64	80.48
	w/o FBAFM	78.82	80.86	81.72
Ds 3	MSWOHHF	82.31	83.73	74.74
	w/o MSWOM	81.02	81.94	76.12
	w/o TAM	80.46	81.22	77.13
	w/o CAFM	81.56	82.74	75.98
	w/o FBAFM	80.32	81.40	77.32

. The MSWOHHF represents the complete model proposed in this paper. Moreover, w/o MSWOM refers to the model without the MSWOM, meaning that no corrections occur for the sentiment words misrecognized by the ASR system. The experimental results show a significant decline in the model performance, although it still outperforms the baseline models, thus proving the effectiveness and superiority of the MSWOHHF fusion network. Moreover, w/o TAM refers to the model without the TAM. The experimental results indicate that compensating for the differences in information density between modalities is necessary as fair information exchange under such conditions yields better results. Further, w/o CAFM and w/o FBAFM refer to the models without the CAFM and without the FBAFM, respectively. The experimental results show that both components are essential as they enhance the interaction between the modalities and thus improve the model’s performance.

5. Conclusions

In this paper, we propose the MSWOHHF framework for multimodal sentiment analysis in real-world scenarios. By utilizing a Multimodal Sentiment Word Optimization Module, the MSWOHHF reduces the sentiment recognition errors caused by the limitations of the ASR model performance. During the multimodal fusion process, we introduce a heterogeneous hierarchical fusion network to address the variations in the information density between the different modalities. Specifically, the Transformer aggregation module enhances the semantic representation of the audio and visual modes, while the Cross-Attention Fusion Module facilitates interactions among textual, audio, and visual data. Furthermore, the Feature-Based Attention Fusion Module dynamically adjusts each modality’s weights, aiming to improve the multimodal fusion’s effectiveness. Finally, a nonlinear layer predicts the emotion labels. The experimental results demonstrate that the MSWOHHF model outperforms the other benchmark models on the Ds 1, Ds 2, and Ds 3 datasets, showcasing its robustness in handling various types of multimodal data.

Our current model is trained primarily on datasets composed of English and Western-centric content. This may limit its generalization to multilingual and cross-cultural contexts as sentiment expressions vary. In future work, we plan to enhance the model’s generalizability by incorporating multilingual and cross-cultural training data to capture these variations better and improve its global applicability. Additionally, we acknowledge that the current model may not fully capture long-term temporal dependency features, and enhancing the modeling of these long-term dependencies will be a focus for future improvements.