An End-to-End Mutually Interactive Emotion–Cause Pair Extractor via Soft Sharing

Abstract

Emotion–cause pair extraction (ECPE), i.e., extracting pairs of emotions and corresponding causes from text, has recently attracted a lot of research interest. However, current ECPE models face two problems: (1) The common two-stage pipeline causes the error to be accumulated. (2) Ignoring the mutual connection between the extraction and pairing of emotion and cause limits the performance. In this paper, we propose a novel end-to-end mutually interactive emotion–cause pair extractor (Emiece) that is able to effectively extract emotion–cause pairs from all potential clause pairs. Specifically, we design two soft-shared clause-level encoders in an end-to-end deep model to measure the weighted probability of being a potential emotion–cause pair. Experiments on standard ECPE datasets show that Emiece achieves drastic improvements over the original two-step ECPE model and other end-to-end models in the extraction of major emotional cause pairs. The effectiveness of soft sharing and the applicability of the Emiece framework are further demonstrated by ablation experiments.

1. Introduction

Recently, emotion–cause extraction (ECE) has gained great popularity in text analysis [1,2]. ECE aims at extracting potential causes that lead to emotional expressions in the text. Instead of using word-level labeled sequence, ECE concentrates on the clause-level sequence, thus fully exploiting the linked relationship between different sentences [3]. This kind of clause-level observation improves the reliability of the ECE analysis greatly. In this respect, Ref. [4] first released a corresponding corpus, which was widely used in the following studies [5,6,7,8,9,10,11]. There are, however, two limitations [12] associated with the ECE task. On the one hand, ECE relies on the annotated text sentiment as the input at inference time, which limits its application. On the other hand, ECE ignores the mutual relationship between emotion and cause sentences in the text.

To solve the existing problems in ECE, Ref. [12] proposed a new task, emotion–cause pair extraction (ECPE), which aims to extract all possible emotion–cause pairs in the text without a given annotated sentiment word. Figure 1 illustrates the goal of ECPE and the key differences in comparison with ECE in the solution process. For the input document, the ECE task first identifies and annotates the emotion of the clauses, and then extracts the cause based on the annotated emotion. In the example shown in the figure, the emotion “anger” is first marked according to clause 7 “Jobs threw a tantrum” (Figure 1), and then the corresponding cause clause, “Scott assigned No.1 to Wozniak and No.2 to Jobs” (Figure 1), is extracted from the input document according to the emotion annotation. The ECPE task, in contrast to the ECE task, does not require annotations of sentiment from emotion clauses but notes that emotion and cause are mutually indicative. Therefore, all possible pairs are matched and filtered, and the two valid emotional cause pairs in this input document are directly derived: (clause 7–clause 1) and (clause 8–clause 1), i.e., the emotion clause “Jobs threw a tantrum” (Figure 1) and its corresponding cause clause “Scott assigned No.1 to Wozniak and No.2 to Jobs” (Figure 1), and the emotion clause “even cried” (Figure 1) and its corresponding cause clause “Scott assigned No.1 to Wozniak and No.2 to Jobs” (Figure 1), without relying on the emotion annotations “anger” and “sadness” in the clause. The matching of two different emotion clauses with one identical cause clause, or the pairing of one emotion clause with two different cause clauses, as in the example in [12], reflects the increased attention of the ECPE task to the connection between the emotion and cause.

Currently, ECPE methods can be divided into two categories: two-step ECPE and end-to-end ECPE. The two-step ECPE [12] is composed by the emotion extraction and cause extraction step and the pairing and filtering step. As it is not an end-to-end model, the accumulated error of the first step will affect the result in the second step. End-to-end models [13,14] consider the mutual interaction between emotion and cause. However, the process of mutual interaction is simply implemented by transferring information from one decoder to another uniaxially, thus harming the mutual transfer of information. Moreover, some of the end-to-end models of [14,15] contain a huge number of parameters, resulting in an excessive training time and unremarkable model performance. In this paper, our target is to effectively extract emotion–cause pairs from all potential clause pairs with fewer parameters.

To address the aforementioned challenges, we propose an end-to-end model, Emiece, that predicts the emotion–cause pair from the raw document. We observe that ECPE can be viewed as three mutually related tasks: a primary task of predicting pairings and two auxiliary tasks of predicting emotion and cause clauses respectively. To efficiently learn the three tasks, we consider adopting multi-task learning to establish the connection of them. Multi-task learning [16,17,18,19,20,21,22,23,24,25,26,27,28] is an effective way to achieve better generalization performance with a group of related tasks, while sharing some common parameters. Since the two auxiliary tasks are highly similar, we attempt to utilize multi-task learning to tackle the problem of ECPE models’ unsatisfactory performance. Inspired by [29], we choose to leverage a soft-sharing approach to exploit the intrinsic connection between emotion and cause.

After the word-level encoder, each clause is passed through the emotion and cause clause probabilities that indicate the importance of the clause in terms of emotion and cause, and then we obtain the weighted representation of emotion clause and cause clause, respectively. The aim of the weighted representation is to make the true representation and the false one as separate as possible in the feature space so that the extracted representation can be more easily classified for the pair predictor in the higher layer. We conduct sufficient experiments on an ECPE task suitable corpus, which is adapted from the English-language corpus benchmark of the NTCIR-13 Workshop [30].

The main contributions of our work can be summarized as follows:

• Mutual transfer of information in emotion and cause extraction. Soft-sharing is applied between emotion and cause encoders. We add the soft-sharing loss to the total loss function in a multi-task learning style to involve mutual interaction between the two auxiliary tasks. Therefore, the two encoders can learn from each other rather than unidirectional learning in previous methods.
• Efficient pair extractor with weighted representation. We utilize the weighted representation of emotion and cause to filter the clauses which tend to be meaningless. Therefore, only the useful emotion-weighted and cause-weighted clause representations can be reserved to improve the efficiency of emotion–cause paring.
• Novel end-to-end ECPE model. We propose a novel end-to-end method that uses two LSTMs to automatically transfer information between the emotion encoder and cause encoder via soft sharing. Since the end-to-end model considers single emotion and cause extraction along with emotion–cause pairing at the same time, it greatly avoids the cumulative errors in separated steps and significantly improves the performance.

2. Related Work

The emotion–cause extraction (ECE) task was first proposed in [1]. As a word-level task, the extraction is fulfilled with traditional machine learning and rule-based approaches [5,6,7,8,9,10,11,31,32,33,34]. For example, in [7], the authors proposed a fine-grained rule-based method for the task and conducted experiments on the Chinese microblog posts corpus labeled by human annotators. Despite the overall performance of the word-level task not being promising enough, it provides a new way to look at the emotion classification task.

Another kind of emotion–cause extraction task is based on a clause that solves the problem of word-level labeling in previous work [35,36,37,38,39]. In [35], the authors employed a multi-kernel learning method for the clause-level task on a Chinese emotion cause corpus. Moreover, with the development of deep learning, multiple recurrent neural networks (RNNs) related models have been proposed to solve clause-level tasks due to their excellent performance in analyzing the relationship between different sequences [40,41]. Long short-term memory (LSTM), an advanced version of RNNs, achieves better performance in related tasks thanks to its forgetting mechanism [38]. Although clause-level methods relax word-level annotations into clause-level annotations and achieve higher performance due to the development of neural networks, it is still restricted by manual annotations. In addition, it neglects the mutual relationship and interaction between emotion and cause [42].

To overcome the mentioned drawbacks of ECE, Ref.  [12] proposed emotion–cause pair extraction (ECPE) and they conducted a two-step hierarchical structure network for the task. This model separates the emotion and cause extraction and the pairing into two steps; therefore, the mistakes made in the first step will affect the results of the second step.

To solve these limitations in [12], several end-to-end ECPE models have been proposed [13,14,15,43]. Ref. [14] proposed to model interactions in emotion–cause pairs by means of a two-dimensional transformer [44], which in turn represents emotion–cause pairs in a 2D form. Then a joint framework [45] is used to integrate the two-dimensional representation, interaction and prediction. The work of [15] introduces multi-label learning (MLL) in the ECPE task. The emotion clause and cause clause are designated as the center of the multi-label learning window [46], respectively, and the window slides as the center position is moved. The two joint parts are integrated to obtain the ultimate result. These two models achieved state-of-the-art performance in the ECPE task. Nonetheless, the enormous amount of parameters makes the training overhead of both models extremely large. After that, the method of Ref.  [13] uses Bi-LSTM to perform word-level embedding for the input clauses and encodes representations for the emotion and reason clauses. Finally, a layer of fully connected network is used to predict the matching pairs. It achieves comparable performance with fewer parameters and a simpler architecture. However, its unsatisfactory performance suggests that it does not fully exploit the intrinsic connection between sentiment and cause.

3. Materials and Methods

3.1. Task Formalization

Formally, the documents consist of texts that are segmented as an ordered set of clauses $\mathcal{D}=\{c_1,c_2,\dots ,c_d\}$. The ECPE task aims to figure out a set of emotion–cause pairs

$$\mathcal{P}=\{\dots ,(c_i,c_j),\dots \}(c_i,c_j\in \mathcal{D}),$$

(1)

where $c_i$ is an emotion clause and $c_j$ is the corresponding cause clause. All we have to do is to construct an end-to-end sentiment–cause matching model, which predicts the set of matching pairs $\widehat{\mathcal{P}}$, where the correctly predicted part constitutes the set ${\widehat{\mathcal{P}}}_C$, making the ${\widehat{\mathcal{P}}}_C$ as close as possible to the target set $\mathcal{P}$.

3.2. Architecture

The whole model contains three layers as illustrated in Figure 2: word-level encoder layer, clause-level encoder layer, and pairing layer. We take the vector representation ${\mathit{v}}_{ij}$ of the j-th word in the i-th clause as input. For each clause, the word vector sequence ${\mathit{v}}_{i,1},{\mathit{v}}_{i,2},\dots ,{\mathit{v}}_{i,m}$ is passed through a word-level encoder, implemented via a Bi-LSTM with attention [47]. The word-level encoder outputs a clause representation $s_i$ for each clause.

The higher level contains two clause-level encoders implemented for emotion clause detection and cause clause detection, respectively. Any of the current mainstream encoders, such as the stacked Bi-LSTM [48] or BERT [49], can be used. The two encoders take the clause representation sequence ${\mathit{s}}_1,{\mathit{s}}_2,\dots ,{\mathit{s}}_d$ as input and generate the emotion and cause representation of the clauses ${\mathit{r}}_i^e,{\mathit{r}}_i^c$. In order to mutually transfer the information obtained by the encoders, we use a soft-sharing strategy between the two encoders. The representations are then fed into detectors (logistic regression layers) to obtain the probability distribution ${\mathit{a}}_i^e$, ${\mathit{a}}_i^c$ of the clause being an emotion clause and a cause clause, respectively, formed as

$$\begin{array}{cc}\hfill {\mathit{a}}_i^e& =\mathrm{softmax}({\mathit{W}}^e{\mathit{r}}_i^e+{\mathit{b}}^e),\hfill \\ \hfill {\mathit{a}}_i^c& =\mathrm{softmax}({\mathit{W}}^c{\mathit{r}}_i^c+{\mathit{b}}^c),\hfill \end{array}$$

(2)

where ${\mathit{W}}^e$, ${\mathit{W}}^c$, ${\mathit{b}}^e$, ${\mathit{b}}^c$ are the parameters for emotion and cause detection layers. It should be noted that ${\mathit{a}}_i^e$ is a $1\times 2$ vector in which one element $p_i^e$ represents the probability that it is indeed an emotion clause and the other element represents the probability that it is not. ${\mathit{a}}_i^c$ and $p_i^c$ are related as above. It is worth noting that these two outputs ${\mathit{a}}_i^e$ and ${\mathit{a}}_i^c$ in our approach also need to participate in pair extraction later, instead of having only one function of obtaining supervisory signals, as in the two auxiliary task outputs in the work of [13]. Thus, in contrast to the cascade of the hierarchical framework of [13], our model achieves parallelism.

These probabilities can be regarded as the attention of emotion and cause encoders. Thus, we multiply the clause representation by the probabilities to obtain the emotion-weighted and cause-weighted clause representation ${\tilde{\mathit{r}}}_i^e$, ${\tilde{\mathit{r}}}_i^c$ as

$$\begin{array}{cc}\hfill {\tilde{\mathit{r}}}_i^e& =p_i^e{\mathit{r}}_i^e,\hfill \\ \hfill {\tilde{\mathit{r}}}_i^c& =p_i^c{\mathit{r}}_i^c,\hfill \end{array}$$

(3)

Once we collect all the weighted representation of clauses into two sets $\mathcal{E}={\tilde{\mathit{r}}}_1^e,{\tilde{\mathit{r}}}_2^e,\dots ,{\tilde{\mathit{r}}}_d^e$ and $\mathcal{C}={\tilde{\mathit{r}}}_1^c,{\tilde{\mathit{r}}}_2^c,\dots ,{\tilde{\mathit{r}}}_d^c$, the Cartesian product is applied on the two sets to generate all the potential emotion–cause pairs $({\tilde{\mathit{r}}}_i^e,{\tilde{\mathit{r}}}_i^c)$. In Figure 3, we use ${\mathit{r}}_{ij}^p={\tilde{\mathit{r}}}_i^e\oplus {\tilde{\mathit{r}}}_i^c\oplus {\mathit{d}}_{ij}$ as the representation of a pair, where ⊕ denotes the concatenation operator and ${\mathit{d}}_{ij}$ is the positional embedding vector that indicates the relative position relation between clause i and j [50]. The pairs are fed into the pairing layer one at a time to obtain the predicted label. The pairing layer is a fully connected layer as

$$\begin{array}{cc}\hfill {\mathit{h}}_{ij}& =\mathrm{ReLU}({\mathit{W}}^h{\mathit{r}}_{ij}^p+{\mathit{b}}^h),\hfill \\ \hfill {\widehat{\mathit{y}}}_{ij}^p& =\mathrm{softmax}({\mathit{W}}^y{\mathit{h}}_{ij}+{\mathit{b}}^y),\hfill \end{array}$$

(4)

where ${\widehat{\mathit{y}}}_{ij}^p$ gives the Bernoulli distribution probabilities of $(c_i,c_j)$ to be an emotion–cause pair. In total, there are three tasks: one primary task for predicting pairs and two auxiliary tasks for predicting emotion and cause clauses. The outputs are ${\widehat{\mathit{y}}}_{ij}^p$,${\mathit{a}}_i^e$,${\mathit{a}}_i^c$, respectively.

3.3. Learning with Mutual Transfer of Information

In order to effectively train Emiece, we set the loss function to be

$$L_{\mathrm{total}}={\lambda }_p{L}_p+{\lambda }_e{L}_e+{\lambda }_c{L}_c+{\lambda }_{sf}{L}_{sf},$$

(5)

where $L_p$, $L_e$ and $L_c$ are the cross-entropy losses of pair extraction, emotion clause detection, and cause clause detection, respectively. $L_{sf}$ is the soft-sharing loss for the mutual transfer of information.

Following [29], we define

$$L_{sf}=\sum _{d\in \mathcal{D}}{\parallel {\varphi }_d^e-{\varphi }_d^c\parallel }^2,$$

(6)

where $\mathcal{D}$ is the set of sharing parameter indices, ${\varphi }^e$ and ${\varphi }^c$ are the emotion and cause encoder parameters, respectively. The works in recent years found [51] that features extracted in the shallow layers of a deep neural network contain more general features of different tasks. The higher the network level, the more task specific the extracted features will be. Following the work of [29,52], we employ a similar soft-sharing strategy on the first-layer encoder in the emotion and cause encoders and keep the second layer of the two encoders unshared, which is further away from the input. A discussion of soft-sharing modules and ablation studies is given in Section 5.2.

To avoid the imbalance of positive pairs and negative pairs in the pair extraction, the loss $L_p$ is revised as

$$L_p=L_p^{+}+{\lambda }_{-}{L}_p^{-},$$

(7)

where $L_p^{+}$,$L_p^{-}$ denotes the term of positive and negative ground truths in the cross-entropy loss function. ${\lambda }_{-}$ is relatively small since the number of negative pairs is much more than positive ones.

3.4. Evaluation Metrics

Following the previous work [12], we also used the same three evaluation metrics: precision, recall, and F1 score. The F1 score takes both precision and recall into account, and thus it is the most crucial of these evaluation metrics. They are defined as follows:

$$\begin{array}{cc}\hfill \mathrm{Precision}& =\frac{|{\widehat{\mathcal{P}}}_c|}{|\widehat{\mathcal{P}}|},\hfill \\ \hfill \mathrm{Recall}& =\frac{|{\widehat{\mathcal{P}}}_c|}{\left|\mathcal{P}\right|},\hfill \\ \hfill \mathrm{F}1\mathrm{score}& =\frac{2\times Precision\times Recall}{Precision+Recall},\hfill \end{array}$$

(8)

according to Section 3.1, $|{\widehat{\mathcal{P}}}_c|$ indicates the number of emotion–cause pairs predicted by the model, $|\widehat{\mathcal{P}}|$ indicates the number of correct pairs among these predicted matches, and $\left|\mathcal{P}\right|$ indicates the number of all emotion–cause pairs in the actual dataset.

4. Results

4.1. Dataset

The dataset was constructed by [13] from an existing emotion–cause extraction (ECE) corpus. The corpus was introduced in the NTCIR-13 Workshop [30] for the ECE challenge. There are 2843 documents taken from several English novels, and each document is annotated with the following:

• Emotion–cause pairs (the set of emotion clauses and their corresponding cause clauses);
• Emotion category of each clause;
• Keywords in the emotion clauses.

Detailed statistics about the dataset are presented in

Table 1. Overall information on the English-language corpus dataset. Information on the four main dimensions related to the ECPE task was counted.

# Document	# Clause ($\left|\mathcal{D}\right|$)	# Emotion-Cause Pair ($\left|\mathcal{P}\right|$)	# Annotated Emotion Type
2843	21,802	3272	6

and

Table 2. The distribution of emotion clauses corresponding to the six annotated emotions in the dataset.

Annotated Emotion	# Corresponding Emotion Clause
happiness	741
surprise	388
sadness	638
fear	622
anger	269
disgust	214

below. In our experiments, we do not leverage the emotion category or keywords and only exploit the emotion–cause pairs in the training process. None of the annotations are used when testing the model. The whole dataset is split by 80%–10%–10% for training, validating, and testing.

4.2. Baselines and Settings

We include four baseline methods in our comparison with Emiece: the original two-step model ECPE [12], a relatively lightweight end-to-end model $\mathtt{E}\mathtt{2}\mathtt{E}-{\mathtt{PExt}}_{\mathtt{E}}$ [13], and two state-of-the-art ECPE models ECPE-2D(BERT) [14] and ECPE-MLL(ISML-6) [15].

• ECPE [12]: As a second step, a Cartesian product is applied to the emotion clauses and causal relationships extracted from the multi-task learning network in the first step in order to compose them into pairs, and a filtering model is trained so that the pair containing the causal relationship is the final output. Bi-LSTM and attention [47,48] is the word-level encoder used in the first extraction step, and Bi-LSTM [48] is used in the emotion and cause extractors as well. Logic regression is used to filter the pairs in the second step.
• ECPE-2D(BERT) [14]: The interactions in the emotion–cause pairs were modeled by a 2D transformer, which in turn represented the pairs in a two-dimensional form, i.e., a square matrix. Two-dimensional representations are integrated with interactions and predictions using a joint framework. The encoding part uses a word-level Bi-LSTM and an attention mechanism [47], while the clause-level emotion extractor and cause extractor leverage BERT [49] to enhance the overall effectiveness of the model.
• ECPE-MLL(ISML-6) [15]: Multi-label learning (MLL) was introduced in the ECPE task. To obtain a representation of the clause, the emotion clause and cause clause are specified as the center of the multi-label learning window. An iterative synchronous multi-task learning (ISML) model with six iterations is used for clause encoding, while the same Bi-LSTM [47] is used for word-level embedding.
• $\mathtt{E}\mathtt{2}\mathtt{E}-{\mathtt{PExt}}_{\mathtt{E}}$ [13]: Using Bi-LSTM plus attention [47], the clause-level representation is obtained based on the word-level one. The clause level representation uses another Bi-LSTM network to further extract contextual information and is used to determine whether the clause is an emotion one or a cause one. Finally, the predicted pair is obtained by a fully connected neural network.

We denote Emiece-LSTM that using as the stacked Bi-LSTM [48] clause-level encoder. Emiece-LSTM is trained for 30 epochs using the Adam optimizer [53]. We set the learning rate $\alpha$ = 0.005, and batch size N = 64. The model parameters $\varphi$ are initialized randomly following uniform distribution $\varphi \sim U(-0.1,0.1)$. We leverage GloVe word embedding [54] of 200 dimensions. The dropout rate is set to 0.8 for word embeddings and ${\ell }_2$ decay is set to ${10}^{-5}$ on softmax parameters. The loss weights are set as ${\lambda }_e$:${\lambda }_c$:${\lambda }_p$:${\lambda }_{sf}$ = 1:1:2.5:0.75 and the negative pair weight ${\lambda }_{-}$ = $0.4$. Here, the value of ${\lambda }_{-}$ is taken with reference to the work of [13], and we have made many attempts to take the value of ${\lambda }_{sf}$, as shown in Figure 4. As a result of setting ${\lambda }_{sf}$ to $0.75$, the model performs best in combination at validation, i.e., it performs best on the more important F1-score metric and relatively well on the other two metrics. We also denote Emiece-BERT with the clause-level encoder setting to BERT [49]. Constrained training server performance, we set the batch size of Emiece-BERT to 16 and vary the learning rate by 2 × 10${}^{-5}$, keeping other hyperparameters the same.

In order to achieve higher model performance through better positional embeddings, randomly initialized embeddings are trained after setting the clipping distance [50] to 10.

4.3. Overall Performance

Table 3. Best results of our model and previous experimental methods with existing metrics after hyper-parameter tuning. Bold fonts indicate the best results for the method, and underlining stands for the second best results. The top half is the performance of pipelines, and the bottom half is the one of our model.

	Emotion Extraction			Cause Extraction			Pair Extraction
	Precision	Recall	F1 Score	Precision	Recall	F1 Score	Precision	Recall	F1 Score
ECPE [12]	$0.6741$	$\underline{0.7160}$	$0.6940$	$0.6039$	$0.4734$	$0.5301$	$0.4694$	$0.4102$	$0.4367$
ECPE-2D(BERT) [14]	$0.7435$	$0.6968$	$0.7189$	$0.6491$	$0.5353$	$0.5855$	$\underline{0.6049}$	$0.4384$	$0.5073$
ECPE-MLL(ISML-6) [15]	$0.7546$	$0.6996$	$\underline{0.7255}$	$0.6350$	$\mathit{0}.\mathit{5919}$	$\underline{0.6110}$	$0.5926$	$0.4530$	$0.5121$
$\mathtt{E}\mathtt{2}\mathtt{E}-{\mathtt{PExt}}_{\mathtt{E}}$ [13]	$0.7163$	$0.6749$	$0.6943$	$0.6636$	$0.4375$	$0.5226$	$0.5134$	$\underline{0.4929}$	$0.5017$
Emiece-LSTM (Ours)	$\underline{0.7702}$	$0.6550$	$0.7071$	$\underline{0.7010}$	$0.4413$	$0.5398$	$0.5693$	$0.4903$	$\underline{0.5257}$
Emiece-BERT (Ours)	$\mathit{0}.\mathit{8263}$	$\mathit{0}.\mathit{7441}$	$\mathit{0}.\mathit{7830}$	$\mathit{0}.\mathit{7135}$	$\underline{0.5522}$	$\mathit{0}.\mathit{6225}$	$\mathit{0}.\mathit{6833}$	$\mathit{0}.\mathit{5325}$	$\mathit{0}.\mathit{5985}$

presents the experimental performance in the ECPE task. Compared to a similar end-to-end approach E2E-PExt${}_{\mathrm{E}}$ [13], the F1 scores of our model Emiece-LSTM improved by 1.28%, 1.72% and 2.4% in the emotion extraction, cause extraction and emotion–cause pair extraction tasks, respectively. This is a strong indication that the way of using the intrinsic connection between emotion and cause to extract match pairs is correct, and the related ablation experiments will be placed in Section 5. Compared to the traditional two-step approach ECPE [12], our Emiece model improves the performance of the emotion–cause pair extraction task by an impressive 8.9%. In addition, our method has better results than the current methods ECPE-2D(BERT) [14] and ECPE-MLL(ISML-6) [15] on the most dominant prediction matching pair task, even though the number of parameters is much smaller than theirs.

Through the case study in Figure 5, we can also visualize that the Emiece method performs well in relatively complex short texts.

5. Ablation Study

We studied the effect of different modules on the experimental results through ablation experiments.

5.1. Clause-Level Encoder

The excellent performance of ECPE-2D(BERT) [14] and ECPE-MLL(ISML-6) [15] with a larger number of parameters in the ECPE task cannot be ignored. Inspired by their work, we replaced the clause-level encoder in Emiece from the stacked Bi-LSTM [48] to BERT [49] as the new Emiece-BERT model. Nonetheless, due to the server storage space limitation, training Emiece-BERT can only set the batch size to 16. To control the variables, we keep the hyperparameters of Emiece-LSTM in line with Emiece-BERT and retrain it.

As can be seen from

Table 4. Comparison of the model effects of Emiece composed of two different clause-level encoders. The results are under the same hyper-parameter setting.

Clause-Level Encoder	Emotion Extraction			Cause Extraction			Pair Extraction
	Precision	Recall	F1 Score	Precision	Recall	F1 Score	Precision	Recall	F1 Score
stacked Bi-LSTM [47]	$0.7425$	$0.6665$	$0.7014$	$0.6570$	$0.4762$	$0.5473$	$0.5398$	$0.4973$	$0.5153$
BERT [49]	$\mathit{0}.\mathit{8263}$	$\mathit{0}.\mathit{7441}$	$\mathit{0}.\mathit{7830}$	$\mathit{0}.\mathit{7135}$	$\mathit{0}.\mathit{5522}$	$\mathit{0}.\mathit{6225}$	$\mathit{0}.\mathit{6833}$	$\mathit{0}.\mathit{5325}$	$\mathit{0}.\mathit{5985}$

, under the end-to-end soft-sharing emotion–cause pair extraction framework, simply replacing a more complex encoder, i.e., BERT [49], makes the model surprisingly effective in improving each metric on each of the three tasks of ECPE. However, this undoubtedly introduces a much larger number of parameters for training. In total, 69 hours are consumed by the BERT model in one training, while 16 hours are consumed by Emiece-LSTM training; such a replacement causes a considerable overhead and makes the model redundant and unwieldy.

5.2. Mutual Transfer of Information

We selected the most comprehensive metric F1 score and the primary task emotion–cause pair extraction to analyze the effectiveness of the mutual transfer of information by varying the soft-sharing setting: no soft-sharing, sharing only the first layer of the emotion and cause encoders, and sharing all layers of both encoders. Figure 6a shows that sharing only the first layer of the encoder achieves better performance compared to the other two cases, indicating that the intrinsic connection between emotion and cause is closer to the lexico-syntactic level [29]. Due to the long training time of model Emiece-BERT, only model Emiece-LSTM is selected here.

In Figure 6b, the results illustrate that the number of parameters shared in the encoder is not linearly correlated with the effect of the prediction results. This is consistent with the results of previous experiments in which the weight of soft sharing was altered by changing the value of ${\lambda }_{sf}$, but the higher the weight is not better. Ref. [52] found that in the seq2seq machine translation model, the low-level layer of the RNNs unit (i.e., the first layer in the encoder) represents the word structure, while the high-level layer focuses on the semantic meaning. Since the semantic information of the emotion clause and the cause clause is quite different, sharing the high-level layer alone will blur the features of the text, not to mention that sharing all layers will confuse the word structure information with the semantic information, resulting in a reduced effect.

6. Conclusions

In this paper, we propose an end-to-end model that mutually transfers information via soft sharing between emotion and cause extraction tasks. By using weighted representations of sentiment and cause filtering nonsensical clauses, we improved the efficiency of emotion–cause pairing. The end-to-end model takes into account both emotion and cause extraction and emotion–cause pairing, thereby greatly reducing the cumulative error. Based on experiments conducted on the standard ECPE dataset, Emiece achieves significant improvements in emotion–cause pairs extraction over the original two-step ECPE model and other end-to-end models.

In the future, (1) we will examine the reasons for the model’s mistakes by performing an interpretability analysis on the possible wrong predictions in the current model. (2) We also attempt to reduce the network depth in order to perform the emotion–cause pair extraction as a solution to the problem that it is difficult to train a clause-level encoder with numerous parameters. (3) It is relatively complex to construct pairs with the Cartesian product, so we will use a more efficient module for pairing prediction.