Predicting Change in Emotion through Ordinal Patterns and Simple Symbolic Expressions

Abstract

Human interlocutors may use emotions as an important signaling device for coordinating an interaction. In this context, predicting a significant change in a speaker’s emotion may be important for regulating the interaction. Given the nonlinear and noisy nature of human conversations and relatively short time series they produce, such a predictive model is an open challenge, both for modeling human behavior and in engineering artificial intelligence systems for predicting change. In this paper, we present simple and theoretically grounded models for predicting the direction of change in emotion during conversation. We tested our approach on textual data from several massive conversations corpora and two different cultures: Chinese (Mandarin) and American (English). The results converge in suggesting that change in emotion may be successfully predicted, even with regard to very short, nonlinear, and noisy interactions.

1. Introduction: Emotion and Emotion Dynamics

Human social relations are formed through interactions that are mostly grounded in natural language [1,2,3]. Researchers focusing on language use [1,4] have repeatedly pointed to the way interlocutors in a conversation invest effort in coordinating their use of language, a process discussed under various names, from “resonance” [5,6] to “synchronization” [7].

As human interactions require an ongoing process of coordination, it is not surprising that emotions play an important role in coordinating an interaction. From a functional perspective, emotions have an important role, allowing interlocutors to signal to each other, in order to ease communication in a quick and unconscious form [8], thus supporting and aiding language use in complex social situations. While language use may be supported by emotions, this process is bidirectional, as the processing of emotion is achieved through the mediation of language. Indeed, language has been shown to mediate emotion understanding [9] to the extent that it is argued that “emotion is represented in the brain as a set of semantic features in a distributed sensory, motor, language and affective network” [10] (p. 813).

The bidirectional relation between language use and emotion is explained by the fact that, while emotion signaling and recognition have an innate and universal dimension (e.g., through the processing of facial expressions), the complexity of processing emotions may result from the activity of a complementary process where the emotion is decoded and interpreted according to cultural norms [11], mainly through language. Therefore, emotions support language use in interaction, but they are also expressed through language, a situation that creates a high level of processing demands and complexity.

1.1. Language Use, Emotions, and Feedback Loops

A second source of complexity in processing emotions is the social relational aspect of emotions. As argued by Parkinson, “Emotions serve to calibrate … individuals’ respective orientations to what is happening” [12] (p. 78, our emphasis). This “relation-alignment perspective” (p. 78) suggests that “talking emotionally involves aligning people’s relations towards what is happening and not simply trying to pin down the semantic features of a prior private experience” (p. 80). In other words, emotions do not trivially mirror internal mental states but are functionally and dynamically used to signal to others and move them to action. Emotions are, therefore, (1) dynamically processed through (2) the exchange of signals between interlocutors.

If linguistic emotional signals are dynamically fed back and forth between interlocutors, then they may be modeled in terms of feedback loops. The theoretical justification for focusing on feedback loops in attempting to understand emotion dynamics follows the idea that “emotions serve to align people’s orientations to one another and to objects and events in their shared environment” and “mutual dynamic adjustments” are made through the interaction [12] (p. 85, our emphasis).

Two major, and generally acknowledged, processes that may underlie changes in emotion and “dynamic adjustments” [12] (p. 85) are, therefore, positive and negative feedback loops. As explained by Brown, “Feedback loops are typically used to accomplish regulation and control. A feedback loop is like an input, but its origin is from within the system itself, not from outside the system” [13] (p. 15). In the case of valence, the most basic dimension of emotion, positive feedback amplifies the system’s output, resulting in the growth or decline of the valence. In contrast, negative feedback moderates the valence and stabilizes it around an equilibrium point. For example, in a triadic sequence of turns produced by two speakers—A₁-B-A₂—A may significantly change the sentiment from A₁ to A₂, as a response to B’s response to the utterance A originally produced (we use the terms ‘valence’ and ‘sentiment’ interchangeably). If the sentiment (i.e., valence) presented by B as a response to A’s turn is much higher than the one originally signaled by A, A may interpret this signal in a way that invites the relative increase of A’s sentiment, not only in comparison to A’s original sentiment but also in comparison to B’s sentiment. Such a sequence, where sentiment is measured on a 0 to 1 scale, may be represented as 0.2, 0.4, or 0.6; it may also be described as a monotonic increasing sequence (see Appendix A for a list of all key notations used in the text).

A complementary process occurs where there is a negative feedback loop that has a moderating effect on the generated emotion. For instance, if B responded with a lower level of valence, A might respond by decreasing the emotional valence to meet a certain cultural ideal of emotional equilibrium. These different patterns of emotional–relational positioning may be represented by a few simple ordinal patterns and their transition probabilities. In this paper, we describe how we modeled the different emotional–relational patterns by using ordinal patterns and used them to form and test a simple model for predicting change in emotion during conversations. We automatically tested and validated our model on massive corpora of Chinese Mandarin and American English.

1.2. Emotion Dynamics and the Prediction of Change

As previously proposed, emotion dynamics [14], or the way emotions unfold during an interaction, may be modeled through a series of feedback loops, where linguistic signaling of emotion by the interlocutors is fed back into the system (i.e., the interacting dyad) to support the coordination of the composing units (i.e., the interacting individuals). This proposed modeling approach relies on a vast literature pointing to the role of feedback in emotion and emotion regulation [14,15,16,17,18,19]. In this context of emotion dynamics, anticipating a change in the interlocutors’ emotion is important in calibrating interactions, specifically in cultures such as China, where “the expression of emotion is carefully regulated out of concern for its capacity to disrupt group harmony and status hierarchies” [20] (p. 245) and emotions play a crucial role in face-work and trust building [21].

At the most basic level, anticipating a change in valence [22], which is the most fundamental dimension of emotion and relates to the pleasantness or unpleasantness of a signal, may be highly important for relational alignment in an interaction. Valence is “one of the most important scientific concepts at the heart of emotion experience” [23] (p. 83) and captures the essential aspect of affect [24], which may be highly important to use and anticipate during a conversation.

While change point detection has been studied in the context of interpersonal relationships [25], such studies do not usually focus on the prediction of change from a social relational perspective but on the identification of change or change point detection. In this paper, we develop a simple computational model for predicting change in valence during a conversation. More specifically, we focus on predicting the direction of change and whether a valence will increase or decrease. This model is grounded in the psychological–social–relational approach to language and emotion. Therefore, and given our theoretical framework, to predict a change in emotion (i.e., an increase or decrease in valence), we must model a nonlinear system where information, in the form of verbal signaling of valence, is fed back and forth between the interlocutors and constitutes a series of feedback loops regulating the emotion dynamics in a way that may be used to predict change. To address this modeling and prediction challenge, we first use the idea of ordinal patterns and explain how they can be used to model feedback loops and predict changes in emotion.

2. Ordinal Patterns

The proposed model is grounded in the representation of a time series through ordinal patterns [26,27,28,29,30,31,32]. As proposed in the seminal paper of Bandt and Pompe [26], a time series may be converted into a set of ordinal permutation patterns that may be highly informative about the dynamic of the system. Ordinal patterns have become a useful tool for analyzing time series in various domains and across numerous applications, given their ability to handle noise and extract important information with minimal assumptions. We first introduce and explain the idea of ordinal patterns and then point to their importance for short-term prediction as detailed by Neuman, Cohen, and Tamir [30].

2.1. Representing a Time Series through Ordinal Patterns

Our presentation draws on that of Neuman, Cohen, and Tamir [30]. Given a one-dimensional time series, S(t) of length, such as a time series of valence expressed by the sequence of turns produced by two interlocutors, we partition the series into overlapping blocks of length D (the embedding dimension) using a time delay τ. Consider the following time series, which uses D = 3 and τ = 1:

S(t) = {34, 3, 5, 23, 247, 234, 12, 1, 2, 3}

This can be broken down into a sequence of overlapping blocks:

The elements in each block or vector are then sorted in ascending order, and the vector is mapped into one of D! permutations (i.e., π_i), each representing the ordinal pattern of the elements. For D = 3, there are six possible permutations:

π₁ = {0,1,2}

π₂ = {0,2,1}

π₃ = {1,0,2}

π₄ = {1,2,0}

π₅ = {2,0,1}

π₆ = {2,1,0}

The first partition in the above time series—{34, 3, 5}—is mapped into the permutation pattern π₅ = {2,0,1}; the second partition—{3, 5, 23}—is mapped into π₁ = {0,1,2}; and so on. This results in a symbolic sequence of permutations: {π_s} s = 1…n. The mapping of the above time series, therefore, produces a time series of permutations:

{34, 3, 5, 23, 247, 234, 12, 1, 2, 3} → {2,0,1},{0,1,2},{0,1,2},{0,2,1},{2,1,0},{2,1,0},{2,0,1},{0,1,2}

2.2. Ordinal Patterns and Short-Term Prediction

The idea of mapping (i.e., representing) a time series of values into a time series of permutations may be highly relevant to short-term prediction in natural environments [30], specifically through the constraints imposed on the transition from permutation π_N to the next overlapping permutation: π_N+1. For example, in the above time series of permutations, the first transition is from permutation {2,0,1} to permutation {0,1,2}. While one might naively assume that a transition from each of the six above-mentioned permutation types to any of the six permutation types is possible, this belief is wrong. Each of the six above-mentioned permutation types may move to one of only three permutation types. This inherent constraint substantially reduces the potential number of transitions from one permutation type to the next, hence potentially improving the prediction of the π_N+1 permutation in a symbolic sequence of permutations: {π_N}N = 1, …, n.

For the case of D = 3 and τ = 1, which is the focus of our paper, there are only three legitimate transitions for each permutation. For example, the second partition that we previously identified—{3, 5, 23}—is mapped into π₁ = {0,1,2}, where the order of the elements in the permutation is such that e₁ < e₂ < e₃. The following partition/permutation (π_N+1) overlaps with the previous two elements of permutation π_N; therefore, its first two elements must be ordered such that e₁ < e₂ and the only degree of freedom is left to the third element. Among the six possible permutation types of D = 3 and τ = 1, there are only three permutation types that are consistent with this constraint: {0,1,2}, {0,2,1}, and {1,2,0}. What it is important to realize is that the constraints imposed on the transition from one permutation to the next significantly reduce the uncertainty associated with the next permutation. The list of legitimate transitions (D = 3, τ = 1) from each permutation type to the next is presented in

Table 1. A list of legitimate transitions from a given permutation type (D = 3, τ = 1).

Permutation	Legitimate Transition to
{0,1,2}	{0,1,2}	{0,2,1}	{1,2,0}
{0,2,1}	{1,0,2}	{2,0,1}	{2,1,0}
{1,0,2}	{0,1,2}	{0,2,1}	{1,2,0}
{1,2,0}	{1,0,2}	{2,0,1}	{2,1,0}
{2,0,1}	{0,1,2}	{0,2,1}	{1,2,0}
{2,1,0}	{1,0,2}	{2,0,1}	{2,1,0}

.

The constrained transitions may be important for predicting a change in emotion. Let us explain and illustrate this point using a specific example. Imagine a triadic sequence of sentiment measurements in a conversation:

B₁-A₁-B₂

where A and B are two distinct individuals and each letter, with its accompanied number, represents a measure of valence/sentiment extracted from the text produced by each interlocutor in turn. For example, if we measure sentiment on a scale ranging from 0 (i.e., negative) to 1 (i.e., positive), then the series might look as follows: 0.4, 0.1, 0.2. The sequence may be represented as the ordinal pattern (i.e., permutation) π₅ = {2,0,1}. The three legitimate transitions from this pattern are to:

{0,1,2} {0,2,1} {1,2,0}

It, therefore, follows that, in two out of the three legitimate transitions (66%), the transitions are such that we should expect to see an increase in sentiment from A₁ to A₂. Given our full ignorance, we can bet that, in 66% of the cases, A’s sentiment will increase from A₁ to A₂. In practice, the transition probabilities may be different, given the actual transition probabilities that appear in the sequence. For instance, in analyzing the multi-party dialogue dataset (MPDD) [33], we found that the probability of transition from π₅ to π₁ or to π₂ is p = 0.86. This transition probability is higher than expected under the assumption of full ignorance, and it significantly improves our ability to predict an increase in sentiment from A₁ to A₂, given that the preceding overlapping permutation (B₁-A₁-B₂) is π₅. As it has been shown that the human brain is sensitive to probabilities and transition probabilities [34,35], the constraints described above may be important in modeling the direction in which emotion changes and may also inspire the design of simple and robust automatic systems for detecting change during a conversation.

In sum, given a triadic sequence of turns, represented as the ordinal pattern permutation π_N = B₁-A₁-B₂, and transition to the next overlapping triadic sequence π_N+1 = A₁-B₂-A₂, the constraints imposed by π_N may be used to predict whether the change from A₁ to A₂ is an increase or decrease. This logic of ordinal patterns, and their actual transition probability, may have a highly important function in short-term prediction, as argued and illustrated by Neuman, Cohen, and Tamir [30], as well as Neuman and Cohen [36], in the context of financial data analysis. Therefore, the novelty and main contribution of this paper is to identify a simple predictive model of change in emotion in a nonlinear, short (N_turns < 100), and interactive time series of turns that is grounded in (1) ordinal pattern representation, (2) scientific understanding of the processing of emotions and sequences, and (3) our ability to automatically identify simple explanatory rules through the symbolic regression approach.

3. General Outline of the Paper

We first present the pre-processing of the data and validate the performance of the Chinese sentiment analysis tool (SnowNLP) that we used. Next, and for our first dataset, we:

• Present the probative evidence in support of the predictive value of π_N;
• Present the predictive value of π_N and its components by fitting a binary logistic regression model and using three decision-based machine learning (ML) models;
• Extract a simple predictive mathematical model from the data using a symbolic classification analysis and testing the predictive value of the model on two other datasets.

Finally, we use the symbolic regression approach, build a predictive symbolic expression for a dataset, and use the expression for a ML model tested on the other datasets. We repeat this process for the two other datasets used in this study.

4. Methodology

4.1. Pre-Processing and the Generation of Ordinal Patterns

Our first step was to represent each conversation as (1) a time series of emotional valence. Our next was to convert the time series into (2) a time series of ordinal patterns (D = 3, τ = 1). We used the MPDD for our first main experiment.

Each conversation was converted into a series of turns, and each turn was scored by SnowNLP. In this way, each conversation was represented as a time series of emotional valence. The length of the conversations ranged from 4 to 20 turns, with an average of 11 and SD of 4.5. Given the inherent constraints of ordinal pattern analysis, for all datasets we analyzed conversations of length ≥ 6 turns.

Our analysis proceeded as follows. We first automatically measured the valence of each turn and, for each conversation, produced a time series of the interlocutors’ sentiment. Next, we identified triadic sequences of turns involving two interlocutors: A and B. Each triadic sequence of valence measurements was represented as A₁-B-A₂, where, sequentially, A is the speaker and B is the listener, B is the speaker and A is the listener, and A is the speaker and B is the listener. See, for example, the following triadic sequence:

• A₁:
• 正鵬, 昨天晚上你姨媽已經去世了。明天下午我們一起去致哀啊！
• Translation: Zhengpeng, your aunt passed away last night. Let’s go pay our last respects and offer our condolences together tomorrow afternoon!
• B:
• 哎喲！姨媽太可憐了。但是，得了絕症這是誰也沒辦法的事，媽，你心裏也別難過。明天你們先去，我放學後再來。
• Translation: Oh dear, my poor aunt! However, there’s nothing one can do about being terminally ill. Mom, don’t be too sad. You guys go ahead [to visit her] tomorrow, and I’ll come later myself when I’m done with school.
• A₂:
• 正鵬，這件事你一定要跟你麗華商量，可你一定要去啊。姨媽死只有這一次。你就是再忙也要擠時間去呀！
• Translation: Zhengpeng, you have to discuss this with Lihua. However, you have to go [to the funeral/memorial]. [Your] aunt dies only this once. No matter how busy you are, you have to make time for it!

We can see that speaker A informs speaker B that his aunt has passed away and proposes that they should go to mourn with the family. B seems to try to avoid his obligation to behave in a socially accepted manner and A responds with anger. In this case, A increases the negative valence, relative to what she said in A₁. Her relatively high level of negative valance is a response to B’s attempt to avoid his social–moral obligations; as such, A’s change in valance signals to B that his response and behavior are inappropriate.

Following the literature pointing to a recognition threshold for emotion [37], we analyzed cases in which the difference in sentiment from A₁ to A₂ was ≥20% out of the range of valence (0–1). This heuristic decision was taken to avoid analyzing cases where the change did not cross a recognition threshold. Next, we used the time series representation of a series of ordinal patterns, identified the triadic sequence A₁-B-A₂, and used the permutation that precedes it (i.e., π_N) to predict the direction of change in emotion from A₁ to A₂.

4.2. Validation Phase

We first tested our model on two datasets of Chinese conversations. To automatically analyze the sentiment, we used the SnowNLP Python library (https://github.com/isnowfy/snownlp, accessed on 1 May 2022). Given the challenges of automatically measuring sentiment in Chinese [38], we decided to test the performance of SnowNLP on two datasets: the languages in the Universal Joy dataset [39] and MPDD [33]. Given that the datasets have slightly different emotion tags, and given the way we grouped them into categories (i.e., positive vs. negative, as explained below), the absolute valence scores should not be considered, instead only their relative differences across the valence tags should be considered.

4.2.1. Validation Test 1: The Languages in Universal Joy Dataset

We used the languages in the Universal Joy dataset [39] and analyzed Chinese Facebook posts tagged according to five emotions: anger, anticipation, fear, joy, and sadness. Anger, fear, and sadness were grouped and tagged as “Negative”, joy was tagged as “Positive”, and anticipation was removed from the analysis. Overall, we used SnowNLP to identify the valence of 2290 posts. If SnowNLP validly measures the sentiment of the posts, then a significant difference should have been observed in the SnowNLP scores of positive vs. negative posts. As can be seen in

Table 2. Mean sentiment scores produced by SnowNLP for negative (NEG) vs. positive (POS) posts (N = 2290).

						95% CI
	Group	N	Mean	SD	SE	Lower	Upper
Sentiment	NEG	690	0.47	0.41	0.02	0.44	0.50
	POS	1600	0.60	0.40	0.01	0.58	0.62

, positive posts scored higher than negative posts.

Given the violation of the normality assumption, we used the Bayesian Mann–Whitney U Test, with 1000 samples, in order to compare the sentiment scores of the two samples, i.e., positive (N = 1600) and negative (N = 690) posts. The difference between the sentiment scores of positive vs. negative posts was found to be statistically significant (p < 0.001), with no overlap between the lower and upper bounds of the CI. The effect size, as measured using the rank-biserial correlation, was −0.19, meaning that positively tagged posts “outperformed” negatively tagged posts, in terms of their sentiment score, in ~60% of the cases. This significant difference supports the validity of SnowNLP for Chinese sentiment analysis.

4.2.2. Validation Test 2: The MPDD

The MPDD is composed of TV series conversions and includes emotion tags that are associated with each produced utterance in each conversation. This dataset contains 25,548 utterances from 4142 dialogues, and we used it as a second validation set for SnowNLP. We measured the average sentiments associated with the different emotion tags and grouped the tags into three valence categories: neutral, negative, and positive.

The average score of the positive utterances (mean = 0.46, SD = 0.33, N = 4942) was higher than that of the neutral (mean = 0.40, SD = 0.40, N = 13,629) and negative (mean = 0.33, SD = 0.31, N = 6977) utterances.

Table 3. Mean of sentiment scores produced by SnowNLP for negative, neutral, and positive utterances (N = 25,548).

				95% CI
Tag	Mean	SD	N	Lower	Upper
NEG	0.33	0.31	6977	0.32	0.34
NEU	0.40	0.31	13,629	0.39	0.40
POS	0.46	0.33	4942	0.45	0.47

presents the mean SnowNLP scores for the tree samples with 95% CI.

As the equality of variance assumption was violated, we compared the sentiment means across the three groups of sentiment (i.e., negative, neutral, and positive) using the Kruskal–Wallis H Test. The test was found to be statistically significant (H = 484.68, p < 0.001), with Dunn’s post hoc comparison and the Bonferroni correction showing statistically significant differences between all conditions (e.g., neutral–negative). These results, produced for two datasets in Chinese, suggest that SnowNLP can validly be used to differentiate valence categories; therefore, we applied it to the analysis of our first test corpus, which was the MPDD. Note, that, given the different clustering of different emotions in the above datasets, it is irrelevant to consider the average valences of the categories (e.g., positive) in absolute terms; instead, it is necessary to examine the relative differences only, as we have done.

5. Measuring the Probative Value of π_N

Given the above theorization, we hypothesized that permutation π_N (i.e., B₁-A₁-B₂) is informative for predicting the direction of change in valence from A₁ to A₂. To test our hypothesis, we measured the probative value of π_N by determining its likelihood ratio (LR), as explained below.

Following Fenton and Neil [40], we first applied a Bayesian approach to the data, asking what was the probative value of the evidence that a certain permutation would appear before our target permutation, for predicting a decrease or increase of A’s sentiment at A₂. Our hypothesis was that the triadic ordinal pattern (i.e., permutation π_N) preceding and overlapping with the ordinal pattern constituting our target triadic sequence (i.e., A₁-B₂-A₂) would have a probative value, as evidence for predicting a change in valence from A₁ to A₂. For example, consider the following sequence:

B₁-A₁-B₂-A₂

This is composed of the two overlapping triadic sequences:

B₁-A₁-B₂ and A₁-B₂-A₂

As explained before, the transition from one ordinal pattern to the next involves constraints that limit the number of permutations observed after a given pattern. These constraints may be highly important for short-term prediction [30]. For example, the monotonic decreasing sequence {2,1,0} may represent the sequence A₁-B₂-A₂, where a decrease in A’s valence is observed. This permutation may follow only one of the three following permutations: {0,2,1}, {1,2,0}, and {2,1,0}. If there is a differential predictive value to each permutation, then their appearance may signal the direction of change in A’s valence.

To test our hypothesis, we first measured the association between π_N (1–6) and the direction of change (UP vs. DOWN). As the percentage of cases where no change existed was less than 1%, these cases were removed from the analysis in all datasets used in this study. The association between π_N (1–6) and the direction of change (UP vs. DOWN) was found to be statistically significant (χ² (5, N = 4581) = 1391, p < 0.001).

Figure 1 presents the conditional probability of observing an increase (UP) or decrease (DOWN) in sentiment from A₁ to A₂, given one of three permutations that has been found to be the most predictive. Permutation 6 has been assigned to the UP direction because we hypothesized that a monotonic increasing sequence may lead to an upward change. Interestingly, it can be observed that the probability of an increase is higher when the triadic sequence (A₁-B-A₂) is preceded by permutations 3 (i.e., {1,0,2}) or 5 (i.e., {2,0,1}), and the probability of observing a decrease in sentiment is higher when the triadic sequence is preceded by permutations 2 (i.e., {0,2,1}) or 4 (i.e., {1,2,0}). These findings may be highly important for understanding emotion dynamics and could not have been ascertained without the use of ordinal patterns.

The scientific meaning of this finding can be clarified when we realize that the common denominator of the permutations significantly preceding an increase may be understood if we denote the tetradic sequence forming the triadic sequence and its overlapping previous triadic sequence as:

B₁-A₁-B₂-A₂

For the permutations preceding the increase (i.e., π_N), we can see that:

B₁ > A₁ and B₂ > A₁

For the permutations preceding the decrease, we can see that:

A₁ > B₁ and A₁ > B₂

In terms of a feedback loop, we can see that permutations 2 and 4 (see Figure 2) involve a negative feedback loop, calibrating A to a lower level of emotion. In contrast, permutations 3 and 5 are “lifters” that calibrate A to a higher level of emotion. In both cases, we have negative feedback loops that direct A to some kind of equilibrium.

Based on this finding and its speculated underlying logic, and to further measure the predictive value of π_N, we defined a new feature, SIGNAL, that scores DOWN when π_N = 2 or 4 and UP when π_N = 3 or 5. The association between SIGNAL and the direction of change was found to be statistically significant (χ² (2, N = 4581) = 1382, p < 0.001).

What is the probative value of the evidence that is provided by π_N? In the dataset, the probability of a decrease or increase in valance from A₁ to A₂ is ~0.5, and, therefore, the odds for anticipating a decrease or increase, are:

$$\frac{P\left(H_{Increase}\right)}{P\left(H_{Decrease}\right)}=\frac{P\left(H_{Decrease}\right)}{P\left(H_{Increase}\right)}=1$$

(1)

These odds roughly characterize all datasets used in this study. We calculated the LR, which is considered to be both important and meaningful for measuring the probative value of evidence [40]:

$$\frac{P(E|Hp)}{P(E|Hd)}$$

(2)

In our case, where the evidence (E) is SIGNAL, this can be depicted as:

$$\frac{P(SIGNAL\uparrow |H\_Increase)}{P(SIGNAL\uparrow |H\_Decrease)}$$

(3)

or

$$\frac{P(SIGNAL\downarrow |H\_Decrease)}{P(SIGNAL\downarrow |H\_Increase)}$$

(4)

The LR for increase, which is the positive LR, was found to be 5.5, and the LR for decrease was found to be 3.9. As explained by Fenton and Neil, “If the LR > 1 then the evidence E results in an increased posterior probability of Hp” [40] (p. 4). As in our case LR > 1, we may calculate the odds version of Bayes:

Posterior odds of Hp = (Prior odds of Hp) × Likelihood ratio

(5)

which, in our case, and given the odds of 1, are the same as the LR ratio. This means that if we start with odds of 1 in favor of an increase or decrease in emotion, then by knowing π_N, we can improve our odds (i.e., prediction) of anticipating an increase by a factor of 5.5 and by a factor of 4 for a decrease in emotion. These results show that the pattern of the emotion dynamics preceding A₂’s utterance has a clear probative value for predicting the direction in which A’s emotion will change, as expected by the feedback loop hypothesis.

6. Using the Binary Logistic and ML Models

Given the probative value of π_N, we may use it and its components (i.e., B₁, A₁, and B₂) to build and test various predictive models. We first used a backward binary logistic regression model with EmotionChange (UP/DOWN) as the dependent variable. For all of the statistical analysis and ML models reported in this section, we used JASP 0.16.2 (https://jasp-stats.org, accessed on 1 May 2022). The model was found to be statistically significant (p < 0.001), with 85% accuracy and precision and 86% recall. The only significant predictors were π_N and A₁. We tested the predictive power of these two features using three ML models. A boosting classification model with tenfold cross-validation gained 88% accuracy, precision, and recall. A decision tree model with 80% of the data used for training gained 85% accuracy, 86% precision, and 85% recall. A random forest model with 80% of the dataset used for validation gained 85% accuracy, precision, and recall.

7. The Symbolic Classification Analysis

Given the successful venture in science to automatically identify simple rules (i.e., equations) governing the behavior of systems [41,42,43] and the idea of “distilling free-form natural laws from experimental data” (the title of [42]), we can ask whether we can test our theorization and further simplify our model by using a symbolic classification analysis. We used HeuristicLab’s (https://dev.heuristiclab.com, accessed on 1 May 2022) Optimizer 3.3.16 [44] to perform a symbolic classification analysis with a maximum symbolic expression length of 5 and maximum symbolic tree depth of 5. By system default, 66% of the dataset was used to train the model and the rest for the test. The model ran for 1000 generations; through examining 99,100 solutions, it found that the best model of classification was a symbolic discriminant function classification solution with 84% accuracy, 81% precision, and 83% recall. The relative variable frequencies across generations appear in Figure 3.

The symbolic expression produced by this analysis is:

$$EmotionChange=\left(c0+c1\ast A1\right)$$

(6)

where

c₀ = 0.95649

c₁ = 1.06270

Its classification threshold is shown in Figure 4.

As the choice of D = 3 was guided by previous work on short-term prediction [30], it was worth examining whether increasing the embedding dimension to 4 would change our results. We, therefore, reanalyzed the MPDD, this time examining the predictive value of permutation length 4 (i.e., D₄) preceding A₂. In this case, we analyzed dialogues where the triadic sequence A₁-B₂-A₂ was preceded by B₁ and A₋₁. Therefore, the D₄ permutation preceding A₂ was composed of the sequence A₋₁-B₁-A₁-B₂, and it was assigned one of the 24 numbers indicating the potential permutations of D₄.

A statistically significant association was found between the direction of change (UP/DOWN) and D₄ permutation preceding A₂ (χ² (23, N = 3913) = 1428.46, p < 0.001). Using the D₄ permutation, with its 24 possible values, in a boosting classification model that included the D₃ permutation preceding A₂ and valence of A₋₁-B₁-A₁-B₂, we were able to successfully predict change with averages of 86% accuracy, precision, and recall. A decision tree model gained an average of 84% for the performance measures, and a random forest model gained an average of 82% for the performance measures. In all of these models, the most important feature was the valence of A₁, followed by permutation D₄. Using a symbolic classification analysis, we gained 84% accuracy, with D₄ showing lower variable relevance than D₃ (0.021 vs. 0.119, respectively). It must be noted, however, that in analyzing permutation D₄, our dataset shrank by 668 cases, which was 15% of the original dataset. Therefore, although permutation D₄ may be of predictive value for a change in emotion, it may be less relevant to very short conversations, such as those we analyzed for the MPDD with an average length of 6. In addition, in this paper, we do not focus on performance measures, per se, but seek to identify a simple and explainable model. While the contribution of permutation D₃ can be explained through a simple and theoretically grounded model of feedback loops, permutation D₄ (with its 24 potential patterns) might be more difficult to incorporate into a simple and explainable model.

The next section tests the validity of the model formed for permutations of length 3 using two additional datasets.

8. Testing the Simple Model

8.1. Analysis 1: The NaturalConv Dataset

To validate our model, we used the NaturalConv dataset [45]. The corpus contains 19,900 conversations in Chinese, covering six topical domains from sport to health and education. The corpus contains 400,000 utterances, with an average turn number of 20.1. For the analysis, we applied the same procedure as presented before. First, we tested the predictive value of π_N for this dataset. Figure 5 presents the conditional probability of a change, given π_N. In this case, too, a significant association was found between π_N and the direction of change in emotion (χ² (5, N = 182,764) = 58,468, p < 0.001).

The baseline prediction for the model was DOWN = 47% and UP = 53%. Using a backward binary logistic analysis, the model was found to be statistically significant (p < 0.001) with 87% accuracy, 89% recall, and 87% precision. All components of π_N were found to be significant predictors. The three ML models (boosting classification, decision tree, and random forest) applied with these features, respectively, gained an average of 88% accuracy, precision, and recall. All these ML classifiers identified A₁ and π_N as the two most significant features in the model.

Next, we used the mathematical model formed through the symbolic classification analysis of the MPDD, as well as the equation/symbolic expression identified by the regression classification, to compute a new target variable: EmotionChange. Using the score generated through the equation as the only feature in a classification and regression tree (CRT) model with tenfold cross-validation gained 88% accuracy, 83% precision, and 96% recall for predicting a change UP, with 95% precision and 78% recall for predicting a change DOWN (we used IBM’s SPSS for the CRT). This finding indicates that the very simple symbolic expression formed for the MPDD may validly be used to predict change in emotion for the NaturalConv dataset.

8.2. Analysis 2: The EmotionLines Dataset

For the English dataset, we chose EmotionLines [46]. This dataset is composed of 1000 dialogues taken from the successful TV series Friends and 1000 dialogues from EmotionPush chat logs. For the analysis of sentiment, we used FlairNLP [47] (https://pypi.org/project/flair, accessed on 1 May 2022). As Flair basically provides a binary classification (0 or 1), we used a heuristic to make these scores continuous by multiplying each class by its corresponding confidence score and multiplying the result by the corresponding sign. For instance, an utterance tagged as 0 with a confidence score of 0.70 was transformed to become −0.70. This heuristic was expected to amplify the score.

We first examined the conditional probability of a change, given the preceding permutation, as shown in Figure 6. Again, the association between the direction of change and π_N was found to be statistically significant (χ² (6, N = 3509) = 1160, p < 0.001).

The baseline prediction for the model was DOWN = 53% and UP = 47%. Using a backward binary logistic analysis, the model was found to be statistically significant (p < 0.001) with 97% accuracy, 96% recall, and 98% precision. All components of π_N were found to be significant predictors. The three ML models (boosting classification, decision tree, and random forest) applied with these features, respectively, gained an average of 96% accuracy, precision, and recall. All these ML classifiers identified A₁ and π_N as the two most significant features in the model.

Applying the same mathematical equation produced for the MPDD and same procedure as applied to NaturalConv, through the CRT model, we gained 97% accuracy, 98% precision, and 96% recall for an UP change, with 96% precision and 98% recall for a DOWN change.

8.3. Analysis 3: Concluding Analysis

Table 4. Concluding table showing the model performances. The baseline for increase (“UP”) in sentiment from A1 to A2 is 50%, 53%, and 47% (for the MPDD, NaturalConv, and EmotionLines, respectively).

Dataset	Expression	Test Dataset and Accuracy
		NaturalConv	EmotionLines	MPDD
MPDD	$\left(\mathrm{c}0+\mathrm{c}1\ast \mathrm{A}1\right)$	88%	97%	-
NaturalConv	$\mathrm{if}(\left(\mathrm{c}0\ast \mathrm{A}1<\mathrm{c}1\right), \mathrm{c}2, \mathrm{c}3$	-	97%	73%
EmotionLines	$\left(\mathrm{c}0-\mathrm{c}1\ast \mathrm{A}1\right)$	88%	-	86%

shows a summary of our analysis. For each dataset, we have built a mathematical model using the symbolic classification analysis and tested the model on the two other datasets by using the score generated through the equation produced for the source dataset (see the first leftmost column in the table), as a single feature in a CRT ML model with a tenfold cross-validation procedure. In other words, we measured the validity of our original model by testing it on the two other datasets.

For the NaturalConv dataset, the simplest model with tree depth/length set to 8 was:

$$EmotionChange=if(\left(c0\ast A1<c1\right), c2, c3$$

(7)

where:

c₀ = 0.38971

c₁ = 0.27942

c₂ = 0.99427

c₃ = 0.10011

In other words:

IF (c₀ ∗ A1) is LessThan c₁ Then c2 Otherwise c₃

with a linear discriminant analysis solution as the best model, with 88% accuracy, 93% precision, and 79% recall for DOWN and 84% precision and 95% recall for UP.

For the EmotionLines dataset, and with a maximum tree depth/length of 5, the equation was:

$$EmotionChange=\left(c0-c1\ast A1\right)$$

(8)

where:

c₀ = 1.0623

c₁ = 1.1201

with a symbolic discrimination function classification as the best model, with 97% accuracy, 96% precision, and 78% recall for DOWN and 98% precision and 96% recall for UP. By validating each model through the other datasets, as explained above, we gained the performance shown in Table 4. This shows 88% average accuracy across all measurements, where accuracy is simply measured as (TP + TN)/(TP + TN + FP + FN), as is common in a ML/NLP research.

9. Conclusions

We first open our conclusion section by summarizing our (1) research novelty, (2) contribution, (3) analysis, and (4) results. The novelty of the paper is in introducing a new approach for identifying a change in emotion during a conversation. The novelties of the approach are in modelling change using ordinal patterns and their components, the automatic discovery of simple mathematical “rules” explaining the change in emotion, and the use of these rules, discovered through the symbolic regression approach, for producing a single feature/variable to be used in a simple ML model (i.e., the CRT). We contribute to the literature by showing how simple “rules of emotion” can be automatically identified and used for producing predictive features in ML models. Moreover, our analysis involves the use of three different datasets. For each dataset, we automatically discover the “rule” and use it for producing predictive features for the two other datasets. The success of the predictive features is tested using a simple CRT ML model, and our results show that the rules discovered for one dataset and tested on the other two gain an average of 88% accuracy, which is far above the baseline for prediction. To the best of our knowledge, such an approach, which is supported by a strong validation methodology and results, has not been presented in the scientific literature dealing with the detection of change in emotion during a conversation. Given this summarization, we now turn to a high-level discussion of our work.

In this paper, we show that the direction of change in emotion may be predicted, even for very short conversations. The proposed approach is novel, as it uses ordinal patterns to model feedback loops and permutations of emotions and their limited number of components to build predictive models. Therefore, our approach is not only grounded in domain expertise relating to the subject matter (i.e., emotion dynamics) but also in building theoretically grounded and simple predictive models, based on automatic rule discovery through the symbolic regression approach, as well as the validation and testing of the models through a simple ML model (i.e., the CRT). Indeed, using the symbolic regression approach, we were able to identify simple symbolic mathematical expressions that can be used to generate a single predictive feature in a CRT ML model. Through automatic discovery of such an expression in one dataset and use of the expression to compute and test a single predictive feature in the two other datasets, it is possible to gain powerful support for the validity and usefulness of the symbolic expression. The 88% average accuracy in prediction across the six validating measurements (see Table 4) supports the thesis that a change in emotion can be predicted with simple and theoretically grounded models. Therefore, the predictive power of the simple models identified in this study is further evidence of the potential of simple predictive models [36,48] and how the development of such models can be applied with the purpose of “living in a world of low levels of predictability” [49] (p. 15).

Modern information technologies have changed our lives in areas from medical diagnosis [50] to blockchain-based systems [51], ways to identify cracks in dams [52], fetal ultrasound standard plane recognition [53], and Big Data architectures providing a platform for such capabilities [54]. In this context, simple models for detecting the direction of change in emotion may be applied in various contexts. For instance, mobile crowdsensing [55] could use our simple approach to monitor the emerging emotional change in a crowd or identify emerging frustrations of users, applying a context model for intelligent campus navigation [56]. The same approach could be used to monitor emotion during crowd evacuation [57] and for many other purposes.

It is important to note that we tested our models against baselines of increases or decreases in emotion. Other models for identifying change in emotion exist. For example, Yu and Zheng [58] developed a deep multimodal network (ECPNet) for predicting change in emotion. However, we cannot compare our results to those gained by these researchers because they used visual and acoustic signals of emotions, while our study focuses on changes in valence expressed in textual data. The same holds for a potential comparison with the work of Huang and Epps [59], who developed a system for detecting change in emotion using speech signals. Our study is, therefore, limited to the identification of the direction in the change of valance by relying on textual data and focusing on the attempt to generate simple and theoretically grounded predictive models, rather than the best predictive models. We plan to compare our approach to those in the above-mentioned studies by extending our analysis to multimodal datasets.