Next Article in Journal
Tocilizumab in Extracranial Giant-Cell Arteritis and Takayasu Arteritis: A Multicentric Observational Comparative Study
Previous Article in Journal
Exercise as Medicine: Tackling Chronic Pain and Depression to Boost Quality of Life—A Narrative Review
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios

Department of Computer Science, Tokyo Metropolitan University, Tokyo 191-0065, Japan
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Submission received: 20 October 2024 / Revised: 3 January 2025 / Accepted: 9 January 2025 / Published: 15 January 2025
(This article belongs to the Section Computer Sciences, Mathematics and AI)

Abstract

:
Skin Conductance Response (SCR) is a physiological index associated with arousing emotions. Previous studies have not explored the relationship between SCR signals and emotions in situations where multiple emotions dynamically fluctuate. Moreover, methods suitable for analyzing such conditions have not yet been established. In this study, we recorded the temporal changes in multiple emotions as subjectively reported by participants using the Temporal Dominance of Emotions (TDE) method. We then matched these subjective reports with the evolving SCR signals through regression analysis. This approach reveals which emotions contribute to increased SCR signals in prolonged, emotionally charged scenarios, such as watching videos or playing video games. To validate our method, we recorded SCR signals while participants played a video game. Participants then performed the TDE task to recall their emotions while viewing recorded footage. This study involved 20 participants. Our analysis showed that emotions such as excitement, tension, and frustration significantly covaried with the physiological signals. These arousing emotions are known to evoke SCR, supporting the validity of our method. This approach introduces a novel experimental methodology for comparing subjective reports and high-responsive physiology signals in settings where multiple emotions dynamically change.

1. Introduction

Emotions are closely linked to physiological activities [1,2]. Many studies on emotions use physiological signals, including blood flow [3,4,5], electrocardiogram [6], electromyography [7,8,9], electroencephalography [10,11,12,13], and electrodermal activity [10,11,12,13]. Additionally, unconscious changes in facial expressions [2,14,15,16] can also be considered part of physiological responses.
Skin Conductance Response (SCR) is the AC component of skin conductance level and a highly responsive signal. SCR begins to increase 1–3 s after stimulus presentation and reaches its peak about 1–3 s afterward [17]. SCR reflects electrical activity in the skin, which varies with changes in sweating linked to arousal, and is associated with arousing emotions such as fear and excitement [18,19]. Despite its quick responsiveness, the majority of studies use skin conductance levels, the DC component, to compare physiological states before and after stimulus presentation [1].
The skin conductance level and response are typically used to investigate changes in bodily states in response to stimuli that can evoke certain types of emotions [1,12,13,20,21,22,23,24,25]. For example, Makioka et al. [13] examined the intensity of subjective fear caused by horror videos by analyzing the peaks of SCR signals. Drummond [20] reported that SCRs were associated with anger triggered by provocative comments from others. Collet et al. [21] observed significant changes in SCR when participants viewed facial expressions of anger. In the study by Blechert et al. [22], SCR signals were evoked by anxiety conditioned to electric shock stimuli. In many of these studies, SCR signals were analyzed in response to emotional stimuli designed to elicit specific emotions, such as fear, anxiety, and anger. Thus far, however, few studies have used SCR signals to investigate responses to prolonged stimuli, such as films, that may evoke multiple types of emotions over time. One notable exception is in the context of learning, where electrodermal activities are typically measured to investigate physiological changes during a prolonged learning process lasting several to tens of minutes [26].
During activities that take a certain amount of time, such as watching movies, listening to music, or playing games, various emotions can arise. In games, emotional content and timing change interactively. To evaluate human emotional changes in response to stimuli that dynamically involve multiple emotions, the Temporal Dominance of Emotions (TDE) method [27,28,29] can be used. The TDE method allows participants to record multiple types of emotions that change on a second-by-second basis while experiencing a stimulus. For example, in the experiment by Galmarini et al. [28], participants recorded emotional changes while tasting coffee under two different music conditions and a silent condition. In the experiment by Merlo et al. [29], emotional changes during hamburger consumption were investigated in relation to differences in packaging color. The TDE method is a unique approach that allows experimental participants to report when and which type of emotions occur over time. However, no established method exists to directly connect TDE evaluations with physiological indices. By utilizing SCR, which exhibits quick responsiveness, the temporally evolving emotions recorded by the TDE method can be supported from the perspective of bodily activities.
As the first step of this study, we propose a method to match the results of the TDE method with time-series SCR signals. For this purpose, we focus on the waveform profile of SCRs. Bach et al. [30] proposed a modeling method using exponential functions and a Gaussian filter for SCR signals elicited by unpleasant electric stimuli. Furthermore, they suggested that changes in SCRs caused by emotional stimuli presented successively, with intervals of a few seconds, can be modeled as a linear combination of the SCR responses to each individual stimulus [30,31]. In experiences such as watching movies, listening to music, or playing games, emotionally charged scenes may continually occur within short periods. A succeeding scene may start before the SCR activity related to the preceding one fully converges. In such situations, the linear model proposed by Bach et al. can be utilized [30,31,32].
This study proposes a method for dynamically matching subjectively reported emotional changes with SCR signals during the experience of emotionally charged content. In the experiment, participants, wearing an SCR measurement device, played a video game designed to evoke multiple types of emotions. They reported the moments when their emotions changed using the TDE method. Our method aligns TDE tasks with SCR signals over time, with different emotions exerting varying impacts on the SCR. This analysis statistically identifies the types of emotions reflected in SCR activities, offering a novel tool for studies on emotional responses.

2. Methods

2.1. Ethical Statement

The protocol of this study was approved by Institutional Review Board, Hino Campus, Tokyo Metropolitan University (H23-11).

2.2. Apparatus

SCR measurements were performed using a dermal activity sensor (AP-U030m II, Nihon Suntech, Osaka, Japan, active frequency range: 0.032–15 Hz) and an amplifier (MaP1720CA, Nihon Suntech, Osaka, Japan). The skin conductance unit was controlled by a data acquisition device (NI USB-6211, National Instrument Corp., Austin, TX, USA) and MATLAB (R2023a, Mathworks Inc., Natick, MA, USA, D a t a A c q u i s i t i o n T o o l b o x ) at the control frequency of 1000 Hz.
The game was displayed on a 21-inch monitor positioned 60 cm from the participant’s head, with sound delivered through headphones. Participants controlled the game avatar using a handheld Xbox Wireless Controller (Microsoft Corp., Redmond, WA, USA).

2.3. Stimuli: Emotionally Evocative Gamification in Minecraft

We utilized a battle and exploration stage developed in Minecraft (Microsoft Corp., Redmond, WA, USA) as the emotionally evocative, interactive gaming content. Figure 1 shows gameplay images. Players were tasked with locating treasures hidden within a dungeon, during which computer-controlled enemies would randomly appear and attack the player’s avatar.
Players collected diamonds from treasure chests placed on each floor of a small three-story house to reach the top floor. Each floor contained 4–6 boxes with diamonds or dummy items, the contents of which were randomly determined. When checking the contents of the item boxes, players may experience emotions such as joy, relief, dominance, or disappointment.
Computer-controlled enemies attacked and interrupted the player’s avatar inside the dungeon, potentially evoking emotions such as excitement, tension, and anger when attacked by hostile characters; anger, disappointment, and frustration when knocked down; and dominance, relief, and joy when defeating them. Enemies appeared randomly when the player was within a certain area of the dungeon, and multiple zombies could attack the player’s avatar simultaneously. Players could defeat a zombie by attacking it 5–6 times. The avatar’s life points decreased when attacked by enemies. If the avatar lost all its life points, it was sent back to the starting point, located outside the house.
Similar stages were used in our earlier study [33]. The game’s difficulty, including the number of enemies and their strength, was adjusted to ensure the content was emotionally evocative. This arrangement was determined by consensus among the authors and two of their colleagues.

2.4. Temporal Dominance of Emotions (TDE) Method

We employed the Temporal Dominance of Emotions (TDE) method [28,34] to record the temporal evolution of emotions. This method captures emotional changes in real time on a second scale during stimuli exposure and has been primarily used in food science, particularly during food and beverage tasting [28,35,36]. Recently, the TDE method has been applied to record emotional experiences during video game play [33,37].
As shown in Figure 2, in the TDE task, participants sequentially selected emotional attributes from a list displayed on the computer monitor. They selected the attribute corresponding to the most dominant emotion at any given moment and selected a new attribute whenever the dominant emotion changed. Once an attribute was selected, it remained active until a different one was selected. At each moment, only one attribute could be selected, and some attributes might not be chosen at all by the participants.
Ten emotional attributes were used: dominant, confused, relieved, angry, frustrated, disappointed, joyful, tense, excited, and relaxed. The descriptions of these attributes for the participants are provided in Table 1. Dominance was defined as the feeling of superiority and confidence.
The process for selecting these emotional attributes was as follows. Each of the five individuals, consisting of the authors and three of their colleagues, played the game and then listed the emotions they felt during gameplay. Through consensus among the five members, the emotional attributes were narrowed down to 30. Each person then selected all the emotions they frequently experienced during gameplay in a check-all-that-apply manner. Finally, by majority vote among the five members, the emotional attributes were reduced to 10. It is noted that these five individuals did not participate in the main experiment.
Prior to the TDE evaluation, participants familiarized themselves with the emotional attributes and their corresponding positions on the screen. If no significant emotional response was felt, participants were encouraged to select the relaxation button.

2.5. Participants

A total of 20 university students (mean age: 23.7; 10 females), who were unaware of the experiment’s purpose, participated in the study. Informed consent was obtained from all participants prior to their involvement.

2.6. Procedures

Participants attached electrodes to the left inner foot to measure SCR and rested for ten min. This area was used as a substitute for the fingers since the participants held the controller with both hands [38,39,40]. They practiced the game controls for three min. After practice, participants rested for 1–2 min until their SCR reached a steady state.
The task for participants was to collect as many diamonds as possible from treasure boxes within the house in 150 s. Note that they could not complete the entire game within this time period. Participants were instructed not to move their left foot during the gaming experiment to prevent SCR fluctuations due to poor electrode–skin contact.
Immediately after the game, participants assessed their emotional changes over time using the TDE method while referring to a recording of their gameplay. They were instructed to recall and report the emotions they experienced during gameplay, rather than their current emotions while watching the video.

2.7. Data Preprocessing of SCR

SCR signals were downsampled from 1000 Hz to 100 Hz, and a low-pass filter with a cutoff frequency of 1 Hz was applied.

3. Dynamic Matching Between TDE and SCR Waveforms

In this study, we describe the method for estimating SCR waveforms based on the temporal changes in emotions assessed using the TDE method. Regression analysis was employed for this purpose as it provides a direct and statistical means to evaluate the relationship between SCR and the emotions reported by participants during the TDE task. Although regression analysis may lack the flexibility and nonlinear capabilities of certain machine learning algorithms, it offers a straightforward approach to analyzing these relationships. For each measurement task—specifically, each gameplay session—the regression analysis was conducted using the n emotions as explanatory variables.
As shown in Figure 3a, the results evaluated by the TDE method are stored as a binary function for each evaluation term:
e j ( t ) = 1 if attribute j is selected at time t , 0 otherwise .
This function takes the value of 1 when emotional attribute j ( j { 1 , , n } ) is selected at time t; otherwise, e j ( t ) = 0 .
Typically, SCR peaks 1–3 s after the onset of a sensory stimulus and decays to half the peak value within another 1–3 s [17]. To model this decaying property of SCR, we used an exponential function based on the model proposed by Bach et al. [30]. Let t k represent the time when an attribute is selected. The attribute remains selected until time t k + d k , where d k is the duration for which the attribute remains selected. As shown in Figure 3b, the function takes its maximum value at t k and begins to decay afterward:
s j ( t ) = exp ( λ ( t t k ) ) if t k < t < t k + d k , 0 otherwise .
We then applied a Gaussian filter N ( t ) to s j ( t ) in order to obtain a smooth profile of the SCR signals [30].
N ( t ) = 1 2 π σ exp t 2 2 σ 2
x j ( t ) = s j ( t ) N ( t )
where σ determines the level of the low-pass filter, and x j ( t ) represents a typical profile of SCR signals, which show a rapid increase to the peak followed by a slow decrease.
As shown in Figure 3c,d, the time-series SCR signal was then estimated using a linear combination of x j ( t ) , where the coefficients a j determine the impact of attribute j on the SCR signal.
y ( t ) = a 0 + i = j n a j x j ( t + Δ t )
Here, a 0 represents the intercept, and n is the number of emotional attributes. The time lag Δ t accounts for the delay between the recorded SCR signals and the TDE records. This time lag includes the SCR latency, which typically starts changing 1–3 s after the onset of the stimulus [17], as well as the latency of the participant’s action to press a button. This linear summation of SCR signals caused by different emotional scenes is based on the work of Bach et al. [30,31], where they suggested that SCR signals evoked by temporally successive stimuli can be approximated by the linear summation of SCR signals corresponding to each stimulus, accounting for the temporal interval between them.
The free parameters of the above formulas are a 0 , a j , λ , σ , and Δ t . These parameters were determined by minimizing the sum of squares of the SCR estimation errors using the fmincon function in MATLAB (R2023a, MathWorks Inc., Natick, MA, USA). The search range for Δ t was set between 2 and 5 s.
The above computation was performed for each trial of every participant. Within each trial, participants did not select all emotional attributes. Out of the 20 participants, the number of participants who selected each attribute at least once was as follows: 19 for dominant, 10 for confused, 18 for relieved, 13 for angry, 13 for frustrated, 8 for disappointed, 17 for joyful, 20 for tense, 19 for excited, and 13 for relaxed.
Further, for each attribute, any samples with regression coefficients a j greater or smaller than two times the standard deviation above or below the mean were excluded from the analysis as outliers. As a result, the final sample sizes for the coefficients of each attribute were: 18 for dominant, 9 for confused, 17 for relieved, 12 for angry, 12 for frustrated, 7 for disappointed, 15 for joyful, 19 for tense, 18 for excited, and 12 for relaxed.
We then tested whether the mean value of the coefficients for each attribute was significantly different from 0 using two-tailed t-tests. Since the tests were repeated n = 10 times, the resulting p-values were adjusted using the Benjamini–Hochberg (BH) method [41].
The Benjamini–Hochberg method is used to adjust p-values for controlling the false discovery rate in multiple testing. This method allows for detecting significant results with a higher probability than the Bonferroni or Holm methods while still controlling for false positives. Following this approach, p-values were ranked in ascending order, and the corrected p-values were calculated as follows:
m i × p
where i is the rank of the p-value in ascending order, and m is the total number of tests, which in this case was m = n = 10 . We set the significance level at α = 0.05 , and any corrected p-values smaller than α were considered significant.

4. Results

The mean correlation coefficient between the predicted SCR and the original SCR waveform was 0.57, with a standard error of 0.029. Figure 4 shows three examples for each of the TDE record and observed and estimated SCRs. Figure 4a–c show the case for which the correlation coefficient between the observation and estimation was 0.58, which is close to the mean performance. Figure 4d–f show those for which the coefficient was the smallest: 0.34. Figure 4g–i show those for which the coefficient was the hightest: 0.80. The mean and standard deviation of the free parameters λ , σ , and Δ t were 0.25 ± 0.12 s 1 , 1.0 ± 0.06 s, and 2.17 ± 1.90 s, respectively.
Table 2 presents the mean values and standard deviations of regression coefficients, the values after excluding outliers, the p-values, the adjusted p-values using the BH method, and Cohen’s d. Regarding Cohen’s d, values greater than 0.8 and 0.5 indicate strong and moderate effects, respectively [42]. Figure 5 presents the means and standard errors of the regression coefficients for all emotional attributes, excluding outliers. The mean value and standard error of the regression coefficient for each attribute were as follows: 0.022 ± 0.0076 ( p = 0.032 ) for dominant, 0.013 ± 0.0070 ( p = 0.10 ) for confused, 0.018 ± 0.0067 ( p = 0.054 ) for relieved, 0.047 ± 0.020 ( p = 0.060 ) for angry, 0.027 ± 0.0080 ( p = 0.030 ) for frustrated, 0.012 ± 0.0072 ( p = 0.14 ) for disappointed, 0.0059 ± 0.0025 ( p = 0.069 ) for joyful, 0.048 ± 0.015 ( p = 0.040 ) for tense, 0.022 ± 0.0079 ( p = 0.029 ) for excited, and 0.017 ± 0.0071 ( p = 0.054 ) for relaxed. The attributes that significantly differed between the mean regression coefficient and 0 were dominant, frustrated, tense, and excited.

5. Discussion

This study proposed a method to align temporally evolving emotions with dynamic changes in SCR signals. The SCR signals predicted by the onset of ten different emotions showed a moderate correlation with the observed SCR signals, yielding a mean correlation coefficient of r = 0.57 . As a preliminary effort to synchronize dynamically reported emotions with SCR signals, we view these results as promising.
It was challenging to predict SCR waveforms from the results of the TDE task when the changes in emotions reported in the TDE task were less frequent than changes in SCR activity. In such cases, the SCR was actively changing, even though no change in emotion was reported, and thus the SCR change could not be predicted. The main factor for this phenomenon may be that participants were either unaware of or underestimated the changes in their emotions. Further, SCR responds to light and sound [43], as well as emotional stimuli [11,17], and spontaneous activity occurs approximately three–five times per minute even in a resting state [1,17]. It is difficult to distinguish such emotionless SCR activity from emotionally aroused activity. Therefore, it would be challenging to perfectly predict SCR based on participant-reported emotions alone.
Additionally, in the typical protocol of TDE methods, only the type of dominant emotion is reported, while the intensity of the emotions is not. Hence, the strength of emotions was not considered in our analysis. However, in reality, the strength of emotions depends on the content of game events. Previous research indicates that the intensity of emotional stimuli is related to SCR magnitude and amplitude [13,17,44]. In this study, the analysis was limited in predictive accuracy because constant regression coefficients were computed. To enhance the accuracy of predicting SCR activity from subjective evaluations, we aim to explore a TDE method that simultaneously records both the type and intensity of emotions. A similar approach has been implemented via the temporal dominance of sensations method, from which the TDE method is derived [45]. However, such a modified TDE method, incorporating intensity assessments, may require advanced training for participants, as discussed in [45].
We found that subjectively reported emotions of excited, frustrated, tense, and dominant were significantly connected to dynamic changes in the SCR signals. Frustrated, tense, and excited represent arousing emotions [46,47], as shown in Figure 6, and it is reasonable that these emotions would be dynamically linked to SCR signals. Dominant expresses a sense of control or dominance over a situation and is not typically recognized as an arousing emotion [46,47]. In our experiments, dominant was often selected when participants were in an advantageous position during battle or after defeating enemies. At these moments, participants were in a state of arousal, and dominant may have been associated with SCR signals.
The regression coefficients for both angry and joyful showed notable trends, with unadjusted significance probabilities of 0.036 and 0.035, respectively. In our experimental setup, some participants reported the feeling of anger immediately after their avatars were defeated by enemies. However, during such moments, various other emotions—such as tense, excited, or frustrated—may arise in rapid succession. Since the response time of SCR is relatively slow, it may not capture rapid emotional shifts effectively. As a result, contrary to expectations, angry was not clearly associated with changes in SCR signals.
The non-arousal attribute relaxed also showed a positive trend. Spontaneous SCRs are activated several times per minute, even in the resting state [1,17]. In this study, relaxed was recommended for the TDE task, especially when no emotion was aroused. In other words, the significant trend of the regression coefficient for relaxed was most likely due to spontaneous SCR activities.
The attributes that had no effect on SCR activation were confused, disappointed, and relieved. These emotions are neutral or negatively arousing (e.g., sleepy) [46]; thus, it is reasonable that they did not increase the SCR signals, which are a measure of arousal.
The further development of the method proposed in this study will contribute to the broader application of TDE methods in the field of emotion science. To date, TDE methods have been primarily utilized in food science; however, they are gradually being adopted in other fields [35,37]. We anticipate that TDE methods will become a common approach across various domains for collecting subjective data from assessors. These applications may include emotion evaluation during gameplay and human-to-human communication. Researchers in these fields often prefer to measure physiological data alongside subjective assessments to provide robust evidence and support comprehensive discussions. The method developed in this study is compatible with TDE methods and facilitates the association between subjective reports and physiological data. However, it should be noted that since skin conductance is predominantly associated with arousing emotions [1], this method may be less suitable for evaluating emotions related to relaxation.
Some limitations of the study are raised.
To match the subjective evaluation with the SCR waveform, the time difference Δ t between the two was used as a free parameter. Although we used the same Δ t value within a single trial, the time until the SCR begins to increase and the rate of increase depends on the type of stimulus [48]. To improve the prediction accuracy of the SCR signal, it may be necessary to set parameters for each emotion.
Regarding the characteristics of the offline TDE method [37], in which participants view recorded footage of events and assess their emotions, further investigation is required. This relatively new approach broadens the application of the TDE method to tasks where the online version is not feasible. However, it remains under debate how accurately the offline TDE method captures emotions during these tasks. In our study, some participants reported minimal emotional changes, despite significant fluctuations in their SCR signals. This discrepancy suggests that certain individuals may struggle with accurately recognizing or reporting their own emotions. Therefore, developing effective training methods to better familiarize participants with the TDE methodology is essential.
This study represents an initial, exploratory investigation into the dynamic relationship between subjectively reported emotions and SCR signals. The relatively small sample size of 20 participants limits the generalizability of the findings and precludes drawing definitive conclusions. Logistical constraints, such as the time-intensive nature of SCR measurements and TDE evaluations, posed challenges to including a larger and more diverse participant pool. Despite these limitations, this study provides a foundation for future research. Subsequent studies should aim to replicate and extend these findings with larger and more diverse participant cohorts, leveraging streamlined experimental protocols to enhance scalability and robustness.

6. Conclusions

SCR, with its superior responsiveness, is often used as a measure of arousal-based emotions. However, its use has been limited in instances where multiple types of emotions dynamically change. We investigated a method for the dynamic matching of SCR signals to subjective changes in these emotions over time. For this objective, we modeled the SCR waveforms of the reported temporal changes of ten types of emotion. Through use of the TDE method, emotional changes were associated with dynamic changes in the SCR signal, with a moderate level of correlation. In particular, arousing emotions had a significant effect on changes in SCR. These emotions included excitement, tension, and frustration. Hence, our first attempt could reasonably relate the temporal evolution of multiple types of emotion and SCR. A more advanced TDE method and consideration of SCR spontaneity may make it possible to map emotions to SCR signals with higher precision. The results of this study will clarify the relationship between physiological signals and emotions and promote the study of dynamic changes in emotion under scenarios in which multiple types of emotion can change.

Author Contributions

Conceptualization, Y.K. and S.O.; methodology, Y.K. and S.O.; software, Y.K.; validation, Y.K. and S.O.; formal analysis, Y.K.; investigation, Y.K. and S.O.; resources, Y.K.; data curation, Y.K.; writing—original draft preparation, Y.K. and S.O.; writing—review and editing, Y.K. and S.O.; visualization, Y.K.; supervision, S.O.; project administration, S.O.; funding acquisition, S.O. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by MEXT Kakenhi (24K22310).

Institutional Review Board Statement

This study was approved by the Institutional Review Board, Hino Campus, Tokyo Metropolitan University (H23-11).

Informed Consent Statement

Written informed consent was obtained from all participants involved in this study.

Data Availability Statement

The data can be accessed by contacting the corresponding author, provided the request aligns with an accountable purpose.

Acknowledgments

We acknowledge the use of ChatGPT 4o for assisting with language editing in the preparation of this manuscript. The tool was used to rectify grammatical errors and refine the text, but all content remain the responsibility of the authors.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Kreibig, S.D. Autonomic Nervous System Activity in Emotion: A Review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef] [PubMed]
  2. Dzedzickis, A.; Kaklauskas, A.; Bucinskas, V. Human Emotion Recognition: Review of Sensors and Methods. Sensors 2020, 20, 592. [Google Scholar] [CrossRef] [PubMed]
  3. Engert, V.; Merla, A.; Grant, J.A.; Cardone, D.; Tusche, A.; Singer, T. Exploring the Use of Thermal Infrared Imaging in Human Stress Research. PLoS ONE 2014, 9, e90782. [Google Scholar] [CrossRef] [PubMed]
  4. Mohamed, Y.; Ballardini, G.; Parreira, M.T.; Lemaignan, S.; Leite, I. Automatic Frustration Detection Using Thermal Imaging. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, Sapporo, Japan, 7–10 March 2022; pp. 451–459. [Google Scholar]
  5. Sorostinean, M.; Ferland, F.; Tapus, A. Reliable Stress Measurement Using Face Temperature Variation With a Thermal Camera in Human-robot Interaction. In Proceedings of the International Conference on Humanoid Robots, Seoul, Republic of Korea, 3–5 November 2015; pp. 14–19. [Google Scholar]
  6. Agrafioti, F.; Hatzinakos, D.; Anderson, A.K. ECG Pattern Analysis for Emotion Detection. IEEE Trans. Affect. Comput. 2011, 3, 102–115. [Google Scholar] [CrossRef]
  7. Ikeda, Y.; Horie, R.; Sugaya, M. Estimating Emotion With Biological Information for Robot Interaction. Procedia Comput. Sci. 2017, 112, 1589–1600. [Google Scholar] [CrossRef]
  8. Mokatren, L.S.; Ansari, R.; Cetin, A.E.; Leow, A.D.; Ajilore, O.A.; Klumpp, H.; Vural, F.T.Y. EEG classification by Factoring in Sensor Spatial Configuration. IEEE Access 2021, 9, 19053–19065. [Google Scholar] [CrossRef]
  9. Qing, C.; Qiao, R.; Xu, X.; Cheng, Y. Interpretable Emotion Recognition Using EEG Signals. IEEE Access 2019, 7, 94160–94170. [Google Scholar] [CrossRef]
  10. Boucsein, W. Electrodermal Activity; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
  11. Sjouwerman, R.; Lonsdorf, T.B. Latency of Skin Conductance Responses Across Stimulus Modalities. Psychophysiology 2019, 56, e13307. [Google Scholar] [CrossRef]
  12. Tara, I.; Okamoto, S.; Akiyama, Y.; Ozeki, H. Timing of Vibratory Stimuli to the Upper Body for Enhancing Fear and Excitement of Audio-visual Content. Int. J. Affect. Eng. 2023, 22, 105–113. [Google Scholar] [CrossRef]
  13. Makioka, T.; Okamoto, S. Vibratory Stimuli to the Thoracoabdominal Region Elicit Stronger Fear Responses Than Those to the Fingers. Int. J. Affect. Eng. 2024, 23, 121–124. [Google Scholar] [CrossRef]
  14. Czerwinski, M.; Hernandez, J.; McDuff, D. Building an AI That Feels: AI Systems With Emotional Intelligence Could Learn Faster and Be More Helpful. IEEE Spectrum 2021, 58, 32–38. [Google Scholar] [CrossRef]
  15. Alonso-Martin, F.; Malfaz, M.; Sequeira, J.; Gorostiza, J.F.; Salichs, M.A. A Multimodal Emotion Detection System During Human–Robot Interaction. Sensors 2013, 13, 15549–15581. [Google Scholar] [CrossRef] [PubMed]
  16. Lisetti, C.L.; Nasoz, F. Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals. EURASIP J. Adv. Signal Process. 2004, 2004, 929414. [Google Scholar] [CrossRef]
  17. Dawson, M.E.; Schell, A.M.; Filion, D.L. Handbook of Psychophysiology; The Electrodermal System; Cacioppo, J.T., Tassinary, L.G., Berntson, G.G., Eds.; Cambridge University Press: Cambridge, UK, 2017; pp. 217–243. [Google Scholar] [CrossRef]
  18. Christopoulos, G.I.; Uy, M.A.; Yap, W.J. The Body and the Brain: Measuring Skin Conductance Responses to Understand the Emotional Experience. Organ. Res. Methods 2019, 22, 394–420. [Google Scholar] [CrossRef]
  19. Critchley, H.D.; Elliott, R.; Mathias, C.J.; Dolan, R.J. Neural Activity Relating to Generation and Representation of Galvanic Skin Conductance Responses: A Functional Magnetic Resonance Imaging Study. J. Neurosci. 2000, 20, 3033–3040. [Google Scholar] [CrossRef]
  20. Drummond, P.D. Facial Flushing During Provocation in Women. Psychophysiology 1999, 36, 325–332. [Google Scholar] [CrossRef]
  21. Collet, C.; Vernet-Maury, E.; Delhomme, G.; Dittmar, A. Autonomic Nervous System Response Patterns Specificity to Basic Emotions. J. Auton. Nerv. Syst. 1997, 62, 45–57. [Google Scholar] [CrossRef]
  22. Blechert, J.; Lajtman, M.; Michael, T.; Margraf, J.; Wilhelm, F.H. Identifying Anxiety States Using Broad Sampling and Advanced Processing of Peripheral Physiological Information. Biomed. Sci. Instrum. 2006, 42, 136–141. [Google Scholar]
  23. Lemmens, P.; Crompvoets, F.; Brokken, D.; van den Eerenbeemd, J.; de Vries, G.J. A Body-conforming Tactile Jacket to Enrich Movie Viewing. In Proceedings of the World Haptics 2009-Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Salt Lake City, UT, USA, 18–20 March 2009; pp. 7–12. [Google Scholar] [CrossRef]
  24. Branje, C.; Nespoil, G.; Russo, F.; Fels, D.I. The Effect of Vibrotactile Stimulation on the Emotional Response to Horror Films. Comput. Entertain. 2013, 11, 1–13. [Google Scholar] [CrossRef]
  25. Westerink, J.H.D.M.; van den Broek, E.L.; Schut, M.H.; van Herk, J.; Tuinenbreijer, K. Computing Emotion Awareness Through Galvanic Skin Response and Facial Electromyography. In Probing Experience; Springer: Chan, The Netherlands, 2008; pp. 149–162. [Google Scholar] [CrossRef]
  26. Horvers, A.; Tombeng, N.; Bosse, T.; Lazonder, A.W.; Molenaar, I. Detecting Emotions through Electrodermal Activity in Learning Contexts: A Systematic Review. Sensors 2021, 21, 7869. [Google Scholar] [CrossRef]
  27. Jager, G.; Schlich, P.; Tijssen, I.; Yao, J.; Visalli, M.; de Graaf, C.; Stieger, M. Temporal Dominance of Emotions: Measuring Dynamics of Food-Related Emotions During Consumption. Food Qual. Prefer. 2014, 37, 87–99. [Google Scholar] [CrossRef]
  28. Galmarini, M.; Silva Paz, R.; Enciso Choquehuanca, D.; Zamora, M.C.; Meszd, B. Impact of Music on the Dynamic Perception of Coffee and Evoked Emotions Evaluated by Temporal Dominance of Sensations (TDS) and Emotions (TDE). Food Res. Int. 2021, 150, 110795. [Google Scholar] [CrossRef] [PubMed]
  29. Merlo, T.C.; Soletti, I.; Saldana, E.; Menegali, B.S.; Martins, M.M.; Teixeira, A.C.B.; dos Santos Harada-Padermo, S.; Dargelio, M.D.; Contreras-Castillo, C.J. Measuring Dynamics of Emotions Evoked by the Packaging Colour of Hamburgers Using Temporal Dominance of Emotions (TDE). Food Res. Int. 2019, 124, 147–155. [Google Scholar] [CrossRef] [PubMed]
  30. Bach, D.R.; Flandin, G.; Friston, K.J.; Dolan, R.J. Modelling Event-related Skin Conductance Responses. Int. J. Psychophysiol. 2010, 75, 349–356. [Google Scholar] [CrossRef]
  31. Bach, D.R.; Friston, K.J. Model-based Analysis of Skin Conductance Responses: Towards Causal Models in Psychophysiology. Psychophysiology 2013, 50, 15–22. [Google Scholar] [CrossRef]
  32. Soshi, T.; Nagamine, M.; Fukuda, E.; Takeuchi, A. Modeling Skin Conductance Response Time Series during Consecutive Rapid Decision-Making under Concurrent Temporal Pressure and Information Ambiguity. Brain Sci. 2021, 11, 1122. [Google Scholar] [CrossRef]
  33. Kosuge, Y.; Okamoto, S. Emohance: Real-time Emotional Amplification in Gaming via Physiological Vibratory Feedback. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Sarawak, Malaysia, 6–10 October 2024. [Google Scholar]
  34. Silva, A.P.; Voss, H.; van Zyl, H.; Hogg, T.; de Graaf, C.; Pintado, M.; Jager, G. Temporal Dominance of Sensations, Emotions, and Temporal Liking Measured in a Bar for Two Similar Wines Using a Multi-sip Approach. J. Sens. Stud. 2018, 33, e12459. [Google Scholar] [CrossRef]
  35. Kantono, K.; Hamid, N.; Shepherd, D.; Lin, Y.H.T.; Skiredj, S.; Carr, B.T. Emotional and Electrophysiological Measures Correlate to Flavour Perception in the Presence of Music. Physiol. Behav. 2019, 199, 154–164. [Google Scholar] [CrossRef]
  36. Shimaoka, N.; Okamoto, S.; Akiyama, Y.; Yamada, Y. Linking Temporal Dominance of Sensations for Primary-Sensory and Multi-Sensory Attributes Using Canonical Correlation Analysis. Foods 2022, 11, 781. [Google Scholar] [CrossRef]
  37. Yu, H.; Okamoto, S.; Kosuge, Y. Offline Temporal Dominance of Emotions Method Using Recorded Videos. In Proceedings of the IEEE Global Conference on Consumer Electronics, Las Vegas, NV, USA, 6–8 January 2024; pp. 899–901. [Google Scholar]
  38. Hossain, M.; Kong, Y.; Posada-Quintero, H.F.; Chon, K. Comparison of Electrodermal Activity from Multiple Body Locations Based on Standard EDA Indices & Quality and Robustness Against Motion Artifact. Sensors 2022, 22, 3177. [Google Scholar] [CrossRef]
  39. Van Dooren, M.; de Vries, J.J.; Janssen, J.H. Emotional Sweating Across the Body: Comparing 16 Different Skin Conductance Measurement Locations. Physiol. Behav. 2012, 106, 298–304. [Google Scholar] [CrossRef] [PubMed]
  40. Boucsein, W.; Fowles, D.; Grimnes, S.; Ben-Shakhar, G.; Rroth, W.T.; Dawson, M.E.; Filion, D.L. Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures. Publication Recommendations for Electrodermal Measurements. Psychophysiology 2012, 49, 1017–1034. [Google Scholar] [CrossRef] [PubMed]
  41. Benjamini, Y.; Hochberg, Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. J. R. Stat. Soc. Ser. 1995, 57, 289–300. [Google Scholar] [CrossRef]
  42. Cohen, J. Statistical Power Analysis for the Behavioral Sciences; Routledge: Abingdon, Oxfordshire, UK, 1988. [Google Scholar]
  43. Bari, D.S.; Rammoo, M.N.S.; Aldosky, H.Y.Y.; Jaqsi, M.K.; Martinsen, G. The five basic human senses evoke electrodermal activity. Sensors 2023, 23, 8181. [Google Scholar] [CrossRef]
  44. Bonnet, L.; Comte, A.; Tatu, L.; Millot, J.L.; Moulin, T.; Medeiros de Bustos, E. The role of the amygdala in the perception of positive emotions: An “intensity detector”. Front. Behav. Neurosci. 2015, 9, 00178. [Google Scholar] [CrossRef]
  45. Schlich, P. Temporal Dominance of Sensations (TDS): A new deal for temporal sensory analysis. Curr. Opin. Food Sci. 2017, 15, 38–42. [Google Scholar] [CrossRef]
  46. Russell, J.A. A Circumplex Model of Affect. J. Personal. Soc. Psychol. 1980, 39, 1161–1178. [Google Scholar] [CrossRef]
  47. Russell, J.A. Core Affect and the Psychological Construction of Emotion. Psychol. Rev. 2003, 110, 145–172. [Google Scholar] [CrossRef]
  48. Kosuge, Y.; Makioka, T.; Okamoto, S.; Tara, I. Differences in dynamics of skin conductance responses caused by videos evoking fear, family bonding, and funniness. IEEE Access, 2024; 153596–153604. [Google Scholar] [CrossRef]
Figure 1. Minecraft game screen: (a,b) scenes where the player is attacking an enemy; (c) starting point of the game; (d) scene where the player collects gems from a box.
Figure 1. Minecraft game screen: (a,b) scenes where the player is attacking an enemy; (c) starting point of the game; (d) scene where the player collects gems from a box.
Sci 07 00011 g001
Figure 2. Interface of the TDE method. The attributes were presented in both English and Japanese. The participant continuously selects the emotion that is most dominantly felt at each moment from a list of attributes. The attribute becomes green while being selected. The button positions are shuffled for each trial.
Figure 2. Interface of the TDE method. The attributes were presented in both English and Japanese. The participant continuously selects the emotion that is most dominantly felt at each moment from a list of attributes. The attribute becomes green while being selected. The button positions are shuffled for each trial.
Sci 07 00011 g002
Figure 3. Process of approximating the SCR waveforms based on the results of the TDE method: (a) shows a binary waveform representing the participant’s attribute evaluation; (b) is (a) multiplied by the forgetting curve; (c) is a Gaussian filter convolved with (b), multiplied by an optimized regression coefficient; (d) shows the sum of (c) and the original SCR waveform.
Figure 3. Process of approximating the SCR waveforms based on the results of the TDE method: (a) shows a binary waveform representing the participant’s attribute evaluation; (b) is (a) multiplied by the forgetting curve; (c) is a Gaussian filter convolved with (b), multiplied by an optimized regression coefficient; (d) shows the sum of (c) and the original SCR waveform.
Sci 07 00011 g003
Figure 4. Examples of TDE tasks and predicted SCR waveforms. The top row (ac) shows data from a participant whose regression result was close to the average in terms of the correlation coefficient. The middle row (df) represents data from a participant with the lowest correlation coefficient. The bottom row (gi) shows data from a participant with the highest correlation coefficient. The left column (a,d,g) displays the results of the TDE tasks. The middle column (b,e,h) presents the transformed SCR signals for each attribute. The color of the lines in the left and middle columns represents the type of emotional attribute. The right column (c,f,i) compares the observed and regressed SCR waveforms.
Figure 4. Examples of TDE tasks and predicted SCR waveforms. The top row (ac) shows data from a participant whose regression result was close to the average in terms of the correlation coefficient. The middle row (df) represents data from a participant with the lowest correlation coefficient. The bottom row (gi) shows data from a participant with the highest correlation coefficient. The left column (a,d,g) displays the results of the TDE tasks. The middle column (b,e,h) presents the transformed SCR signals for each attribute. The color of the lines in the left and middle columns represents the type of emotional attribute. The right column (c,f,i) compares the observed and regressed SCR waveforms.
Sci 07 00011 g004
Figure 5. Mean regression coefficients and standard errors for each emotional attribute in the regressed SCR waveforms. * indicates adjusted p < 0.05 . Outliers were excluded.
Figure 5. Mean regression coefficients and standard errors for each emotional attribute in the regressed SCR waveforms. * indicates adjusted p < 0.05 . Outliers were excluded.
Sci 07 00011 g005
Figure 6. Circumplex model of affect used in the subjective evaluation. Adapted from [33,46,47]. Blue dots indicate attributes with significant effects on SCR. Joyful, confused, and relieved were not included in [46,47] but are placed near attributes with similar meanings. Dominant represents the pronounced attribute of the third axis.
Figure 6. Circumplex model of affect used in the subjective evaluation. Adapted from [33,46,47]. Blue dots indicate attributes with significant effects on SCR. Joyful, confused, and relieved were not included in [46,47] but are placed near attributes with similar meanings. Dominant represents the pronounced attribute of the third axis.
Sci 07 00011 g006
Table 1. Ten emotional attributes and their descriptions used in the TDE method. Participants were informed of these attributes prior to the experiment.
Table 1. Ten emotional attributes and their descriptions used in the TDE method. Participants were informed of these attributes prior to the experiment.
AttributeDescription
DominantI feel superior and confident.
ConfusedI am confused and do not know what to do.
RelievedI feel relieved and at peace.
AngryI feel angry or annoyed.
FrustratedI am frustrated and I cannot do what I want.
DisappointedI feel sad or down.
JoyfulI am enjoying myself and having fun.
TenseI feel a sense of danger or urgency.
ExcitedI feel excited, surprised, and ready to fight.
RelaxedI feel relaxed or experienced no significant emotions.
Table 2. Result of regression coefficient for predicting SCR waveforms based on the TDE tasks. Table shows mean and standard error before and after outlier treatment, number of samples after outlier treatment, p-values, corrected p-values, and Cohen’s d values. * denotes the value rejected by BH method with p < 0.05 .
Table 2. Result of regression coefficient for predicting SCR waveforms based on the TDE tasks. Table shows mean and standard error before and after outlier treatment, number of samples after outlier treatment, p-values, corrected p-values, and Cohen’s d values. * denotes the value rejected by BH method with p < 0.05 .
AttributeMean (SE)Mean Value After Outlier Processing (SE)Sample Number Excluding Outliersp-Value Before AdjustmentAdjusted p-Value (BH Method)Cohen’s d
Dominant0.030 (0.011)0.022 (0.0076)180.00960.032 *0.70
Confused0.027 (0.015)0.013 (0.0070)90.0940.100.67
Relieved0.038 (0.021)0.018 (0.0067)170.0880.0540.67
Angry0.065 (0.025)0.047 (0.020)120.0360.0600.71
Frustrated0.059 (0.033)0.027 (0.0080)120.00590.030 *1.02
Disappointed0.028 (0.017)0.012 (0.0072)70.140.140.68
Joyful0.035 (0.020)0.0059 (0.0025)150.0350.0690.66
Tense0.066 (0.022)0.048 (0.015)190.00400.040 *0.75
Excited0.031 (0.011)0.022 (0.0079)180.0120.029 *0.68
Relaxed0.023 (0.0087)0.017 (0.0071)120.0380.0540.80
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kosuge, Y.; Okamoto, S. Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci 2025, 7, 11. https://doi.org/10.3390/sci7010011

AMA Style

Kosuge Y, Okamoto S. Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci. 2025; 7(1):11. https://doi.org/10.3390/sci7010011

Chicago/Turabian Style

Kosuge, Yuki, and Shogo Okamoto. 2025. "Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios" Sci 7, no. 1: 11. https://doi.org/10.3390/sci7010011

APA Style

Kosuge, Y., & Okamoto, S. (2025). Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci, 7(1), 11. https://doi.org/10.3390/sci7010011

Article Metrics

Back to TopTop