Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios
Abstract
:1. Introduction
2. Methods
2.1. Ethical Statement
2.2. Apparatus
2.3. Stimuli: Emotionally Evocative Gamification in Minecraft
2.4. Temporal Dominance of Emotions (TDE) Method
2.5. Participants
2.6. Procedures
2.7. Data Preprocessing of SCR
3. Dynamic Matching Between TDE and SCR Waveforms
4. Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Kreibig, S.D. Autonomic Nervous System Activity in Emotion: A Review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef] [PubMed]
- Dzedzickis, A.; Kaklauskas, A.; Bucinskas, V. Human Emotion Recognition: Review of Sensors and Methods. Sensors 2020, 20, 592. [Google Scholar] [CrossRef] [PubMed]
- Engert, V.; Merla, A.; Grant, J.A.; Cardone, D.; Tusche, A.; Singer, T. Exploring the Use of Thermal Infrared Imaging in Human Stress Research. PLoS ONE 2014, 9, e90782. [Google Scholar] [CrossRef] [PubMed]
- Mohamed, Y.; Ballardini, G.; Parreira, M.T.; Lemaignan, S.; Leite, I. Automatic Frustration Detection Using Thermal Imaging. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, Sapporo, Japan, 7–10 March 2022; pp. 451–459. [Google Scholar]
- Sorostinean, M.; Ferland, F.; Tapus, A. Reliable Stress Measurement Using Face Temperature Variation With a Thermal Camera in Human-robot Interaction. In Proceedings of the International Conference on Humanoid Robots, Seoul, Republic of Korea, 3–5 November 2015; pp. 14–19. [Google Scholar]
- Agrafioti, F.; Hatzinakos, D.; Anderson, A.K. ECG Pattern Analysis for Emotion Detection. IEEE Trans. Affect. Comput. 2011, 3, 102–115. [Google Scholar] [CrossRef]
- Ikeda, Y.; Horie, R.; Sugaya, M. Estimating Emotion With Biological Information for Robot Interaction. Procedia Comput. Sci. 2017, 112, 1589–1600. [Google Scholar] [CrossRef]
- Mokatren, L.S.; Ansari, R.; Cetin, A.E.; Leow, A.D.; Ajilore, O.A.; Klumpp, H.; Vural, F.T.Y. EEG classification by Factoring in Sensor Spatial Configuration. IEEE Access 2021, 9, 19053–19065. [Google Scholar] [CrossRef]
- Qing, C.; Qiao, R.; Xu, X.; Cheng, Y. Interpretable Emotion Recognition Using EEG Signals. IEEE Access 2019, 7, 94160–94170. [Google Scholar] [CrossRef]
- Boucsein, W. Electrodermal Activity; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
- Sjouwerman, R.; Lonsdorf, T.B. Latency of Skin Conductance Responses Across Stimulus Modalities. Psychophysiology 2019, 56, e13307. [Google Scholar] [CrossRef]
- Tara, I.; Okamoto, S.; Akiyama, Y.; Ozeki, H. Timing of Vibratory Stimuli to the Upper Body for Enhancing Fear and Excitement of Audio-visual Content. Int. J. Affect. Eng. 2023, 22, 105–113. [Google Scholar] [CrossRef]
- Makioka, T.; Okamoto, S. Vibratory Stimuli to the Thoracoabdominal Region Elicit Stronger Fear Responses Than Those to the Fingers. Int. J. Affect. Eng. 2024, 23, 121–124. [Google Scholar] [CrossRef]
- Czerwinski, M.; Hernandez, J.; McDuff, D. Building an AI That Feels: AI Systems With Emotional Intelligence Could Learn Faster and Be More Helpful. IEEE Spectrum 2021, 58, 32–38. [Google Scholar] [CrossRef]
- Alonso-Martin, F.; Malfaz, M.; Sequeira, J.; Gorostiza, J.F.; Salichs, M.A. A Multimodal Emotion Detection System During Human–Robot Interaction. Sensors 2013, 13, 15549–15581. [Google Scholar] [CrossRef] [PubMed]
- Lisetti, C.L.; Nasoz, F. Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals. EURASIP J. Adv. Signal Process. 2004, 2004, 929414. [Google Scholar] [CrossRef]
- Dawson, M.E.; Schell, A.M.; Filion, D.L. Handbook of Psychophysiology; The Electrodermal System; Cacioppo, J.T., Tassinary, L.G., Berntson, G.G., Eds.; Cambridge University Press: Cambridge, UK, 2017; pp. 217–243. [Google Scholar] [CrossRef]
- Christopoulos, G.I.; Uy, M.A.; Yap, W.J. The Body and the Brain: Measuring Skin Conductance Responses to Understand the Emotional Experience. Organ. Res. Methods 2019, 22, 394–420. [Google Scholar] [CrossRef]
- Critchley, H.D.; Elliott, R.; Mathias, C.J.; Dolan, R.J. Neural Activity Relating to Generation and Representation of Galvanic Skin Conductance Responses: A Functional Magnetic Resonance Imaging Study. J. Neurosci. 2000, 20, 3033–3040. [Google Scholar] [CrossRef]
- Drummond, P.D. Facial Flushing During Provocation in Women. Psychophysiology 1999, 36, 325–332. [Google Scholar] [CrossRef]
- Collet, C.; Vernet-Maury, E.; Delhomme, G.; Dittmar, A. Autonomic Nervous System Response Patterns Specificity to Basic Emotions. J. Auton. Nerv. Syst. 1997, 62, 45–57. [Google Scholar] [CrossRef]
- Blechert, J.; Lajtman, M.; Michael, T.; Margraf, J.; Wilhelm, F.H. Identifying Anxiety States Using Broad Sampling and Advanced Processing of Peripheral Physiological Information. Biomed. Sci. Instrum. 2006, 42, 136–141. [Google Scholar]
- Lemmens, P.; Crompvoets, F.; Brokken, D.; van den Eerenbeemd, J.; de Vries, G.J. A Body-conforming Tactile Jacket to Enrich Movie Viewing. In Proceedings of the World Haptics 2009-Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Salt Lake City, UT, USA, 18–20 March 2009; pp. 7–12. [Google Scholar] [CrossRef]
- Branje, C.; Nespoil, G.; Russo, F.; Fels, D.I. The Effect of Vibrotactile Stimulation on the Emotional Response to Horror Films. Comput. Entertain. 2013, 11, 1–13. [Google Scholar] [CrossRef]
- Westerink, J.H.D.M.; van den Broek, E.L.; Schut, M.H.; van Herk, J.; Tuinenbreijer, K. Computing Emotion Awareness Through Galvanic Skin Response and Facial Electromyography. In Probing Experience; Springer: Chan, The Netherlands, 2008; pp. 149–162. [Google Scholar] [CrossRef]
- Horvers, A.; Tombeng, N.; Bosse, T.; Lazonder, A.W.; Molenaar, I. Detecting Emotions through Electrodermal Activity in Learning Contexts: A Systematic Review. Sensors 2021, 21, 7869. [Google Scholar] [CrossRef]
- Jager, G.; Schlich, P.; Tijssen, I.; Yao, J.; Visalli, M.; de Graaf, C.; Stieger, M. Temporal Dominance of Emotions: Measuring Dynamics of Food-Related Emotions During Consumption. Food Qual. Prefer. 2014, 37, 87–99. [Google Scholar] [CrossRef]
- Galmarini, M.; Silva Paz, R.; Enciso Choquehuanca, D.; Zamora, M.C.; Meszd, B. Impact of Music on the Dynamic Perception of Coffee and Evoked Emotions Evaluated by Temporal Dominance of Sensations (TDS) and Emotions (TDE). Food Res. Int. 2021, 150, 110795. [Google Scholar] [CrossRef] [PubMed]
- Merlo, T.C.; Soletti, I.; Saldana, E.; Menegali, B.S.; Martins, M.M.; Teixeira, A.C.B.; dos Santos Harada-Padermo, S.; Dargelio, M.D.; Contreras-Castillo, C.J. Measuring Dynamics of Emotions Evoked by the Packaging Colour of Hamburgers Using Temporal Dominance of Emotions (TDE). Food Res. Int. 2019, 124, 147–155. [Google Scholar] [CrossRef] [PubMed]
- Bach, D.R.; Flandin, G.; Friston, K.J.; Dolan, R.J. Modelling Event-related Skin Conductance Responses. Int. J. Psychophysiol. 2010, 75, 349–356. [Google Scholar] [CrossRef]
- Bach, D.R.; Friston, K.J. Model-based Analysis of Skin Conductance Responses: Towards Causal Models in Psychophysiology. Psychophysiology 2013, 50, 15–22. [Google Scholar] [CrossRef]
- Soshi, T.; Nagamine, M.; Fukuda, E.; Takeuchi, A. Modeling Skin Conductance Response Time Series during Consecutive Rapid Decision-Making under Concurrent Temporal Pressure and Information Ambiguity. Brain Sci. 2021, 11, 1122. [Google Scholar] [CrossRef]
- Kosuge, Y.; Okamoto, S. Emohance: Real-time Emotional Amplification in Gaming via Physiological Vibratory Feedback. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Sarawak, Malaysia, 6–10 October 2024. [Google Scholar]
- Silva, A.P.; Voss, H.; van Zyl, H.; Hogg, T.; de Graaf, C.; Pintado, M.; Jager, G. Temporal Dominance of Sensations, Emotions, and Temporal Liking Measured in a Bar for Two Similar Wines Using a Multi-sip Approach. J. Sens. Stud. 2018, 33, e12459. [Google Scholar] [CrossRef]
- Kantono, K.; Hamid, N.; Shepherd, D.; Lin, Y.H.T.; Skiredj, S.; Carr, B.T. Emotional and Electrophysiological Measures Correlate to Flavour Perception in the Presence of Music. Physiol. Behav. 2019, 199, 154–164. [Google Scholar] [CrossRef]
- Shimaoka, N.; Okamoto, S.; Akiyama, Y.; Yamada, Y. Linking Temporal Dominance of Sensations for Primary-Sensory and Multi-Sensory Attributes Using Canonical Correlation Analysis. Foods 2022, 11, 781. [Google Scholar] [CrossRef]
- Yu, H.; Okamoto, S.; Kosuge, Y. Offline Temporal Dominance of Emotions Method Using Recorded Videos. In Proceedings of the IEEE Global Conference on Consumer Electronics, Las Vegas, NV, USA, 6–8 January 2024; pp. 899–901. [Google Scholar]
- Hossain, M.; Kong, Y.; Posada-Quintero, H.F.; Chon, K. Comparison of Electrodermal Activity from Multiple Body Locations Based on Standard EDA Indices & Quality and Robustness Against Motion Artifact. Sensors 2022, 22, 3177. [Google Scholar] [CrossRef]
- Van Dooren, M.; de Vries, J.J.; Janssen, J.H. Emotional Sweating Across the Body: Comparing 16 Different Skin Conductance Measurement Locations. Physiol. Behav. 2012, 106, 298–304. [Google Scholar] [CrossRef] [PubMed]
- Boucsein, W.; Fowles, D.; Grimnes, S.; Ben-Shakhar, G.; Rroth, W.T.; Dawson, M.E.; Filion, D.L. Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures. Publication Recommendations for Electrodermal Measurements. Psychophysiology 2012, 49, 1017–1034. [Google Scholar] [CrossRef] [PubMed]
- Benjamini, Y.; Hochberg, Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. J. R. Stat. Soc. Ser. 1995, 57, 289–300. [Google Scholar] [CrossRef]
- Cohen, J. Statistical Power Analysis for the Behavioral Sciences; Routledge: Abingdon, Oxfordshire, UK, 1988. [Google Scholar]
- Bari, D.S.; Rammoo, M.N.S.; Aldosky, H.Y.Y.; Jaqsi, M.K.; Martinsen, G. The five basic human senses evoke electrodermal activity. Sensors 2023, 23, 8181. [Google Scholar] [CrossRef]
- Bonnet, L.; Comte, A.; Tatu, L.; Millot, J.L.; Moulin, T.; Medeiros de Bustos, E. The role of the amygdala in the perception of positive emotions: An “intensity detector”. Front. Behav. Neurosci. 2015, 9, 00178. [Google Scholar] [CrossRef]
- Schlich, P. Temporal Dominance of Sensations (TDS): A new deal for temporal sensory analysis. Curr. Opin. Food Sci. 2017, 15, 38–42. [Google Scholar] [CrossRef]
- Russell, J.A. A Circumplex Model of Affect. J. Personal. Soc. Psychol. 1980, 39, 1161–1178. [Google Scholar] [CrossRef]
- Russell, J.A. Core Affect and the Psychological Construction of Emotion. Psychol. Rev. 2003, 110, 145–172. [Google Scholar] [CrossRef]
- Kosuge, Y.; Makioka, T.; Okamoto, S.; Tara, I. Differences in dynamics of skin conductance responses caused by videos evoking fear, family bonding, and funniness. IEEE Access, 2024; 153596–153604. [Google Scholar] [CrossRef]
Attribute | Description |
---|---|
Dominant | I feel superior and confident. |
Confused | I am confused and do not know what to do. |
Relieved | I feel relieved and at peace. |
Angry | I feel angry or annoyed. |
Frustrated | I am frustrated and I cannot do what I want. |
Disappointed | I feel sad or down. |
Joyful | I am enjoying myself and having fun. |
Tense | I feel a sense of danger or urgency. |
Excited | I feel excited, surprised, and ready to fight. |
Relaxed | I feel relaxed or experienced no significant emotions. |
Attribute | Mean (SE) | Mean Value After Outlier Processing (SE) | Sample Number Excluding Outliers | p-Value Before Adjustment | Adjusted p-Value (BH Method) | Cohen’s d |
---|---|---|---|---|---|---|
Dominant | 0.030 (0.011) | 0.022 (0.0076) | 18 | 0.0096 | 0.032 * | 0.70 |
Confused | 0.027 (0.015) | 0.013 (0.0070) | 9 | 0.094 | 0.10 | 0.67 |
Relieved | 0.038 (0.021) | 0.018 (0.0067) | 17 | 0.088 | 0.054 | 0.67 |
Angry | 0.065 (0.025) | 0.047 (0.020) | 12 | 0.036 | 0.060 | 0.71 |
Frustrated | 0.059 (0.033) | 0.027 (0.0080) | 12 | 0.0059 | 0.030 * | 1.02 |
Disappointed | 0.028 (0.017) | 0.012 (0.0072) | 7 | 0.14 | 0.14 | 0.68 |
Joyful | 0.035 (0.020) | 0.0059 (0.0025) | 15 | 0.035 | 0.069 | 0.66 |
Tense | 0.066 (0.022) | 0.048 (0.015) | 19 | 0.0040 | 0.040 * | 0.75 |
Excited | 0.031 (0.011) | 0.022 (0.0079) | 18 | 0.012 | 0.029 * | 0.68 |
Relaxed | 0.023 (0.0087) | 0.017 (0.0071) | 12 | 0.038 | 0.054 | 0.80 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kosuge, Y.; Okamoto, S. Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci 2025, 7, 11. https://doi.org/10.3390/sci7010011
Kosuge Y, Okamoto S. Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci. 2025; 7(1):11. https://doi.org/10.3390/sci7010011
Chicago/Turabian StyleKosuge, Yuki, and Shogo Okamoto. 2025. "Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios" Sci 7, no. 1: 11. https://doi.org/10.3390/sci7010011
APA StyleKosuge, Y., & Okamoto, S. (2025). Dynamic Matching of Emotions and Skin Conductance Responses in Interactive and Prolonged Emotional Scenarios. Sci, 7(1), 11. https://doi.org/10.3390/sci7010011