Submit to Sensors Review for Sensors Propose a Special Issue

Journal Menu

Journal Browser

► Journal Browser

Multimodal Sensing for Understanding Behavior and Personality

Special Issue Editors
Special Issue Information
Keywords
Benefits of Publishing in a Special Issue
Published Papers

A special issue of Sensors (ISSN 1424-8220). This special issue belongs to the section "Intelligent Sensors".

Deadline for manuscript submissions: closed (31 December 2020) | Viewed by 56505

Share This Special Issue

Special Issue Editors

Prof. Dr. Grzegorz J. Nalepa

E-Mail Website
Guest Editor

Institute of Applied Computer Science, Jagiellonian Univeristy, 31-007 Krakow, Poland
Interests: artificial intelligence; knowledge engineering; affective computing; explainability
Special Issues, Collections and Topics in MDPI journals

Prof. Dr. Marcin Grzegorzek

E-Mail Website
Guest Editor

Institute of Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23538 Lübeck, Germany
Interests: pattern recognition; medical informatics
Special Issues, Collections and Topics in MDPI journals

Prof. Dr. Jose M. Juarez

E-Mail Website
Guest Editor

Computer Science Faculty, Universidad de Murcia, 30100 Murcia, Spain
Interests: medical informatics; artificial intelligence; clinical decision support systems

Prof. Dr. John F. Rauthmann

E-Mail Website
Guest Editor

Department of Psychology, University of Lübeck, Maria-Goeppert Straße 9a, D-23562 Lübeck, Germany
Interests: personality psychology; individual differences; personality computing; sensing of psychological signals

Special Issue Information

Dear Colleagues,

Sensors are everywhere. By the early 2020s, their number will have already exceeded one trillion. This development changes our society, e.g., the rapid innovation in wearable technology led to a societal phenomenon called Quantified Self (QS), a community of people who use the capabilities of technical devices to gain a profound understanding of collected self-related data. This huge amount of personal data generated every day may lead to a significant improvement of the accuracy of artificial intelligence (AI) methods, including machine learning and pattern recognition algorithms.

Understanding human behavior is crucial to personalized systems’ services for a wide variety of scenarios. However, a holistic assessment of human behavior and personality requires a proper combination of different methods. Multimodal sensing and context-driven information fusion is the key to allow systems to provide their services in the most suitable and efficient manner.

With this Special Issue, we would like to attract novel and original scientific contributions describing the newest research achievements in the area of sensor-based behavior and personality understanding using a wide range of AI methods, including pattern recognition and machine learning algorithms. We welcome both technical papers and submissions with a holistic vision, considering a more humanistic point of view. Finally, we encourage emphasis on relevant application domains of multimodal behavior and personality sensing.

Prof. Dr. Grzegorz J. Nalepa
Prof. Dr. Marcin Grzegorzek
Prof. Dr. Jose M. Juarez
Prof. Dr. John F. Rauthmann
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Sensors is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

Multimodal sensing
Social sensing
Context-awareness
Information fusion
Personality
Quantified self
Individual differences
Ambient assisted living
Smart ambient spaces
Personalized healthcare
Physical activity assessment
Sensor-based sleep assessment
Human gait analysis and assessment
Physiotherapy assistance

Benefits of Publishing in a Special Issue

Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue polices can be found here.

Published Papers (11 papers)

Download All Papers

Order results

Result details

Show export options Show export options

Select all

Export citation of selected articles as:

Research

Jump to: Review

14 pages, 2492 KiB

Open AccessArticle

Multimodal Signal Analysis for Pain Recognition in Physiotherapy Using Wavelet Scattering Transform

by Aleksandra Badura, Aleksandra Masłowska, Andrzej Myśliwiec and Ewa Piętka

Sensors 2021, 21(4), 1311; https://doi.org/10.3390/s21041311 - 12 Feb 2021

Cited by 13 | Viewed by 3496

Abstract

Fascial therapy is an effective, yet painful, procedure. Information about pain level is essential for the physiotherapist to adjust the therapy course and avoid potential tissue damage. We have developed a method for automatic pain-related reaction assessment in physiotherapy due to the subjectivity of a self-report. Based on a multimodal data set, we determine the feature vector, including wavelet scattering transforms coefficients. The AdaBoost classification model distinguishes three levels of reaction (no-pain, moderate pain, and severe pain). Because patients vary in pain reactions and pain resistance, our survey assumes a subject-dependent protocol. The results reflect an individual perception of pain in patients. They also show that multiclass evaluation outperforms the binary recognition. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

18 pages, 884 KiB

Open AccessArticle

Detecting Walking Challenges in Gait Patterns Using a Capacitive Sensor Floor and Recurrent Neural Networks

by Raoul Hoffmann, Hanna Brodowski, Axel Steinhage and Marcin Grzegorzek

Sensors 2021, 21(4), 1086; https://doi.org/10.3390/s21041086 - 5 Feb 2021

Cited by 13 | Viewed by 3679 | Correction

Abstract

Gait patterns are a result of the complex kinematics that enable human two-legged locomotion, and they can reveal a lot about a person’s state and health. Analysing them is useful for researchers to get new insights into the course of diseases, and for physicians to track the progress after healing from injuries. When a person walks and is interfered with in any way, the resulting disturbance can show up and be found in the gait patterns. This paper describes an experimental setup for capturing gait patterns with a capacitive sensor floor, which can detect the time and position of foot contacts on the floor. With this setup, a dataset was recorded where 42 participants walked over a sensor floor in different modes, inter alia, normal pace, closed eyes, and dual-task. A recurrent neural network based on Long Short-Term Memory units was trained and evaluated for the classification task of recognising the walking mode solely from the floor sensor data. Furthermore, participants were asked to do the Unilateral Heel-Rise Test, and their gait was recorded before and after doing the test. Another neural network instance was trained to predict the number of repetitions participants were able to do on the test. As the results of the classification tasks turned out to be promising, the combination of this sensor floor and the recurrent neural network architecture seems like a good system for further investigation leading to applications in health and care. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

31 pages, 4339 KiB

Open AccessArticle

Personality-Based Affective Adaptation Methods for Intelligent Systems

by Krzysztof Kutt, Dominika Drążyk, Szymon Bobek and Grzegorz J. Nalepa

Sensors 2021, 21(1), 163; https://doi.org/10.3390/s21010163 - 29 Dec 2020

Cited by 11 | Viewed by 3542

Abstract

In this article, we propose using personality assessment as a way to adapt affective intelligent systems. This psychologically-grounded mechanism will divide users into groups that differ in their reactions to affective stimuli for which the behaviour of the system can be adjusted. In order to verify the hypotheses, we conducted an experiment on 206 people, which consisted of two proof-of-concept demonstrations: a “classical” stimuli presentation part, and affective games that provide a rich and controllable environment for complex emotional stimuli. Several significant links between personality traits and the psychophysiological signals (electrocardiogram (ECG), galvanic skin response (GSR)), which were gathered while using the BITalino (r)evolution kit platform, as well as between personality traits and reactions to complex stimulus environment, are promising results that indicate the potential of the proposed adaptation mechanism. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

19 pages, 7952 KiB

Open AccessFeature PaperArticle

An IoT and Fog Computing-Based Monitoring System for Cardiovascular Patients with Automatic ECG Classification Using Deep Neural Networks

by Jaime A. Rincon, Solanye Guerra-Ojeda, Carlos Carrascosa and Vicente Julian

Sensors 2020, 20(24), 7353; https://doi.org/10.3390/s20247353 - 21 Dec 2020

Cited by 34 | Viewed by 5773

Abstract

Telemedicine and all types of monitoring systems have proven to be a useful and low-cost tool with a high level of applicability in cardiology. The objective of this work is to present an IoT-based monitoring system for cardiovascular patients. The system sends the ECG signal to a Fog layer service by using the LoRa communication protocol. Also, it includes an AI algorithm based on deep learning for the detection of Atrial Fibrillation and other heart rhythms. The automatic detection of arrhythmias can be complementary to the diagnosis made by the physician, achieving a better clinical vision that improves therapeutic decision making. The performance of the proposed system is evaluated on a dataset of 8.528 short single-lead ECG records using two merge MobileNet networks that classify data with an accuracy of 90% for atrial fibrillation. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

35 pages, 6778 KiB

Open AccessArticle

Recognition of Typical Locomotion Activities Based on the Sensor Data of a Smartphone in Pocket or Hand

by Markus Ebner, Toni Fetzer, Markus Bullmann, Frank Deinzer and Marcin Grzegorzek

Sensors 2020, 20(22), 6559; https://doi.org/10.3390/s20226559 - 17 Nov 2020

Cited by 13 | Viewed by 3269

Abstract

With the ubiquity of smartphones, the interest in indoor localization as a research area grew. Methods based on radio data are predominant, but due to the susceptibility of these radio signals to a number of dynamic influences, good localization solutions usually rely on additional sources of information, which provide relative information about the current location. Part of this role is often taken by the field of activity recognition, e.g., by estimating whether a pedestrian is currently taking the stairs. This work presents different approaches for activity recognition, considering the four most basic locomotion activities used when moving around inside buildings: standing, walking, ascending stairs, and descending stairs, as well as an additional messing around class for rejections. As main contribution, we introduce a novel approach based on analytical transformations combined with artificially constructed sensor channels, and compare that to two approaches adapted from existing literature, one based on codebooks, the other using statistical features. Data is acquired using accelerometer and gyroscope only. In addition to the most widely adopted use-case of carrying the smartphone in the trouser pockets, we will equally consider the novel use-case of hand-carried smartphones. This is required as in an indoor localization scenario, the smartphone is often used to display a user interface of some navigation application and thus needs to be carried in hand. For evaluation the well known MobiAct dataset for the pocket-case as well as a novel dataset for the hand-case were used. The approach based on analytical transformations surpassed the other approaches resulting in accuracies of 98.0% for pocket-case and 81.8% for the hand-case trained on the combination of both datasets. With activity recognition in the supporting role of indoor localization, this accuracy is acceptable, but has room for further improvement. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

21 pages, 456 KiB

Open AccessArticle

Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

by Maciej Dzieżyc, Martin Gjoreski, Przemysław Kazienko, Stanisław Saganowski and Matjaž Gams

Sensors 2020, 20(22), 6535; https://doi.org/10.3390/s20226535 - 16 Nov 2020

Cited by 39 | Viewed by 6749

Abstract

To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

20 pages, 4190 KiB

Open AccessArticle

Hybrid System of Emotion Evaluation in Physiotherapeutic Procedures

by Patrycja Romaniszyn-Kania, Anita Pollak, Marta Danch-Wierzchowska, Damian Kania, Andrzej P. Myśliwiec, Ewa Piętka and Andrzej W. Mitas

Sensors 2020, 20(21), 6343; https://doi.org/10.3390/s20216343 - 6 Nov 2020

Cited by 6 | Viewed by 3697

Abstract

Nowadays, the dynamic development of technology allows for the design of systems based on various information sources and their integration into hybrid expert systems. One of the areas of research where such systems are especially helpful is emotion analysis. The sympathetic nervous system controls emotions, while its function is directly reflected by the electrodermal activity (EDA) signal. The presented study aimed to develop a tool and propose a physiological data set to complement the psychological data. The study group consisted of 41 students aged from 19 to 26 years. The presented research protocol was based on the acquisition of the electrodermal activity signal using the Empatica E4 device during three exercises performed in a prototype Disc4Spine system and using the psychological research methods. Different methods (hierarchical and non-hierarchical) of subsequent data clustering and optimisation in the context of emotions experienced were analysed. The best results were obtained for the k-means classifier during Exercise 3 (80.49%) and for the combination of the EDA signal with negative emotions (80.48%). A comparison of accuracy of the k-means classification with the independent division made by a psychologist revealed again the best results for negative emotions (78.05%). Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

15 pages, 1024 KiB

Open AccessArticle

Writhing Movement Detection in Newborns on the Second and Third Day of Life Using Pose-Based Feature Machine Learning Classification

by Iwona Doroniewicz, Daniel J. Ledwoń, Alicja Affanasowicz, Katarzyna Kieszczyńska, Dominika Latos, Małgorzata Matyja, Andrzej W. Mitas and Andrzej Myśliwiec

Sensors 2020, 20(21), 5986; https://doi.org/10.3390/s20215986 - 22 Oct 2020

Cited by 26 | Viewed by 5507

Abstract

Observation of neuromotor development at an early stage of an infant’s life allows for early diagnosis of deficits and the beginning of the therapeutic process. General movement assessment is a method of spontaneous movement observation, which is the foundation for contemporary attempts at objectification and computer-aided diagnosis based on video recordings’ analysis. The present study attempts to automatically detect writhing movements, one of the normal general movement categories presented by newborns in the first weeks of life. A set of 31 recordings of newborns on the second and third day of life was divided by five experts into videos containing writhing movements (with occurrence time) and poor repertoire, characterized by a lower quality of movement in relation to the norm. Novel, objective pose-based features describing the scope, nature, and location of each limb’s movement are proposed. Three machine learning algorithms are evaluated in writhing movements’ detection in leave-one-out cross-validation for different feature extraction time windows and overlapping time. The experimental results make it possible to indicate the optimal parameters for which 80% accuracy was achieved. Based on automatically detected writhing movement percent in the video, infant movements are classified as writhing movements or poor repertoire with an area under the ROC (receiver operating characteristics) curve of 0.83. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

18 pages, 2789 KiB

Open AccessArticle

Marker-Based Movement Analysis of Human Body Parts in Therapeutic Procedure

by Muhammad Hassan Khan, Martin Zöller, Muhammad Shahid Farid and Marcin Grzegorzek

Sensors 2020, 20(11), 3312; https://doi.org/10.3390/s20113312 - 10 Jun 2020

Cited by 22 | Viewed by 6528

Abstract

Movement analysis of human body parts is momentous in several applications including clinical diagnosis and rehabilitation programs. The objective of this research is to present a low-cost 3D visual tracking system to analyze the movement of various body parts during therapeutic procedures. Specifically, a marker based motion tracking system is proposed in this paper to capture the movement information in home-based rehabilitation. Different color markers are attached to the desired joints’ locations and they are detected and tracked in the video to encode their motion information. The availability of this motion information of different body parts during the therapy can be exploited to achieve more accurate results with better clinical insight, which in turn can help improve the therapeutic decision making. The proposed framework is an automated and inexpensive motion tracking system with execution speed close to real time. The performance of the proposed method is evaluated on a dataset of 10 patients using two challenging matrices that measure the average accuracy by estimating the joints’ locations and rotations. The experimental evaluation and its comparison with the existing state-of-the-art techniques reveals the efficiency of the proposed method. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

15 pages, 944 KiB

Open AccessArticle

Using Complexity-Identical Human- and Machine-Directed Utterances to Investigate Addressee Detection for Spoken Dialogue Systems

by Oleg Akhtiamov, Ingo Siegert, Alexey Karpov and Wolfgang Minker

Sensors 2020, 20(9), 2740; https://doi.org/10.3390/s20092740 - 11 May 2020

Cited by 5 | Viewed by 4699

Abstract

Human-machine addressee detection (H-M AD) is a modern paralinguistics and dialogue challenge that arises in multiparty conversations between several people and a spoken dialogue system (SDS) since the users may also talk to each other and even to themselves while interacting with the system. The SDS is supposed to determine whether it is being addressed or not. All existing studies on acoustic H-M AD were conducted on corpora designed in such a way that a human addressee and a machine played different dialogue roles. This peculiarity influences speakers’ behaviour and increases vocal differences between human- and machine-directed utterances. In the present study, we consider the Restaurant Booking Corpus (RBC) that consists of complexity-identical human- and machine-directed phone calls and allows us to eliminate most of the factors influencing speakers’ behaviour implicitly. The only remaining factor is the speakers’ explicit awareness of their interlocutor (technical system or human being). Although complexity-identical H-M AD is essentially more challenging than the classical one, we managed to achieve significant improvements using data augmentation (unweighted average recall (UAR) = 0.628) over native listeners (UAR = 0.596) and a baseline classifier presented by the RBC developers (UAR = 0.539). Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Figure 1

Review

Jump to: Research

32 pages, 983 KiB

Open AccessReview

AI Approaches towards Prechtl’s Assessment of General Movements: A Systematic Literature Review

by Muhammad Tausif Irshad, Muhammad Adeel Nisar, Philip Gouverneur, Marion Rapp and Marcin Grzegorzek

Sensors 2020, 20(18), 5321; https://doi.org/10.3390/s20185321 - 17 Sep 2020

Cited by 42 | Viewed by 7315

Abstract

General movements (GMs) are spontaneous movements of infants up to five months post-term involving the whole body varying in sequence, speed, and amplitude. The assessment of GMs has shown its importance for identifying infants at risk for neuromotor deficits, especially for the detection of cerebral palsy. As the assessment is based on videos of the infant that are rated by trained professionals, the method is time-consuming and expensive. Therefore, approaches based on Artificial Intelligence have gained significantly increased attention in the last years. In this article, we systematically analyze and discuss the main design features of all existing technological approaches seeking to transfer the Prechtl’s assessment of general movements from an individual visual perception to computer-based analysis. After identifying their shared shortcomings, we explain the methodological reasons for their limited practical performance and classification rates. As a conclusion of our literature study, we conceptually propose a methodological solution to the defined problem based on the groundbreaking innovation in the area of Deep Learning. Full article

(This article belongs to the Special Issue Multimodal Sensing for Understanding Behavior and Personality)

► Show Figures

Journal Menu

Journal Browser

Multimodal Sensing for Understanding Behavior and Personality

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (11 papers)

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI