An Overview of Approaches and Methods for the Cognitive Workload Estimation in Human–Machine Interaction Scenarios through Wearables Sensors

Abstract

Background: Human-Machine Interaction (HMI) has been an important field of research in recent years, since machines will continue to be embedded in many human actvities in several contexts, such as industry and healthcare. Monitoring in an ecological mannerthe cognitive workload (CW) of users, who interact with machines, is crucial to assess their level of engagement in activities and the required effort, with the goal of preventing stressful circumstances. This study provides a comprehensive analysis of the assessment of CW using wearable sensors in HMI. Methods: this narrative review explores several techniques and procedures for collecting physiological data through wearable sensors with the possibility to integrate these multiple physiological signals, providing a multimodal monitoring of the individuals’CW. Finally, it focuses on the impact of artificial intelligence methods in the physiological signals data analysis to provide models of the CW to be exploited in HMI. Results: the review provided a comprehensive evaluation of the wearables, physiological signals, and methods of data analysis for CW evaluation in HMI. Conclusion: the literature highlighted the feasibility of employing wearable sensors to collect physiological signals for an ecological CW monitoring in HMI scenarios. However, challenges remain in standardizing these measures across different populations and contexts.

1. Introduction

Human-machine interaction (HMI) refers to the dynamic and multidimensional interface between humans and machines, encompassing technical and cognitive dimensions [1,2]. It involves real-time interaction between humans and machines through a human-machine interface [3]. HMI serves as an essential information interaction medium between humans and intelligent devices, with broad application prospects in various fields such as medical care, education, and military applications [4]. Notably, the significance of HMI extends to its role also in labor productivity and efficiency, particularly in the context of Industry 4.0, where it has deeply transformed the manufacturing landscape [5,6]. Additionally, the analysis of mission reliability in man-machine systems emphasizes the importance of considering human cognitive function and the interaction between humans and machines [7,8]. In this perspective, the measurement of cognitive workload (CW) during HMI has attracted great attention for the purposes of design, training, and evaluation [9]. In this field of application, non-invasive contactless or wearable sensors are preferred. The assessment of CW in HMI through wearable sensors is a crucial area of research with implications for various fields such as healthcare, sports, industry, and military settings. Wearable sensors offer the potential to monitor CW in real-time, providing valuable insights into human performance and well-being [10,11]. CW monitoring is usually performed by measuring the brain activity during the execution of tasks. To this aim, portable neuroimaging technologies, such as electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS), are highly suitable for assessing continuously the individuals’ CW [12,13,14,15]. However, it should be highlighted that wearing helmets with located electrodes or optodes could be not comfortable in several situations and environments. For this reason, wearable sensors, such as digital t-shirts, smartwatches, and smart jewels, have emerged as innovative tools for assessing CW, offering real-time monitoring and potential applications in various domains. The potential of wearable devices, including smart glasses and smartwatches, in the realm of HMI and CW evaluation has been emphasized by Lim et al. (2015) [16]. Skin-mountable and wearable sensors, capable of being affixed to garments or directly adhered to the skin, could provide the continuous monitoring of human activities, including the assessment of CW in real-time [17]. According to Khundaqji et al. (2020) [18], smart t-shirts have been recognized as a credible and consistent tool for tracking certain physiological measurements, hence showcasing their capability in evaluating mental strain [16]. Moreover, wearable flexible electronics, including cardiac electrodes and pressure sensors, have shown promise in detecting mental stress and monitoring physiological signals, offering potential applications in CW assessment [19]. In the same context, wearable tactile sensors have demonstrated their potential and are extensively used for monitoring human health and movementsin HMI applications [20].

The wearable sensors potential to monitor the CW during HMI has been enhanced by introducing machine learning (ML) techniques applied to physiological signals. ML is a subfield within the discipline of artificial intelligence (AI), whereby systems are endowed with the ability to acquire knowledge and enhance their performance via experience, without the need for explicit programming [21]. The task entails the creation of algorithms capable of analyzing and interpreting intricate data, adjusting to novel situations, and generating intelligent choices or predictions based on incoming data [22]. For instance, ML applied to physiological signals and imaging data, could provide a strong contribution for diagnostic purposes [23,24]. ML can be of relevant importance in the assessment of cognitive effort, when applied to physiological data for HMI. In fact, physiological signals, including heart rate (HR), brain activity, skin conductance, and eye movements, but also facial expressions, provide contemporaneous information on an individual’s cognitive and emotional condition [25,26,27]. By using ML algorithms to analyze these signals, it becomes feasible to understand and measure cognitive burden, which refers to the level of mental exertion shown by the brain.

This overview aims to investigate the role of wearable sensors in the CW evaluation during HMI. Particularly, the main contributions of the survey are:

• to comprehensively investigate the potential of smart sensor technology in the assessment of CW. This encompasses exploring the effectiveness, accuracy, and practicality of various smart sensors in detecting and quantifying the cognitive load under different conditions and in various groups of populations with different characteristics.
• to identify the principal physiological signals employed for CW monitoring in HMI applications, focusing on approaches based on both unimodal and multimodal physiological signal acquisitions.
• to describe the ML-based approaches used to evaluate the CW from physiological signals, acquired through wearable sensors.

For this overview, PubMed/MEDLINE, Science Direct, IEEE explore, and Scopus databases were searched up to 5 years before the inception.

The search concentrated on the words “cognitive workload”, “human machine interaction”, and “wearable” within the following fields: article title, abstract, and keywords. Notably, only papers published in the last five years were considered. Following the removal of duplicates, two reviewers (DP and SI) independently assessed the titles and abstracts of all potentially relevant publications for inclusion. If there was a difference of opinion, the decision of a third reviewer was used to reach an agreement. Subsequently, the two reviewers obtained complete versions of the papers and independently examined them to determine whether they met the criteria for inclusion. If a consensus could not be achieved via dialogue, a third reviewer’s assessment was used to resolve conflicts (DC). Specifically, reviews and articles authored in a language other than English were not included.

Hence, the paper is structured into different sections about the presented topics. Section 2 provides a comprehensive review on the potentiality of wearable sensors to acquire physiological signals for CW monitoring, briefing describing the various signals that could be recorded. Section 3 describes examples of applications of wearables for CW evaluation in HMI relying on a single physiological signal (i.e., unimodal approach), and examples concerning multimodal approaches. Section 4 describes the employment of ML algorithms for physiological signals data analysis in the context of wearable sensors applied to HMI for CW monitoring, and, finally, a discussion regarding the different application fields, future perspectives, limitations and conclusions are reported in Section 5, Section 6 and Section 7.

The overall topic of the survey is illustrated in Figure 1.

2. Physiological Signals Acquired through Wearable Sensors for HMI

In the field of healthcare, the employment of wearable sensors to collect physiological signals has gained notable attention. The continuous monitoring of physiological parameters that has revolutionized the personalized medicine and the detection of subtle changes in the body are enabled by using these novel sensors. Their development, including movement, physiological, and biochemical sensors, have shown promise in improving disease detection and prognostication through the analysis of changes in physiology over time [28]. Furthermore, the development of flexible, stretchable, and miniaturized wearable sensors has shown great potential for personalized healthcare applications, including the monitoring of motion, physiological, electrophysiological, and electrochemical signals [29]. These advancements in wearable sensor technology have paved the way for applications in healthcare monitoring, disease diagnostics, and HMI [30,31,32,33]. Overall, wearable sensors have the potential to play a transformative role in healthcare and ergonomics, offering continuous, non-invasive monitoring of physiological signals of individuals [31,34]. Importantly, it should be noted that CW monitoring through behavioral and gesture analysis has been proposed so far. For instance, Madeo et al. examine the use of gesture analysis in the creation of novel approaches to HMI, specifically in the evaluation of CW in interactive systems [35].

The physiological signals that can be acquired through such a device are mainly the heart rate (HR) through photoplethysmography (PPG), the electrodermal activity (EDA), and breathing rate.

PPG is an optical method that accurately assesses changes in blood volume inside the small blood vessels of tissue without the need for intrusive procedures [36,37]. It has garnered interest due to its capacity to evaluate heart rate variability (HRV), a crucial measure of autonomic nervous system function and psychophysiological well-being [38,39,40]. In detail, HRV is a measure of the fluctuation in time intervals between successive heartbeats and it indicates the dynamic interaction between the sympathetic and parasympathetic divisions of the autonomic nervous system. The significance of HRV is in its function as a biomarker for cardiovascular health, stress tolerance, and general state of well-being [41,42,43,44,45,46,47,48,49].

The continuous HR monitoring is particularly assured by devices whose characteristic is to extend battery life, providing a practical and sustainable solution for long-term monitoring. In this perspective, the potential of smartphones and smartwatches in these applications is further underscored, the integration of smartphones and smartwatches has revolutionized the landscape of remote psychophysiological assessment. Particularly, this technology has been utilized for health monitoring assessment [50,51,52], daily living and sports activities detection, highlighting their versatility in monitoring and promoting health and well-being [53,54,55,56].

Electrodermal activity (EDA), often referred to as Galvanic Skin Response (GSR), is a physiological signal that indicates changes in the electrical characteristics of the skin caused by sweat gland function [57,58]. EDA is regarded as a delicate indicator of sympathetic arousal and has been extensively used to evaluate emotional and cognitive functions, as well as the functioning of the autonomic nervous system [59,60].

The breathing rate, which refers to the frequency of breaths taken within a minute, is a vital factor in psychophysiological evaluation as it mirrors the physiological and emotional condition of the body [61,62]. The significance of respiratory rate resides in its intimate correlation with the autonomic nervous system and the control of emotions. Fluctuations in respiration rate serve as an indication of tension, anxiety, and relaxation, making it a helpful indicator for psychophysiological evaluation [63,64,65,66]. Furthermore, the pace at which one breathes is strongly connected to HRV, offering valuable information about the body’s reaction to stress and emotional stimulation [43,62].

Ref. [35] In addition, eye tracking technology has become an indispensable tool in the assessment of CW across diverse domains, offering insights through physiological indicators like pupil dilation, eye movement patterns, and fixation durations. This method’s efficacy in evaluating CW is supported by several research in various contexts, such as goal-directed tasks, robotic surgical training, clinical simulations, and space teleoperation, underlining its broad applicability. For instance, Bailey & Iqbal showcased the potential of eye tracking in monitoring CW fluctuations during interactive tasks through pupil dilation [67]. Similarly, Zhang & Cui validated the reliability of the Tobii Pro Nano eye tracker in detecting CW changes in real-time during N-back tasks [68]. Wu et al., further, demonstrated the predictive capability of eye-tracking metrics in estimating CW levels with an accuracy of 84.7% during robotic surgical training [69]. The integration of eye tracking with other physiological measurements, such as EEG, has been shown to refine CW assessments [70].

Moreover, in this perspective, the development of smart jewelry has also increased the impact of physiological monitoring in ecological conditions. Smart jewelry is a growing industry that combines fashion and technology and falls under the category of wearable gadgets. The collection includes a wide variety of jewelry pieces, such as bracelets, rings, necklaces, and earrings, that are equipped with modern technical elements for different purposes. Smart jewelry is specifically designed to effortlessly blend into daily routines, providing features like instantaneous health monitoring, communication, and personal security [71,72,73]. These gadgets are equipped with sensors and communication modules that provide the monitoring of biometric signals, tracking of activities, and provision of emergency alarms. Smart jewelry has attracted attention for its potential in eHealth-related services because of its ability to discreetly and unobtrusively monitor vital signs and physical activity. In addition, smart jewelry is positioned to transform the conventional jewelry sector by integrating cutting-edge technology such as Internet of Things (IoT) connection, blockchain-based authentication, and even emotional and contextual feedback via embedded LED displays. The convergence of aesthetics and usefulness in smart jewelry offers novel prospects for customized healthcare, fashion, and self-expression, rendering it a propitious domain for more investigation and advancement.

Figure 2 describes the principal physiological signals used for CW monitoring.

In the following sections it is described how these signals acquired through wearable sensors have been used to monitor the human CW.

3. Cognitive Workload Monitoring through Unimodal and Multimodal Approach

CW can be roughly categorized into two types according to the use of the emotion recognition modality i.e., unimodal and multimodal. Unimodal methods generally employ single channels, such as face images, speech or physiological signals and text to classify different emotional states. Multimodal approach uses two or more emotion channels to analyze emotion or CW, comprehensively.

The feature extraction process is a crucial part of both the unimodal and multimodal approach because distinct features can facilitate precise results. The feature learning method for unimodal approach can be used for multimodal one. Features are separately extracted, according to the characteristics of single modalities [25].

The main papers reported in this section are summarized in

Table 1. Main papers included into this section of the review after the literature review procedure.

Authors	Objective	Signals Acquired
Rupprecht et al. [74]	Combination of a dynamic projection system with visual detection of human location and gestures in a spacious work area.	Gesture recognition through high-resolution RGB-camera and real-time object recognition algorithm YOLOv3.
Ciccarelli et al. [75]	Development of a system to avoid uncomfortable and unsafe postures	Posture recognition through Intel RealSense technologies and the Cubemos Skeleton Tracking SDK.
Beggiato et al. [76]	Investigation of factors that influence comfort and discomfort in automated driving.	HR, EDA, pupil diameter, and eye blink rate
Biswas and Prabhakar [77]	Investigation of the possibility of identifying drivers’ cognitive load and immediate awareness of emerging road risks via the use of Saccadic Intrusion.	Eye gaze and saccadic intrusion through Tobii TX-2
Li et al. [78]	Evaluation of human-computer interactions on a augmented visualization Primary Flight Display (PFD) compared with the traditional PFD.	Pupil Labs eye tracker to assess pupil position and dimensions.
Liang et al. [79]	An assessment of how the input method and display mode of the situation map impact the performance of Early Warning Aircraft (EWA) during reconnaissance tasks, considering varying levels of information complexity.	Tobii Glasses 2 for eye movement recording and HRV through a PPG device.
Khamaisi et al. [80]	Assessment of the users experience of workers in manufacturing sites.	• HTC Vive Tracker suite, to evaluate the human body angles; • Empatica E4 wristband, to record HR and EDA; • Zephyr Bioharness 3 thoracic band, to collect RR intervals; • HTC Vive Pro Eye headset equipped with Tobii eye tracking system.
Peruzzini et al. [81]	Development of a mixed reality setup where operators are digitized and monitored to evaluate both physical and cognitive ergonomics.	• Siemens JACK to collect the users’ postures and movements in the real space; • Tobii Glasses 2 to collect the users’ eye fixations; • Zephyr Bioharness to collect HR, HRV, breathing rate (BR), acceleration (VMU), and posture.
Peruzzini et al. [82]	A procedure to employ digital technologies to enhance product-process design to analyse the workers behaviours	The setup consists of a video camera used to replicate the virtual environment and generate a digital replica of the workplace. Additionally, an advanced eye tracking system (specifically, the Glasses 2 by Tobii) is employed to analyze the precise eye fixation of real users. Furthermore, a multi-parametric wearable sensor is utilized to monitor real-time physiological parameters.

. Notably, the 58.3% of the papers considered employed a multimodal approach for CW monitoring, whereas the 41.7% of the studies used an unimodal framework.

3.1. Unimodal Physiological Monitoring of Cognitive Workload

The studies reporting a unimodal approach to monitor the CW in HMI are mainly related to study examples of gesture recognition, eye movements, facial expressions and heart rate monitoring.

Concerning the investigation of the gestures and pose of the individuals, Rupprecht et al. [74] assessed the effectiveness and user-friendliness of incorporating a dynamic projection system with visual human location and gesture detection, in industrial site assembly chores. The Task Completion Time (TCT) according to the General Assembly Task Model (GATM) was used to evaluate process efficiency, and the results showed the potential for efficiency improvement. The guided in-situ instructions, which displayed the mounting position and orientation directly at the correct place, were found to be particularly useful in terms of process and usability. Participants provided feedback on the experiments, suggesting improvements such as smoother and more accurate gesture control, simplified information using pictograms, and a mixture of both static in-view and guided in-situ instructions.

Similarly, Ciccarelli et al. [75] presented a system aimed at boosting physical ergonomics in the context of human-robot cooperation. The primary objective was to mitigate the likelihood of musculoskeletal illnesses and promote the overall well-being of workers. The system employed workers’ anthropometric parameters, monitored their posture using inertial and visual systems, considered job requirements, and did real-time risk assessment to optimize robot behavior. The first laboratory tests demonstrated the dependability and precision of the posture monitoring system using the Cubemos Skeleton Tracking SDK and XSens(r) technology. However, limitations were identified, such as occlusion and failure to detect all body points, which could be addressed through AI algorithms and the use of multiple depth cameras. The system also incorporated gesture recognition for task activation and control, with gestures like “OK” and “STOP” implemented. Further enhancements were suggested to improve recognition accuracy and accommodate different tasks. A virtual simulation of a real case study in a quality control workstation demonstrated the system’s effectiveness in improving worker posture and reducing ergonomic risks.

Concerning the eyes movement analysis, Biswas and Prabhakar [77] examined the velocities of saccadic intrusions. Outliers were removed by using the outer fence estimated from the median and inter-quartile range. The average velocities of saccadic intrusion exhibited variability across subjects, with 10 out of 12 people demonstrating velocities above 3°/s. A unidirectional analysis of variance (ANOVA) revealed a statistically significant impact of conditions on the z-scores of saccadic intrusions, with a difference between the control and N-back conditions.

Li et al. [78] developed an augmented visualization Primary Flight Display (PFD) that considerably enhanced pilots’ situation awareness and decreased perceived workload compared to the traditional PFD. Pilots, using the augmented PFD, could better identify the status of flight modes, leading to shorter response times in cognitive information processing. Fixation durations on augmented PFDs were significantly shorter than traditional PFDs, indicating improved attention distribution and situation awareness. The subjective workload ratings using NASA-TLX showed that the augmented PFD design reduced mental demand and frustration, compared to the traditional PFD. The proposed augmented visualization design facilitated pilots’ attention distribution by decreasing fixation duration and increasing the frequency of fixations.

The study by Viegas et al. [83] introduced a technique capable of identifying mental stress by analyzing facial expressions captured only via video footage. The primary benefit of using video for personal stress detection was in the convenient availability of webcams during computer use. A total of 17 distinct Action Units (AUswere recovered, ranging from higher-level to lower-level facial expressions, on a frame-by-frame basis. The authors achieved a 74% accuracy in differentiating between stress and non-stress scenarios utilizing leave-one-subject-out (LOSO) for person independent categorization. The authors achieved an accuracy of 91% using 5-fold cross-validation for classifying individuals based on dependency. A total of 17 distinct AUs were examined to identify and measure mental stress resulting from a multitasking activity coupled with social assessment. Authors used several elementary classifiers to conduct subject-specific and subject-agnostic analyses. In addition, other basic classifiers were used, including Random Forest, LDA, Gaussian Naive Bayes, and Decision Tree. The topic independent classification yields a range of outcomes, ranging from 29% for the 6-class issue to 74% for the binary classification. Conversely, the findings that are reliant on the individual range from 65% to 91% respectively.

The paper by Pongsakomsathien et al. [84] evaluated the feasibility of employing a wearable cardiac monitoring device for real-time HR recording in aerospace applications. The validity of the measurements from the wearable device was compared to a clinically validated device. The study focused on a challenging aerospace task involving time-critical HMI and high CW. The results supported the suitability of the sensor for the intended Cognitive Human-Machine Interfaces and Interactions (CHMI²) system application. The paper discussed the factors that can influence the validity performance of cardiac measurement data, such as hardware and software processing, filtering, and data handling methods, as well as issues like loose straps and contaminated electrodes.

3.2. Multimodal Physiological Monitoring of Cognitive Workload

The papers reporting a multimodal approach to monitor the CW are mainly related to study examples of different physiological data acquisitions at the same time, particularly eye tracking, heart rate variability, breathing rate and posture.

Beggiato et al. [76] gathered physiological measurements, including HR, pupil diameter, and eye blink rate, in order to detect discomfort during autonomous driving scenarios. HR consistently reduced during unpleasant conditions, but pupil diameter rose and eye blink rate decreased in visually observed uncomfortable scenarios. These modifications were considered as indicative of heightened information-processing demands and CW. The findings represented a foundation for creating an algorithm able to identify discomfort in real-time, to be tested on the road, using an autonomous car. Liang et al. [79] determined that using a touch screen input method took longer and involved more physical effort compared to using a mouse. The study was realized using, as system the user interacted with, “The Human machine interface for early warning aircraft”, and monitoring the eye tracking measurement, HR, and HRV. It improved information processing by lowering the average time spent on fixations. Additionally, using the color mode on the scenario map reduced the number of rapid eye movements (saccades) in comparison to the grayscale version. Particularly, grayscale mode helped to mitigate variations in CW caused by various input modalities, hence minimizing visual fatigue. Grayscale mode required more visual scanning and spatial orientation, while color mode lessened the burden of visual search and time demand. Utilizing color-coding on the scenario map enhances job efficiency and diminishes effort in comparison to monochromatic displays. The paper by Khamaisi et al. [80] outlined a methodology for evaluating the User Experience (UX) of employees with the aim of fostering a human-centered approach to manufacturing facilities and improving overall sustainability. The assessment entailed the use of non-intrusive wearable sensors to track human activities and gather physiological indicators, alongside questionnaires for subjective self-evaluation. The method was used on a virtual reality (VR) simulation of complex work sequences at an oil and gas pipes production site. This allowed for the detection of potentially stressful circumstances for the operators, both physically and mentally. The research highlighted the significance of acquiring a profound comprehension of the work environment and organization, together with expertise in users’ requirements and ergonomics, to enhance workers’ welfare, working circumstances, and industrial outcomes. The paper by Peruzzini et al. [81] presented a mathematical model that explores capacity management within the framework of Industry 4.0, using several costing models such as ABC (Activity Based Casting) and TDABC (Time-Driven Activity Based Casting). The research emphasized the balance between maximizing capacity and achieving operational efficiency, demonstrating that increasing capacity may mask operational inefficiencies. The study suggested using a mixed reality (MR) arrangement to facilitate the design for serviceability. Physiological signals, including HR, HRV, breathing rate, and posture, were observed to evaluate the physical and cognitive burden of users while performing maintenance activities in the MR set-up. The findings demonstrated that the optimal design solution effectively decreases the burden of operators, enhanced postural comfort, and mitigated cognitive stress. Ciccarelli and colleagues [85] introduced a technology that facilitated the tracking of operator’s activities, the analysis of data, and the execution of corrective measures to promote social sustainability in the workplace. The instrument enabled the monitoring of workers’ actions, as well as their physical and cognitive exertion, environmental comfort, and adherence to production standards such as work rhythm and product/component specifications. The data were collected via Internet of Things (IoT) devices, such as chest bands, wristbands, smart glasses, and inertial measurement units. The physical ergonomics data were analyzed by tracking the movement of body segments and measuring relative anatomical joint angles, steps, force load, and vibrations. This analysis was conducted to predict and prevent musculoskeletal problems. The cognitive ergonomics data were collected using physiological measures such HR, HRV, breathing rate, eye-related parameters (electrooculogram, blinks, fixation), and EDA. These factors enabled the identification of concerns pertaining to the psychological and cognitive reactions of employees. Ciccarelli et al. also presented a systematic review, analyzing the occupational risks faced by workers and the proposed solutions [86]. The Occupational Safety and Health Administration (OSHA) classified workplace risks into primary categories: safety hazards and ergonomic hazards. Recently, a new categorization has been introduced to include psychosocial and organizational risks, acknowledging the growing understanding of psychosocial factors. With respect to this goal, technology solutions included IoT systems, Digital Twin, and expanded reality were considered. Various studies suggested IoT frameworks for evaluating ergonomic assessment. These frameworks utilize wearable sensors to measure physiological parameters such as HR, HRV, respiratory rate, EDA, EEG, and pupillometry. The purpose of these assessments is to evaluate CW through AI technology, including ML, and computer vision.

The research by Peruzzini and colleagues [82] introduced a systematic approach for using current digital technologies to aid in product-process design. This approach involved analyzing workers’ behaviors and evaluating their perceived experience in industrial settings. The study proposed a systematic protocol analysis to objectively and quantitatively assess the workers’ experience. The goal was to assist in defining the needs for product-process design via the use of digital technology. This study examined the analysis of the perceived human experience in the working environment within the framework of smart factories. Specifically, it established a framework for assessing human stress and comfort to aid in the development of industrial systems, taking into account both product and process characteristics in a cohesive manner. The setup consisted of a GoPro Hero3 video camera for recording real workers and the work-space environment. Additionally, a Tecnomatix Jack software tool by Siemens was used to model the virtual environment and create a “digital twin” of the workplace. The setup also includes VICON Bonita cameras for full body tracking of real workers and real object tracking. To analyze the eye fixation of real users in the mixed reality (MR) environment, a high-quality eye tracking system called Glasses 2 by Tobii was used. Furthermore, a multi-parametric wearable sensor called BioHarness 3.0 by Zephyr was used for real-time monitoring of physiological parameters to analyze the physical and mental stress of real users. Regarding eye tracking, the collected data was very valuable for monitoring the interaction between humans and systems, as well as for establishing connections between eye-related data and human stress, CW, and emotions.

4. Cognitive Workload Assessment through Machine Learning Approaches

The paper by Smith et al. [87] centered on the estimation of the physical effort in human-robot teams that operate in unpredictable circumstances. The workload was divided into many components, including cognitive, visual, verbal, auditory, gross motor, fine motor, and tactile. The research employed wearable sensors, such as HR monitors, motion trackers, and surface electromyography sensors, to gather physiological measurements. These measurements were used to estimate the workload of gross motor skills, fine motor skills, and tactile abilities. The ML models built using these physiological measures were deemed unreliable as a result of task ambiguity and the presence of noise in the data.

Dell’Agnola et al. [88] proposed a new weighted-learning method for Support Vector Machine (SVM) to optimize the model for specific subjects. The study discussed the selection of ML algorithms based on factors such as data size and system requirements, with SVM being commonly used, but not consistently indicated as the best model. The personalized weighted-learning approach considered person-dependent variance in physiological response to workload, and the performance was evaluated using the NASA Task Load Index. The paper emphasized the multidimensional nature of CW and the need to consider task type, duration, and subjective psychological experiences. The specific results of the study were not mentioned in the provided sources.

Masinelli et al. [89] proposed a ML technique for multimodal CW monitoring and evaluation during manual labor to improve the lifetime of wearable sensors. The acquisition unit consisted of different wearable sensors worn by the workers during their manual labor.

The study of Zanetti et al. [90] proposed a ML design methodology and data processing strategy for real-time CW monitoring on resource-constrained wearable devices. The proposed CW monitoring solution achieved an accuracy of 74.5% and a geometric mean of 74.0% between sensitivity and specificity for classification on unseen data. The model optimization technique resulted in a model that is 27.5 times smaller than the default parameters, while the multi-batch data processing scheme decreased the amount of RAM memory used by 14 times. The method utilized a mere 1.28% of the total processing time, hence enabling the machine to attain a battery life of 28.5 h. The paper further demonstrated the validation of the suggested approach, compared it with a complete-capacity solution, utilized an artifact removal strategy, and provided execution profiling statistics on the e-Glass platform. The research by Pongsakornsathien et al. [91] explored the latest developments in sensor networks for aerospace cyber-physical systems, with a specific emphasis on the application of Cognitive Human-Machine Interfaces and Interactions. This article examined the essential neurophysiological metrics used in this environment and explores their correlation with the cognitive states of the operator. The article also introduced appropriate data analysis methods that use ML and statistical inference. These methods were capable of effectively processing both neurophysiological and operational data to produce precise estimates of cognitive states. The chosen sensors were eye tracking sensors, used to extract gaze features and pupillometry, and passive control systems, able to detect several eye activity characteristics, including fixations, blink rate, saccades, pupil diameter, visual entropy, and dwell duration. These variables were associated with the cognitive state of the operator. In particular, saccades presented two modes: real-time data transmission and data recording. The selected sensor was mainly used to measure the diameter of the pupil, as well as cardiac and respiratory activities, whereas neuroimaging technologies were used to observe and enhance comprehension of the brain’s functioning.

Neuroimaging techniques may be classified into two primary categories for attaining their goals: direct observation of neural activity in response to stimuli, and indirect measurement of neural activity through metabolic indicators. The first method involves the use of sensors, such as EEG, which capture the electrical signals produced by cerebral activity, caused by the activation of neurons [92,93]. The second method involves the use of sensors like fNIRS, which employs a spectroscopic approach to measure the levels of blood oxygenation in the brain’s cortex [94].

Moreover, facial expression analysis is a frequently used technique for assessing the emotional states of individuals. Just like speech patterns, analyzing facial expressions does not need any specialist equipment other than a camera (such as an RGB camera) and it is non-intrusive [95]. In this context, various ML methodologies were used to analyze image data, including Neural Fuzzy Systems, artificial neural networks, and convolutional neural networks [96,97].

The main papers described in this section are summarized in

Table 2. Main papers included into this section of the overview after the completion of the literature review procedure.

Authors	Objective	Signals Acquired	AI
Zanetti et al. [90]	The study focuses on building a cognitive workload monitoring (CWM) solution to assess the drone operators’ CW level during a simulated search and rescue mission.	EEG (Biosemi ActiveTwo system)	Random Forest
Smith et al. [87]	Development of models for estimating grossand fine motor, and tactileworkload.	• The BioharnessTM to collect heart rate, respiration rate, and postural magnitude; • Xsens MTw Awinda to measure participant’s body pose; • two Myo armbands, each equipped with 8-channels surface EMG and a forearm inertial metrics.	Several ML algorithms not specified
Dell’Agnola et al. [88]	ML algorithm for real-time cognitive workload monitoring of drones rescue operators.	Respiration, ECG, PPG, and skin temperature.	A new weighted-learning method for Support Vector Machine (SVM)
Pongsakornsathien et al. [84]	Development of a Cognitive Human-Machine Interfaces and Interactions (CHMI2) framework.	ECG through the BH system.	Adaptive Neuro-Fuzzy Inference System (ANFIS)
Ciccarelli et al. [85]	The supervision of operator’s tasks and the execution of remedial measures to ensure social sustainability in the workplace.	These data are acquired by IoT devices (i.e., chest band, wristband, smart glasses, inertial measurement units).	machine learning algorithms as STAI and NASA-TLX
Ciccarelli et al. [86]	A systematic review analyzing the occupational risks of the workers and the possible solutions.	wearable sensors to evaluate mental and cognitive workload collecting physiological parameters such as heart rate, Heart rate Variability, Respiratory rate (RR), Electrodermal Activity (EDA), electroencephalography (EEG) and pupillometry.	use of AI technologie, such as machine learning (ML) and computer vision
Viegas et al. [83]	A method able to assess mental stress through facial expressions detected only on video	17 Action Units (AUs) from upper-level to lower-level face frame-wise have been extracted.	Different classifiers were used: Random Forest, LDA, Gaussian Naive Bayes and Decision Tree.
Pongsakornsathien et al. [91]	Advances in sensor networks for aerospace cyber-physical systems are discussed, focusing on Cognitive Human-Machine Interfaces and Interactions implementations.	Eye fixations, blink rate, saccades, pupil diameter, visual entropy and dwell time; neuroimaging technologies	Neural Fuzzy Systems and networks, artificial neural networks, convolutional neural networks

.

5. Discussion

In this review the importance of the CW monitoring in several application fields is investigated. In fact, CW monitoring finds applications in various fields, including ergonomics and computer science.

In the field of ergonomics, the assessment of CW is a multifaceted endeavor that involves the integration of sensitive measurement techniques, such as physiological assessments, to accurately evaluate the mental demands placed on individuals during various tasks. The diverse applications of CW monitoring in ergonomics underscore its significance in enhancing human performance and well-being across different domains, including aviation, medicine, automotive, and HMI [98].

In the context of aviation, the assessment of CW has an exceptional importance due to its direct impact on flight safety and performance. In fact, the probability of human error in aviation due to heightened emotional states and CW can increase the probability of human error [99]. This highlights the critical need for real-time monitoring of CW to mitigate the risk of errors in aviation operations. The integration of advanced techniques for CW monitoring based on physiological data [100], and the consideration of CW in the design of aviation systems, cockpits, and operations underscore the significance in ensuring safe and efficient air travel [101].

The assessment of CW in the automotive is crucial for understanding the mental demands placed on individuals within the driving context. In fact, road accidents are one of the most prevalent causes of injury and death, and they are frequently due to the exaggerated CW and fatigue of the drivers [59,102]. Hence, in this field, predicting such cognitive states by means of ML algorithms applied to neurophysiological recordings could be crucial in preventive accidents.

In the context of medicine and healthcare, the assessment of CW represents an important aspect due to its direct impact on clinical performance, patient safety, and overall healthcare outcomes [103]. This suggests the relevance to apply advanced monitoring techniques to assess CW in clinical environments, thereby enhancing the understanding of mental demands placed on healthcare professionals during patient care. Additionally, monitoring CW has implications for mental health, as chronic exposure to high cognitive loads can lead to stress, burnout, and other mental health issues. Timely interventions not only help prevent these outcomes but also contribute to a healthier work environment.

The literature review demonstrated that ML plays a pivotal role in the quantification and monitoring of CW, offering advanced techniques for automated classification and real-time assessment, resulting highly suited for HMI applications. In fact, ML, combined with advanced digital signal processing, can achieve automated and accurate classifications of CW in natural environments, enhancing decision-making, safety, and performance across various operational contexts.

CW monitoring emerges as a pivotal component in the domain of HMI, offering substantial advantages for the efficacy of interactions and the well-being of human operators. Its significance is underscored in environments where humans are engaged with complex systems. Monitoring CW is instrumental in identifying instances of excessive stress or cognitive burden on operators, which could precipitate errors or accidents. Early detection of such instances facilitates timely interventions to mitigate CW, thereby enhancing operational safety. Furthermore, understanding CW enables the optimization of system design and the allocation of tasks between humans and machines, ensuring tasks are aligned with human cognitive capabilities. This alignment prevents operators from being underloaded, which can lead to boredom and loss of attention, or overloaded, which can result in errors and diminished performance. The concept of adaptive systems, which dynamically adjust their behavior based on the operator’s current cognitive load, relies heavily on CW monitoring. These systems might automate certain tasks when detecting high cognitive load or provide additional support or feedback to the operator. In the context of training and skill development, monitoring CW allows for the customization of training programs to meet individual needs, ensuring operators are well-prepared for their tasks. It also aids in identifying areas requiring further training. Ensuring the CW remains at an optimal level enhances overall user experience and satisfaction. Systems designed with an understanding of CW are likely to be more user-friendly and meet the operators’ needs more effectively.

Importantly, as assessed in the literature survey, to monitor CW effectively, various methods and technologies are utilized, including physiological measurements, performance measures, and subjective assessments. Integrating these measures into human-machine systems necessitates careful consideration of privacy, ethics, and the practicality of monitoring methods across different operational contexts.

6. Limitations and Future Directions

In the realm of HMI, CW monitoring is indispensable for optimizing system performance and user experience. Yet, the endeavor to effectively measure and adapt to CW in real-time presents a multifaceted challenge characterized by several limitations inherent to current methodologies.

A primary concern is the intrusiveness and practicality of some existing CW monitoring techniques. Methods that rely on physiological signals, such as EEG or HRV, often require equipment that can be obtrusive and uncomfortable for the user. This not only impacts the feasibility of deploying such measures in everyday scenarios but may also influence the cognitive load, thereby skewing the results. In this context, the employment of wearable sensors could enhance the application of CW in ecological contexts.

The accuracy and reliability of physiological measurements further compound the issue. In fact, physiological metrics can be susceptible to a range of external factors, from emotional states to health conditions, which may not necessarily correlate with the CW. The subjective nature of self-reporting measures introduces an additional layer of complexity, as these can be influenced by personal perceptions, potentially misrepresenting the actual workload.

Real-time monitoring, essential for the development of adaptive systems, faces its own set of hurdles. The technical challenge of processing data swiftly enough to make real-time adjustments, combined with the need for non-intrusive monitoring methods, presents a serious obstacle.

Moreover, the variability in individual responses to CW necessitates a tailored approach. Factors such as expertise, age, and cognitive capacity mean that a one-size-fits-all solution is untenable, requiring systems that can adapt to the nuances of individual user profiles. Notably, integrating CW monitoring into system design entails a multidisciplinary effort, bridging psychology, human factors, and engineering disciplines. The complexity of such integration, however, poses a barrier, necessitating customized solutions that can accommodate the specific demands of different applications.

Finally, the ethical and privacy implications of CW monitoring cannot be overlooked. The use of physiological and behavioral data raises considerable concerns regarding user privacy and autonomy. Establishing a framework for ethical monitoring practices is crucial, yet developing universally accepted guidelines remains an ongoing challenge.

Future studies in the monitoring of CW hold substantial potential for advancing our understanding of mental demands and enhancing performance across various domains. For instance, the implications of CW in financial decision-making processes should be further investigated, elucidating the impact of cognitive workload on decision quality and financial risk management [104]. Additionally, future studies could delve into CW implications for creative tasks and the associated brain network dynamics [105].

Importantly, the variability in individual responses to cognitive demands, coupled with external influences such as stress or fatigue, requests a personalized approach in interpreting physiological data. This investigates how individual physiological responses are not homogenous, but rather are influenced by a complex interplay of intrinsic and extrinsic factors. Such an approach acknowledges the differential impact of cognitive loads on diverse individuals, factoring in personal thresholds and resilience to stressors. It also calls for adaptable models that can dynamically adjust to these varying individual responses, ensuring that interpretations of physiological signals are contextually relevant and accurate. Future studies should indeed focus on this tailored approach, since it is essential for the accurate assessment of CW, enhancing the precision and applicability of monitoring systems across different user groups.

The imperative for future research in the domain of CW monitoring is to advance the integration of edge computing technologies within everyday life tasks, thereby enhancing real-time assessment capabilities. This promotes the development of computationally efficient models and methodologies tailored for monitoring and detecting individuals’ CW statuses with greater accuracy and reliability. Specifically, this involves designing algorithms that are not only robust and adaptive to varying contexts but also optimized for low-power and high-performance computations typical of edge computing environments. In the context of implementing these advancements, mobile devices, such as smartphones, emerge as a pivotal component. Their ubiquitous nature and advanced computational capabilities make them ideal candidates for CW monitoring within an IoT framework. This integration would allow for a seamless and unobtrusive collection of data, leveraging the sensors and computational power available in these devices. Moreover, the IoT framework could facilitate a more interconnected and responsive system, where data from various sources can be integrated and analyzed in real-time, thereby supporting humans in their day-to-day activities more effectively.

7. Conclusions

This narrative review examines the CW monitoring through physiological signals in several applications. Notably, the literature shows the potential of physiological signals to provide real-time, objective metrics of CW, which has profound implications for diverse fields such as workplace safety and HMI. However, challenges remain in standardizing these measures across different populations and contexts. Variability in individual responses to cognitive demands and the influence of external factors like stress or fatigue necessitate tailored approach to interpreting physiological data. There is also a need for advanced computational models that can accurately parse these complex signals and translate them into meaningful CW assessments.

Future research directions should focus on refining the accuracy and applicability of physiological markers, developing inclusive models that account for individual differences. The interdisciplinary nature of this field will continue to foster collaborations that push the boundaries of our understanding of the human brain at work, leading to innovations that enhance both productivity and well-being in various settings.