Facial Emotion Recognition Using Conventional Machine Learning and Deep Learning Methods: Current Achievements, Analysis and Remaining Challenges

Abstract

Facial emotion recognition (FER) is an emerging and significant research area in the pattern recognition domain. In daily life, the role of non-verbal communication is significant, and in overall communication, its involvement is around 55% to 93%. Facial emotion analysis is efficiently used in surveillance videos, expression analysis, gesture recognition, smart homes, computer games, depression treatment, patient monitoring, anxiety, detecting lies, psychoanalysis, paralinguistic communication, detecting operator fatigue and robotics. In this paper, we present a detailed review on FER. The literature is collected from different reputable research published during the current decade. This review is based on conventional machine learning (ML) and various deep learning (DL) approaches. Further, different FER datasets for evaluation metrics that are publicly available are discussed and compared with benchmark results. This paper provides a holistic review of FER using traditional ML and DL methods to highlight the future gap in this domain for new researchers. Finally, this review work is a guidebook and very helpful for young researchers in the FER area, providing a general understating and basic knowledge of the current state-of-the-art methods, and to experienced researchers looking for productive directions for future work.

1. Introduction

Facial emotions and their analysis play a vital role in non-verbal communication. It makes oral communication more efficient and conducive to understanding the concepts [1,2].

It is also conducive to detecting human attention, such as behavior, mental state, personality, crime tendency, lies, etc. Regardless of gender, nationality, culture and race, most people can recognize facial emotions easily. However, a challenging task is the automation of facial emotion detection and classification. The research community uses a few basic feelings, such as fear, aggression, upset and pleasure. However, differentiating between many feelings is very challenging for machines [3,4]. In addition, machines have to be trained well enough to understand the surrounding environment—specifically, an individual’s intentions. When machines are mentioned, this term includes robots and computers. A difference is that robots involve communication abilities to a more innovative extent since their design consists of some degree of autonomy [5,6]. The main problem is classifying people’s emotions is variations in gender, age, race, ethnicity and image quality or videos. It is necessary to provide a system capable of recognizing facial emotions with similar knowledge as possessed by humans. Recently, FER has become an emerging field of research, particularly for the last few decades. Computer vision techniques, AI, image processing and ML are widely used to expand effective automated facial recognition systems for security and healthcare applications [7,8,9,10].

Face detection is the first step of locating or detecting face(s) in a video or single image in the FER process. The images do not consist of faces only, but instead present with complex backgrounds. Indeed, human beings can easily predict facial emotions and other facial features of an image, but these are difficult tasks for machines without excellent training [11,12]. The primary purpose of face detection is to separate face images from the background (non-faces). Some face detection domains are gesture recognition, video surveillance systems, automated cameras, gender recognition, facial feature recognition, face recognition, tagging and teleconferencing [13,14]. These systems need first to detect faces as inputs. There is a color sensor for image acquisition that captures color images everywhere. Hence, current face recognition techniques depend heavily on grayscale, and there are only a few techniques that are able operate with color images. To achieve better performance, these systems either implement window-based or pixel-based techniques, which are the key categories of strategies for facial recognition. The pixel-based method lags in separating the face from the hands of another skin region of the person [15,16].

In contrast, the window-based method loses the capacity to view faces from various perspectives. Model matching approaches are the most frequently used techniques for FER, including face detection. In contrast, the window-based approach cannot view faces from different angles. Currently, several state-of-the-art classifiers, such as ANN, CNN, SVM, KNN, and random forest (RF), are employed for different features’ extraction and in the recognition of tumors in healthcare, in biometrics, in handwriting studies and in detecting faces for security measures [17,18,19].

Main Contributions

Over the past three decades, there has been a lot of research reported in the literature on facial emotion recognition (FER). Nevertheless, despite the long history of FER-related works, there are no systematic comparisons between traditional ML and DL approaches. Kołakowska [20] presented a review whose main focus was on conventional ML approaches. Ghayoumi [21] proposed a short review of DL in FER, and Ko et al. [22] presented a state-of-the-art review of facial emotion recognition techniques using visual data. However, they only provided the differences between conventional ML techniques and DL techniques. This paper presents a detailed investigation and comparisons of traditional ML and deep learning (DL) techniques with the following major contributions:

• The main focus is to provide a general understanding of the recent research and help newcomers to understand the essential modules and trends in the FER field.
• We present the use of several standard datasets consisting of video sequences and images with different characteristics and purposes.
• We compare DL and conventional ML approaches for FER in terms of resource utilization and accuracy. The DL-based approaches provide a high degree of accuracy but consume more time in training and require substantial processing capabilities, i.e., CPU and GPU. Thus, recently, several FER approaches have been used in an embedded system, e.g., Raspberry Pi, Jetson Nano, smartphones, etc.

This paper provides a holistic review of facial emotion recognition using the traditional ML and DL methods to highlight the future gap in this domain for new researchers. Further, Section 1 presents the related background on facial emotion recognition; Section 2 and Section 3 contain a brief review of traditional ML and DL. In Section 4, a detailed overview is presented of the FER datasets. Section 5 considers the performance of the current research work, and, finally, the research is concluded in Section 6.

2. Facial Emotion Recognition Using Traditional Machine Learning Approaches

Facial emotions are beneficial for investigating human behavior [23,24] as exhibited in Figure 1. Psychologically, it is proven that the facial emotion recognition process measures the eyes, nose, mouth and their locations.

The earliest approach used for facial emotion intensity estimation was based on distance urged. This approach uses high-dimensional rate transformation and regional volumetric distinction maps to categorize and quantify facial expressions. In videos, most systems use Principal Component Analysis (PCA) to represent facial expression features [25]. PCA has been used to recognize the action unit to express and establish different facial expressions. Other facial expressions are structured and recognized by mistreatment PCA for providing a facial action unit [26].

Siddiqi et al. [27] detected and extracted the face portion via the active contour model. The researchers used Chan–Vese and Bhattacharyya’s energy functions to optimize the distance between face and context, and reduce the differences within the face. In addition, noise is reduced using wavelet decomposition, and the geometric appearance features of facial emotions and facial movement features using optical flow are extracted.

There is no need for high computational power and memory for conventional ML methods such as DL methods. Therefore, these methods need further consideration to implement embedded devices that perform classification in real time with low computational power and provide satisfactory results. Accordingly,

Table 1. Summary of the representative conventional FER approaches.

References	Datasets	Decision Methods	Features	Emotions Analyzed
Varma et al. [28]	FECR	support vector machine (SVM) classifier and HMM	PCA and LDA	Six emotions
Reddy et al. [29]	CK+	support vector machine (SVM) classifier	Haar Wavelet Transform (HWT), Gabor wavelets and nonlinear principal component analysis (NLPCA)	Six emotions
Sajjad et al. [30]	MMI JAFFE CK+	support vector machine (SVM) classifier	ORB SIFT SURF	Seven emotions
Nazir et al. [31]	CK+ MMI	KNN, SMO and MLP for classification	HOG, DCT	Seven emotions
Zeng et al. [32]	CK+	DSAE	LBP, SIFT, Gabor Function and HOG	Seven emotions
Uddin et al. [33]	Real-time dataset from visual depth camera	Deep Belief Network (DBN)	MLDP-GDA features	Six emotions
Al-Agha et al. [34]	BOSPHORUS	Euclidean distance	Geometric descriptor	Four emotions
Ghimire et al. [35]	CK+ MMI MUG	EBGM, KLT and AdaBoost-ELM	Salient geometric features	Seven emotions
Wang and Yang [36]	BVTKFER BCurtinFaces	random forest classifier	LBP	Six emotions
Wu et al. [37]	BBC, SPOS MMI UvANEMO	support vector machine (SVM)	RSTD, four conventional features—raw pixels, Gabor, HOG and LBP	Two emotions Smile (genuine and fake)
Acevedo et al. [38]	CK+	Conditional Random Field (CRF) and KNN	Geometric descriptor	Seven emotions
Kim [39]	CK JAFFE	embedded hidden Markov model (EHMM)	ASM and 2D DCT	Seven emotions
Cornejo et al. [25]	CK+ JAFFE MUG	PCA, LDA, K-NN and SVM	Gabor wavelets and geometric features	Seven emotions
Siddiqi et al. [27]	CK JAFFE USTCNVIE Yale FEI	hidden Markov model (HMM)	Chan–Vese energy function, Bhattacharyya distance function wavelet decomposition and SWLDA	Six emotions
Chang et al. [40]	CK+	Support Vector Regression (SVR)	feature descriptors, AAMs, Gabor wavelets	Seven emotions

presents a brief summary

3. Facial Emotion Recognition Using Deep-Learning-Based Approaches

Deep learning (DL) algorithms have revolutionized the computer vision field in the current decade with RNN and CNN [41,42,43]. These DL-based methods are used for feature extraction, recognition and classification tasks. The key advantage of a DL approach (CNN) is to overcome the dependency on physics-based models and reduce the effort required in preprocessing and feature extraction phases [44,45]. In addition, DL methods enable end-to-end learning from input images directly. For these purposes, in several regions, including FER, scene awareness, face recognition and entity recognition, DL-based methods have obtained encouraging results from the state-of-the-art [46,47]. There are generally three layers in a DL-CNN, (1) convolution layer, (2) subsampling layer and (3) FC layer, as exhibited in Figure 2. The CNN takes the image or feature maps as the input, and slides these inputs together with a series of filter banks to produce feature maps that reflect the facial image’s spatial structure. Inside a feature map, the weights of convolutional filters are connected, and the feature map layer inputs are locally connected [48,49,50]. By implementing one of the most popular pooling approaches, i.e., max pooling, min pooling or average pooling, the second type of layer, called subsampling, is responsible for reducing the given feature maps [51,52]. A CNN architecture’s last FC layer calculates the class probability of an entire input image. Most DL-based techniques can be freely adapted with a CNN to detect emotions.

Li et al. [53] proposed a 3D CNN architecture to recognize several emotions from videos. They extracted deep features and used three benchmark datasets for the experimental evaluation, namely CASME II, CASME and SMIC. Li et al. [54] performed additional face cropping and rotation techniques for feature extraction using a convolutional neural network (CNN). Tests were carried out on the CK+ and JAFFE databases to test the proposed procedure.

A convolution neural network (CNN) with a focus function (ACNN) was suggested by Li et al. [55] that could interpret the occlusion regions of the face and concentrate on more discriminatory, non-occluded regions. A CNN is an end-to-end learning system. First, the different depictions of facial regions of interest (ROIs) are merged. Then, each representation is weighted by a proposed gate unit that calculates an adaptive weight according to the area’s significance. Two versions of ACNN have been developed in separate ROIs: patch-based CNN (pACNN) and global-local-based ACNN (gACNN). Lopes et al. [56] classified human faces into several emotion groups. They used three different architectures for classification: (1) a CNN with 5 C layers, (2) a baseline with one C layer and (3) a deeper CNN with several C layers. Breuer and Kimmel [57] trained a model using various datasets of FER to classify seven basic emotions. Chu et al. [58] presented multi-level mechanisms for detecting facial emotions by combining temporal and spatial features. They used a CNN architecture for spatial feature extraction and LSTMs to model the temporal dependencies. Finally, they fused the output of both LSTMs and CNNs to provide a per-frame prediction of twelve facial emotions. Hasani and Mahoor [59] presented the 3D Inception-ResNet model and LSTM unit, which were fused to extract temporal and spatial features from the input frames of video sequences. (Zhang et al., [60] and Jain et al. [61] suggested a multi-angle optimal pattern-dependent DL (MAOP-DL) system to address the problem of abrupt shifts in lighting and achieved proper alignment of the feature set by utilizing optimal arrangements centered on multi-angles. Their approach first subtracts the backdrop, isolates the subject from the face images and later removes the facial points’ texture patterns and the related main features. The related features are extracted and fed into an LSTM-CNN for facial expression prediction.

Al-Shabi et al. [62] qualified and collected a minimal sample of data for a model combination of CNN and SIFT features for facial expression research. A hybrid methodology was used to construct an efficient classification model, integrating CNN and SIFT functionality. Jung et al. [63] proposed a system in which two CNN models with different characteristics were used. Firstly, presence features were extracted from images, and secondly, temporal geometry features were extracted from the temporal facial landmark points. These models were fused into a novel integration scheme to increase FER efficiency. Yu and Zhang [64] used a hybrid CNN to execute FER and, in 2015, obtained state-of-the-art outcomes in FER. They used an assembly of CNNs with five convolution layers for each facet word. Their method imposed transformation on the input image in the training process, while their model produced predictions for each subject’s multiple emotions in the testing phase. They used stochastic pooling to deliver optimal efficiency, rather than utilizing peak pooling (

Table 2. Deep learning-based approaches for FER.

References	Datasets	Methods	Emotions Analyzed
Li et al. [65]	CK+ JAFFE	convolutional neural network (CNN)	Seven emotions
Jain et al. [61]	CK+ MMI	For the facial label prediction, they used LSTM-CNN	Six emotions
Li et al. [53]	SMIC CASME CASME II	3D flow-based CNN model for video-based micro-expression recognition	Five emotions
Xie et al. [66]	CK+ JAFFE	Attention-based SERD and the MPVS-Net	Six emotions
Li et al. [55]	RAF-DB AffectNet SFEW CK+ MMI Oulu-CASIA	CNN with ACNN	Seven emotions
Lopes et al. [56]	CK+ JAFFE BU-3DFE	CNN and specific image preprocessing steps	Seven emotions
Breuer and Kimmel [57]	CK+ FER2013	CNN with transfer learning	Seven emotions
Chu et al. [58]	BP4D	CNN is used to extract spatial representations and LSTMs used for temporal dependencies	Twelve emotions
Hasani and Mahoor [59]	CK+ DISFA	LSTM units extracted both temporal and spatial relations from facial images; as inputs of the network, they used facial landmark points	Twenty-three primary and compound emotions
Zhang et al. [60]	CK+ Oulu-CASIA MMI	PHRNN and MSCNN	Seven emotions
Al-Shabi et al. [62]	CK+ FER-2013	Dense SIFT and regular SIFT are merged with CNN features	Seven emotions
Jung et al. [63]	CK+ Oulu-CASIA	DTAN and DTGN	Seven emotions
Yu and Zhang [64]	FER-2013 SFEW	Used an assembly of DCNNs with seven convolution layers	Seven emotions

).

The hybrid CNN-RNN and CNN-LSTM techniques have comparable architectures, as discussed in the previous section and exhibited in Figure 3. In short, CNN-RNN’s simple architecture combines an LSTM with a DL software visual feature extractor, such as the CNN model. The hybrid techniques are, thus, equipped to distinguish emotions from image sequences. Figure 3 indicates that each graphic attribute has been translated to the LSTM blocks and describes a variable or fixed-length vector. Finally, performance is given for the prediction in a recurrent sequence learning module and the SoftMax classifier is used in [58].

Generally, DL-based methods determine classifiers and features by DNN experts, unlike traditional ML methods. DL-based methods extract useful features directly from training data using DCNNs [67,68]. However, the massive training data are a challenge for facial expressions under different conditions to train DNNs. Furthermore, DL-based methods need high computational power and a large amount of memory to train and test the model compared to traditional ML methods. Thus, it is necessary to decrease the computational time during the inferencing of DL methods.

4. Facial Emotion Datasets

Experts from different institutions have generated several datasets to evaluate reported methods for facial expression classification [69,70]. Accordingly, a detailed overview of some benchmark datasets is presented.

• The CK+ Cohen Kanade Dataset:

The Cohen Kanade database [71] is inclusive in that it consists of subject images of all sexes and races and is open to the public. This dataset consists of seven essential emotions that often include neutral emotions. The images’ resolution is 640 × 490 or 640 × 480 dimensions, with grayscale (8-bit) existing in the dataset. Approximately 81% of subjects are Euro-American, 13% are Afro American, and approximately 6% are from other races. In the CK+ dataset, the ratio of females is nearly 65%. The dataset consists of 593 images captured from 120 different people, and the age of these people varies from 18 to 30 years.

• Bosphorus dataset

This dataset consists of 2D and 3D faces for emotion recognition, facial action detection, 3D face reconstruction, etc. There are 105 humans with 4666 faces in different poses in the dataset. This dataset is different from other datasets in the following aspects.

• A rich collection of facial emotions is included:
  • Per person, at least 35 facial emotions are recoded;
  • FACS scoring;
  • Each third person is a professional actresses/actors.
• Systematic head poses are included.
• A variety of facial occlusions are included (eyeglasses, hands, hair, moustaches and beards).

• SMIC dataset

SMIC [54] contains macro- and micro-emotions, but the focus is on macro-emotions; this category includes 164 videos taken from 16 subjects. The videos were captured using a powerful 100 frame per second camera.

• BBC dataset

The BBC database is from the ‘Spot the fake smile’ test on the BBC website (http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/ accessed on 17 December 2021), which consists of 10 genuine and 10 posed smile videos collected with a resolution of 314 × 286 pixels at 25 frames/s from seven females and thirteen males.

The image representation of these datasets is shown in Figure 4 such that each row represents an individual dataset.

5. Performance Evaluation of FER

Performance evaluation based on quantitative comparison is an important technique to compare experimental results [72,73]. Benchmark comparisons on publicly available datasets are also presented. Two different mechanisms are used to evaluate the reported system’s accuracy: (1) cross-dataset and (2) subject-independent. Firstly, a subject-independent task separates each dataset into two parts: validation and training datasets. This process is also known as K-fold cross-validation [74,75]. K-fold cross-validation is used to overcome overfitting and provide insight into how the model will generalize to an unknown, independent dataset.

6. Evaluation Metrics/Performance Parameters

The evaluation metrics include overall accuracy, F-measure, recall and precision [76] Each evaluation metric is discussed separately below.

• Precision: This indicates how accurate or precise a model is, and it yields the actual positive values from a set of predictions by any algorithm. Precision is an excellent measure to indicate when the costs of false positives are relatively very high [77,78,79].

Precision = (True positive)/(True positive + False positive)

(1)

• Recall: A recall value computes how many actual positive values a mode captures while labeling it positive. This metric is used to select the best model when there is a high cost of false negatives [80,81].

Precision = (True positive)/(True positive + False negative)

(2)

• F1 Score: The F1 score is acquired when a balance is needed between precision and recall. F1 is a function of precision and recall [82,83].

F1 = 2 * (Precision * Recall)/(Precision + Recall)

(3)

• True Negatives (TN): Samples in the results are graded as not adequately belonging to the required class [84,85,86].
• False Negatives (FN): Samples are mistakenly used in data identified as not belonging to the required class [87,88,89,90].
• True Positives (TP): Samples in the data are appropriately categorized as belonging to the desired class [91,92,93].
• False Positives (FP): Samples in the data are categorized incorrectly as belonging to the target class [77,94].

Comparisons on Benchmark Datasets

It is challenging to compare different ML and deep-learning-based facial recognition strategies due to the different setups, datasets and machines used [95,96]. However, the latest comparison of different approaches is presented in

Table 3. Conventional ML-based approaches for FER.

References	Datasets	Accuracy (%)	Techniques
Varma et al. [28]	FECR	98.40 (SVM) 87.50 (HMM)	PCA and LDA for feature selection and SVM and HMM classifier
Reddy et al. [29]	CK+	98.00	Haar Wavelet Transform (HWT), Gabor wavelets and nonlinear principal component analysis (NLPCA) feature extractor and SVM classifier
Sajjad et al. [30]	MMI JAFFE CK+	99.1 on MMI accuracy (92) on JAFFE accuracy (90) on CK+	ORB SIFT SURF using SVM
Nazir et al. [31]	CK+ MMI	99.6 by using only 32 features with KNN	HOG, DCT features and KNN, SMO and MLP for classification
Zeng et al. [32]	CK+	95.79	LBP, SIFT, Gabor Function and HOG features and Deep Sparse Auto-Encoder (DSAE) classifier
Uddin et al. [33]	Real-time dataset from visual depth camera	96.25	MLDP-GDA features fed to Deep Belief Network (DBN) for prediction
Ghimire et al. [35]	CK+ MMI MUG	97.80 77.22 95.50	Salient geometric features and EBGM, KLT and AdaBoost-ELM class
Yang et al. [96]	BVT-KFER BCurtinFaces	93.33 92.50	LBP features and a random forest classifier
Wu et al. [37]	BBC SPOS MMI UvA-NEMO	90.00 81.25 92.21 93.95	RSTD, four conventional features—raw pixel, Gabor, HOG and LBP features Support vector machine (SVM) classifiers
Acevedo et al. [38]	CK+	89.3 86.9	Geometric descriptor and Conditional Random Field (CRF) and KNN classifier
Al-Agha et al. [34]	BOSPHORUS	85.00	Geometric descriptor and Euclidean distance is used for the prediction
Kim [39]	CK JAFFE	77.00 64.7	ASM and 2D DCT features and embedded hidden Markov model (EHMM)
Siddiqi et al. [27]	CK JAFFE USTC-NVIE Yale FEI	99.33 96.50 99.17 99.33 99.50	Features are extracted by using Chan–Vese energy function, Bhattacharyya distance function wavelet decomposition and SWLDA and hidden Markov model (HMM) is used for prediction
Cornejo et al. [25]	CK+ JAFFE MUG	99.78 99.17 99.94	Geometric + PCA + LDA + KNN Geometric + PCA + LDA + K − NN Geometric + PCA + LDA + K − NN
Chang et al. [40]	CK+	Mean Error (AAMs 0.316) Mean Error (Gabor 0.349) MAEs (AAMs 0.324) MAEs (Gabor0.651)	feature descriptors, AAMs, Gabor wavelets and support vector regression (SVR) are used for classification

.

As shown in Table 3 and

Table 4. Comparisons of deep-learning-based approaches for FER.

References	Datasets	Accuracy (%)	Methods
Li et al. [65]	CK+ JAFFE	97.38 0.14 SD 97.18 0.30 SD	Convolutional neural network (CNN)
Li et al. [55]	SMIC CASME CASME II	55.49 54.44 59.11	3D flow-based CNN model for video-based micro-expression recognition
Xie et al. [66]	CK+ JAFFE	With 10-fold cross-validation accuracy 95.88 99.32	Expressional Region Descriptor (SERD) and Multi-Path Variation-Suppressing Network (MPVS-Net)
Li et al. [55]	RAF-DB AffectNet SFEW CK+ MMI Oulu-CASIA	91.64	Convolutional neutral network (CNN) with attention mechanism (ACNN)
Lopes et al. [56]	CK+ JAFFE BU-3DFE	Average of C6class (96.76) Average of Cbin (98.92) Average of C6class (72.89) Average of Cbin (90.96) Average of C6class (53.44) Average of Cbin (84.48)	Convolutional neural network and specific image preprocessing steps
Breuer and Kimmel [57]	CK+ FER2013	98.62 ± 0.11 72 ± 10.5	Convolutional neural network (CNN) with transfer learning
Chu et al. [58]	BP4D	Accuracy on 3-fold (53.2) Accuracy on 10-fold (82.5)	Extracted spatial representations by CNN and for temporal dependencies, LSTMs were used
Hasani and Mahoor [58]	CK+ DISFA	89.5 56.0	Facial landmark points fed to CNN
Jain et al. [61]	CK+ MMI	96.17 98.72	LSTM CNN using texture patterns for facial expression prediction
(Zhang et al. 2017)	CK+ Oulu-CASIA MMI	97.78 86.25 79.29	Part-Based Hierarchical Bidirectional Recurrent Neural Network (PHRNN) and Multi-Signal Convolutional Neural Network (MSCNN)
Al-Shabi et al. [62]	CK+ FER-2013	99.1 73.4	Dense SIFT and regular SIFT are merged with CNN features
Jung et al. [63]	CK+ Oulu-CASIA	97.25 81.46	Fusion of deep temporal appearance network (DTAN) and deep temporal geometry network (DTGN) is employed to predict facial expressions
Yu and Zhang [64]	SFEW	Validation accuracy 55.96 Test accuracy 61.29	Assembly of DCNNs with seven convolution layers

, deep-learning-based approaches outperform conventional approaches.

In experimental tests, DL-based FER methods have achieved high precision; however, a range of issues remain that require further investigation:

• As the framework becomes increasingly deeper for preparation, a large-scale dataset and significant computational resources are needed.
• Significant quantities of datasets that are manually compiled and labeled are required.
• A significant amount of memory is required for experiments and testing, which is time-consuming.

The approaches mentioned above, especially those based on deep learning, require massive computation power. Moreover, these approaches are developed for specific emotions, and thus are not suitable to classify other emotional states. Therefore, developing a new framework to be applied to the whole spectrum of emotions would be of significant relevance, and could be expanded to classify complex facial expressions.

7. Conclusions and Future Work

In this paper, a detailed analysis and comparison are presented on FER approaches. We categorized these approaches into two major groups: (1) conventional ML-based approaches and (2) DL-based approaches. The convention ML approach consists of face detection, feature extraction from detected faces and emotion classification based on extracted features. Several classification schemes are used in conventional ML for FER, consisting of random forest, AdaBoost, KNN and SVM. In contrast with DL-based FER methods, the dependency on face physics-based models is highly reduced. In addition, they reduce the preprocessing time to enable “end-to-end” learning in the input images. However, these methods consume more time in both the training and testing phases. Although a hybrid architecture demonstrates better performance, micro-expressions remain difficult tasks to solve due to other possible movements of the face that occur unwillingly.

Additionally, different datasets related to FER are elaborated for the new researchers in this area. For example, human facial emotions have been examined in a traditional database with 2D video sequences or 2D images. However, facial emotion recognition based on 2D data is unable to handle large variations in pose and subtle facial behaviors. Therefore, recently, 3D facial emotion datasets have been considered to provide better results. Moreover, different FER approaches and standard evaluation metrics have been used for comparison purposes, e.g., accuracy, precision, recall, etc.

FER performance has increased due to the combination of DL approaches. In this modern age, the production of sensible machines is very significant, recognizing the facial emotions of different individuals and performing actions accordingly. It has been suggested that emotion-oriented DL approaches can be designed and fused with IoT sensors. In this case, it is predicted that this will increase FER’s performance to the same level as human beings, which will be very helpful in healthcare, investigation, security and surveillance.