Emotions Matter: A Systematic Review and Meta-Analysis of the Detection and Classification of Students’ Emotions in STEM during Online Learning

Abstract

In recent years, the rapid growth of online learning has highlighted the need for effective methods to monitor and improve student experiences. Emotions play a crucial role in shaping students’ engagement, motivation, and satisfaction in online learning environments, particularly in complex STEM subjects. In this context, sentiment analysis has emerged as a promising tool to detect and classify emotions expressed in textual and visual forms. This study offers an extensive literature review using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) technique on the role of sentiment analysis in student satisfaction and online learning in STEM subjects. The review analyses the applicability, challenges, and limitations of text- and facial-based sentiment analysis techniques in educational settings by reviewing 57 peer-reviewed research articles out of 236 articles, published between 2015 and 2023, initially identified through a comprehensive search strategy. Through an extensive search and scrutiny process, these articles were selected based on their relevance and contribution to the topic. The review’s findings indicate that sentiment analysis holds significant potential for improving student experiences, encouraging personalised learning, and promoting satisfaction in the online learning environment. Educators and administrators can gain valuable insights into students’ emotions and perceptions by employing computational techniques to analyse and interpret emotions expressed in text and facial expressions. However, the review also identifies several challenges and limitations associated with sentiment analysis in educational settings. These challenges include the need for accurate emotion detection and interpretation, addressing cultural and linguistic variations, ensuring data privacy and ethics, and a reliance on high-quality data sources. Despite these challenges, the review highlights the immense potential of sentiment analysis in transforming online learning experiences in STEM subjects and recommends further research and development in this area.

1. Introduction

The global online learning market is expected to surpass USD 848.12 billion by 2030 [1]. It has become popular, especially in response to the COVID-19 pandemic. As a result, educators have had to adapt their teaching methods to accommodate the needs of online learners. One of the most challenging problems for instructors is keeping learners engaged in the learning process. User engagement is a user experience attribute that assesses a person’s cognitive, affective, and behavioural investment when interacting with a digital system [2]. In contrast, learner engagement refers to students’ level of attention, interest, optimism, and passion when learning or being taught [3]. Engagement is critical for learning because it increases learners’ likelihood of retaining information and completing the course [4,5]. Keeping students motivated in an online learning environment can be challenging because of many distractions, such as noise from family members or pets, technology issues, and other demands such as self-discipline and time management. To ensure students’ success, educators must develop innovative ways to keep them engaged and motivated. Interactive technologies are one of the techniques to increase learner engagement in online learning. By providing a more immersive and dynamic learning environment, interactive technology helps keep learners interested [6]. For example, online discussion forums and chat rooms can be utilised to increase student collaboration and promote active learning. Online quizzes and games, on the other hand, can make studying more interesting and engaging while providing students with instant feedback on their progress [7]. Virtual reality (VR) and augmented reality (AR) can also be used to create more engaging and immersive learning experiences. Educators can assist learners in staying engaged and motivated in their studies by introducing these technologies into their online courses.

Sentiment analysis (SA) is becoming increasingly significant in many fields, including marketing, healthcare, and education, since it enables the capture and evaluation of emotions expressed through various modalities, such as text, facial gestures, electroencephalogram (EEG) signals, and voices [8]. Like other modalities, EEG signals are also important for capturing emotions, as they directly measure brain activity associated with emotional states [9]. They can provide insights into people’s feelings, attitudes, and opinions by analysing sentiment, which can inform decision making and improve the quality of services and goods. For example, sentiment analysis of customer feedback can assist businesses in improving their products or services, whereas sentiment analysis of patient feedback can assist healthcare practitioners in improving patient care. Furthermore, sentiment analysis can be used to monitor online conversations for potential threats such as cyberbullying or hate speech and in a classroom setting to analyse students’ facial expressions and infer their emotional states or sentiment during learning activities. As sentiment analysis continues to develop and expand, it can revolutionise how we understand and interact with each other, both online and offline.

Based on sentiment analysis, it is evident that educators and learners consistently emphasise and value the importance of student engagement in online learning [10]. Sentiment analysis which examines online conversations and social media activity may disclose students’ views and opinions about online learning and the efficiency of various engagement tactics such as interactive multimedia, gamification, and personalised learning. In general, sentiment analysis has revealed that students are more likely to be interested and motivated in interactive, collaborative, and immersive online learning settings [11]. Educators may contribute to creating a more helpful and effective online learning experience for all students by assessing their feelings towards online learning and utilising this knowledge to drive their teaching practices. Furthermore, sentiment analysis has also demonstrated that learner engagement positively correlates with academic achievement [11]. When learners actively engage in learning, they are more likely to retain information, apply it to real-world situations, and achieve higher grades. In contrast, disengaged learners are more likely to struggle with motivation and may have lower levels of academic achievement [12]. This highlights the importance of implementing effective engagement strategies in online learning, particularly during the COVID-19 pandemic, which has forced many learners to learn remotely. By leveraging sentiment analysis to identify the most effective strategies for promoting engagement, educators can help ensure their learners are successful and prepared for future academic and professional pursuits. Figure 1 shows an illustration of text-based and facial sentiment analysis in online learning.

Although sentiment analysis has grown in popularity in recent years to assess people’s attitudes and opinions on various topics, including online learning, several issues are associated with using sentiment analysis to analyse chat, face gestures, and videos. One of the most challenging problems is that different modes of communication can be complicated and multifaceted, making it difficult to capture and understand sentiment effectively [13]. Individuals, for example, might use sarcasm or irony in written or verbal discussions, which sentiment analysis algorithms may find difficult to distinguish [14]. Similarly, facial expressions and body language can convey a wide range of emotions and attitudes, which can be difficult to categorise using automated sentiment analysis approaches [15]. Other challenges with sentiment analysis are the possibility of data bias, language barriers, cultural variations, and individual writing styles, which may all impact chat data.

In light of the issues associated with chat-based and facial gesture sentiment analysis in the context of online and e-learning, there is a clear need for further research to improve the accuracy and reliability of these modalities. Developing more advanced techniques can address the challenges in accurately interpreting text-based and facial expressions and improve the overall accuracy of sentiment analysis. Furthermore, a systematic review study is necessary to analyse the current datasets, techniques, and performance of text-based and facial gesture sentiment analysis to make recommendations to improve sentiment analysis in online learning.

Furthermore, this paper presents four research questions, which are given in

Table 1. Research questions.

Question No.	Research Question
RQ1	How has text-based sentiment analysis been used in the educational domain to facilitate learners?
RQ2	What are the challenges/limitations of text-based sentiment analysis in online learning?
RQ3	How has facial sentiment analysis been used in the educational domain to facilitate learners?
RQ4	What are the challenges/limitations of facial sentiment analysis in online learning?

. It is important to note that these research questions are answered in the Results and Analysis section (i.e., Section 5), based on the review and evaluation carried out in our study.

Sentiment analysis is a widely applied technique in various domains, including education, where it has attracted considerable attention from researchers. However, comprehensive, and systematic literature reviews on sentiment analysis in education are scarce and limited. For instance, Oghu et al. [16] conducted a systematic literature review on the techniques, resources, and benefits of sentiment analysis in education; however, they included a limited number of papers in their study. Similarly, Dolianiti et al. [17] reviewed the current state of sentiment analysis in education, but they did not examine the experimental processes and methods of sentiment analysis in education, nor did they consider the sentiment categories, research objectives, and key findings of the studies. Another study, by Jin Zhou et al. [18], attempted to fill this gap by conducting a systematic literature review on sentiment analysis in education from 2010 to 2020, focusing on the research purposes, sentiment categories, data sources, methods, and findings of the studies. However, this study also had some limitations, such as excluding some relevant papers that did not match the search terms, only reviewing journal publications, and not performing a meta-analysis of the included studies. Moreover, most of the existing literature reviews on sentiment analysis in education are focused on text-based data, while other forms of data, such as EEG signals, audio, and video, still need to be addressed. For example, Kastrati et al. [19] reviewed the approaches/techniques and solutions for text-based sentiment analysis systems, as well as the evaluation metrics and datasets used for assessing their performance. They discussed the use of natural language processing (NLP) techniques, such as lexicon- and dictionary-based approaches, for text-based SA. However, they did not address the challenges and opportunities of applying sentiment analysis to other types of data that are increasingly available in educational settings. Wankhade et al. [20] performed a comprehensive and systematic literature review on the methods and evaluation of sentiment analysis tasks and their applications in various domains. They evaluated the strengths and limitations of the methods and identified the future research challenges of sentiment analysis regarding both the methods and the data types. However, this survey does not discuss the sentiment analysis in the educational and online learning domains.

Therefore, there is a need for a more comprehensive and updated literature review on sentiment analysis in education that covers both text-based and facial-based data sources along with different methods and applications in various educational contexts. This paper provides a holistic overview of the current state of the art and future directions of sentiment analysis in education.

Furthermore, our study emphasises the vast potential of sentiment analysis in reshaping online learning experiences within STEM subjects. In addition, it calls for additional research and advancements in this domain to unlock further potential.

2. Theoretical Background

This section provides an essential foundation for understanding the key concepts and techniques related to sentiment analysis. It encompasses two main subsections, i.e., text-based sentiment analysis and facial sentiment analysis. The former is divided into four levels and the latter is divided into two broad levels, as shown in Figure 2. We further explore these levels of sentiment analysis and investigate the techniques employed in both text and facial sentiment analysis. This section aims to establish a basis for the investigations and findings presented in this paper by comprehensively addressing both text-based and facial sentiment analysis.

2.1. Text-Based Sentiment Analysis

Text-based sentiment analysis uses NLP techniques to extract useful information from a text and classify that information into certain sentiment classes such as happiness, sadness, stress, angriness, and other emotions [21]. Text-based sentiment analysis aims to extract the required information and meaningful insights from text, allowing educators to make informed decisions. Text-based sentiment analysis can be used in certain areas, such as emotion detection from Twitter posts, detecting learners’ emotions about teaching methods and content using student responses or surveys, and detecting consumers’ emotions when utilising a product or service. For example, the authors in [22] used sentiment analysis on Twitter data to gather comments on movies before they were released. This study forecast box-office revenues for upcoming movies and discovered a significant association between the amount of attention an upcoming movie has on Twitter and its ranking in the future. Another study used sentiment analysis on the reviews of mobile apps to select highly rated apps for autism spectrum disorder (ASD) [23]. Similarly, the authors in [24] used sentiment analysis in the educational domain, utilising students’ chat data during online learning sessions. This study explored the effectiveness of different tutoring strategies and teaching content by classifying students’ responses into positive and negative emotions.

After building our basic understanding of text-based sentiment analysis, we further explore it by investigating two key aspects: the levels of analysis and the range of techniques utilised in this field.

2.1.1. Text-Based Sentiment Analysis Levels

Text-based sentiment analysis can be applied on four levels of granularity, i.e., (1) document-level analysis, (2) sentence-level analysis, (3) phrase-level analysis, and (4) aspect-level analysis. The different levels of text-based sentiment analysis are further discussed below:

(a)

Document-Level Sentiment Analysis: The approach of document-level sentiment analysis involves analysing an entire document and assigning a single polarity to the document as a whole [20]. While this approach is not frequently used, it can be helpful in categorising chapters or pages of a book as positive, negative, or neutral sentiments. Suppose an article on “using technology in the educational domain to enhance student learning” can be reviewed using document-level sentiment analysis to determine the overall sentiment expressed in the article regarding the impact of technology integration on student learning outcomes. Both supervised and unsupervised learning methods can classify the document [25]. One of the most significant challenges in document-level sentiment analysis is cross-domain and cross-language sentiment analysis [25]. For instance, domain-specific sentiment analysis has achieved high accuracy while maintaining domain sensitivity. These tasks require a set of domain-specific and -limited words to be used as the feature vector.

(b)

Sentence-Level Sentiment Analysis: Sentence-level sentiment analysis evaluates the emotions of individual sentences inside a text. For example, “The teacher explained this topic very well”. In this case, the sentence-level sentiment analysis technique would analyse and classify the sentence as positive. While compared to document-level sentiment analysis, this approach provides a more granular sentiment analysis. For sentence-level sentiment analysis, machine learning methods such as naive Bayes (NB), support vector machine (SVM), and deep learning models, e.g., recurrent neural networks (RNNs) and convolutional neural networks (CNNs), can be utilised [26]. Handling negation, sarcasm, and irony, and dealing with domain-specific language and jargon are typical challenges of sentence-level sentiment analysis. Despite these challenges, sentence-level sentiment analysis has applications in various domains, including customer feedback analysis, social media monitoring, and product review analysis. This approach provides a more thorough understanding of the sentiment expressed in a text by analysing the sentiment of individual sentences, which can be used to improve decision making and consumer satisfaction.

(c)

Phrase-Level Sentiment Analysis: The sentiment of individual phrases or expressions inside a sentence or text is analysed at the phrase-level. For example, “The teaching style and content of the topic helped me in understanding this complex information”. In this example, the phrase “helped me in understanding this complex information” indicates a positive sentiment as it suggests that the teaching style and content effectively facilitated the comprehension of complex information. Compared to sentence-level sentiment analysis, this approach provides an even more granular analysis. For phrase-level sentiment analysis, machine learning algorithms such as SVM, decision trees, neural networks, and rule-based techniques are often utilised [27]. Dealing with context-dependent phrases, everyday idioms, and unclear words are the most common challenges of phrase-level sentiment analysis. However, phrase-level sentiment analysis has many applications, including customer review analysis, social media monitoring, opinion mining, etc. This approach provides a more sophisticated understanding of the sentiment represented in a text by analysing the sentiment of individual phrases, which can aid in strengthening decision making and consumer satisfaction.

(d)

Aspect-Level Sentiment Analysis: The sentiment of specific aspects or features of a product or service described in a text is analysed using aspect-level sentiment analysis. This method is beneficial for businesses as it reveals which components of their product or service customers most appreciate or dislike. Machine learning algorithms such as SVM, RNN, and CNN are commonly used for aspect-level sentiment analysis [28]. In a massive open online course (MOOC) review, for example, aspect-level sentiment analysis would analyse the sentiment of specific factors such as learning material, teaching quality, and instructor experience, rather than just giving an overall positive or negative rating.

2.1.2. Text-Based Sentiment Analysis Techniques

Text-based sentiment analysis is a prominent and widely used form of sentiment analysis, so various techniques have been developed to perform sentiment analysis. Lexicon-based, machine-learning-based, and hybrid methods are some of the most commonly used techniques in text-based sentiment analysis. Figure 3 provides an overview of different sentiment analysis techniques for emotion detection and classification. This section focuses on two prominent approaches: the lexicon-based and machine learning approaches. It will provide the necessary understanding of the methodologies employed to extract sentiment information from textual data, highlighting their distinctive methodologies and practical applications.

(a)

Lexicon-Based Approach: Lexicon-based sentiment analysis is an established method that determines the sentiment of a text by using pre-defined dictionaries of terms and their associated sentiment scores. Each word in the text is scored based on its polarity, i.e., positive, negative, or neutral. The sum or average of the scores of the words in the text is then used to calculate the overall sentiment of the text [30]. Lexicon-based techniques have the advantage of being reasonably simple to implement and requiring minimal annotated data for training. However, the coverage and quality of the lexicons used may limit their accuracy [31]. To address this limitation, researchers have developed various lexicons specific to particular domains or languages [32]. Furthermore, lexicon-based approaches can be combined with other techniques, such as part-of-speech tagging and syntactic parsing, to improve the accuracy of the sentiment analysis [33]. Some of the prominent techniques of lexicon-based sentiment analysis used in the literature are given in

Table 2. Lexicon-based sentiment analysis techniques.

Technique	Description	Reference(s)
SentiWordNet	A publicly available lexical resource for opinion mining which assigns to each synset of WordNet three sentiment scores: positivity, negativity, and objectivity.	[34,35,36]
VADER	A rule-based sentiment analysis tool that uses a lexicon of words and their intensity scores, as well as grammatical rules, to determine the polarity of a given text.	[37,38],
SenticNet	A concept-level sentiment analysis framework that assigns sentiment scores to concepts based on their semantic orientation, conceptual polarity, and semantic relatedness to other concepts.	[39,40]
AFINN	A list of English words rated for valence with an integer between minus five (negative) and plus five (positive).	[41,42]
NRC Emotion Lexicon	A list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive).	[43,44]
Pattern	A Python package that includes a sentiment analysis module based on a lexicon of sentiment words and a rule-based classifier. It can handle negations, idioms, and slang, and can also be trained on custom data.	[45,46]
TextBlob	A Python library that includes a sentiment analysis module based on the pattern analyser. It also includes a naive Bayes classifier that can be trained on custom data.	[47,48]

.

(b)

Machine Learning Approach: Machine-learning-based approaches have been widely used in sentiment analysis due to their ability to learn complex patterns and relationships in data automatically. One prominent strategy is using supervised learning algorithms such as SVM, NB, and decision trees to classify text as positive, negative, or neutral based on labelled training data [49]. Unsupervised learning is another strategy, which involves grouping similar documents based on their sentiment using techniques such as k-means clustering or latent Dirichlet allocation (LDA) [50].

Deep learning approaches, such as RNN and CNN, have also been successfully used in sentiment analysis tasks [51]. These algorithms require enormous training data and computer resources, yet they can achieve excellent accuracy and generalisation across domains and languages. Overall, machine-learning-based techniques for sentiment analysis on text data provide a robust and adaptable method. A list of the most commonly used machine learning techniques for sentiment analysis is given in

Table 3. Machine learning sentiment analysis techniques.

Technique	Description	Reference(s)
Naive Bayes	A probabilistic algorithm that uses Bayes’ theorem to classify text as positive, negative, or neutral.	[52,53]
Support Vector Machine (SVM)	A supervised learning algorithm that separates data into different classes using a hyperplane.	[54,55]
Random Forest	An ensemble learning algorithm that constructs multiple decision trees to classify data.	[56,57]
Convolutional Neural Network (CNN)	A type of neural network that uses convolutional layers to automatically learn features from input data.	[58,59]
Long Short-Term Memory (LSTM)	A type of recurrent neural network that is capable of capturing long-term dependencies in input data.	[60,61]
Artificial Neural Networks (ANNs)	A set of algorithms that attempt to recognise underlying relationships in a data set through a process that mimics how the human brain operates.	[62,63]
BERT	A pre-trained language model that uses deep neural networks to generate contextualised word embeddings. BERT has been shown to achieve state-of-the-art results in a wide range of natural language processing tasks, including sentiment analysis.	[64,65]

.

2.2. Facial Sentiment Analysis

Facial sentiment analysis (FSA) is an emerging field that has gained much attention recently due to its potential applications in various domains, including healthcare, education, marketing, and entertainment. FSA involves computer vision techniques to recognise facial expressions and automatically extract individuals’ emotional states. The main goal of FSA is to understand human emotions and behaviour by analysing facial features such as eyebrows, eyes, nose, and mouth. FSA has been utilised in many applications, such as emotion recognition, stress detection, and deception detection. Facial expression recognition to gauge students’ emotional responses offers clear benefits over relying solely on self-reported data. Self-reports can be influenced by subjectivity and dishonesty, as students may not accurately convey their true feelings. Facial expressions, on the other hand, provide an objective, measurable indicator of emotions that is not affected by conscious biases. The automatic nature of facial recognition also leads to more natural, instinctual reactions from students rather than ones filtered through deliberation. We anticipate that real-time feedback promotes the generation of novel solutions, collaboration, and student engagement, which have a strong pedagogical impact on learning and teaching methodologies. Deep learning algorithms have significantly improved FSA performance in recent years, allowing for more accurate and reliable analysis.

Several studies have explored the potential of FSA in various domains. For instance, FSA has been used in healthcare to recognise emotions in patients with depression, anxiety, and bipolar disorder. A study [66] developed a deep-learning-based FSA model to identify depression in patients by analysing their facial expressions. The model achieved an accuracy of 93.3%, demonstrating FSA’s potential in detecting mental disorders. In the education domain, FSA has been utilised to monitor students’ engagement and attention levels during online learning [67]. This section will explore the different levels at which facial expressions are analysed to determine emotional states and investigate various methodologies employed to detect and interpret emotions from facial expressions.

2.2.1. Levels of Facial Sentiment Analysis

FSA can be performed at different levels depending on the level of granularity required to extract emotional information from facial expressions. The three primary levels of FSA are (1) face-level analysis, (2) region-level analysis, and (3) landmark-level analysis, which are further discussed below:

(a)

Face-Level Analysis: At the face level, FSA involves recognising emotions shown by individuals through facial expressions. This type of analysis is critical in fields including psychology, marketing, and human–computer interaction, where understanding emotions and their effects are vital. Several approaches to performing face-level sentiment analysis have been proposed, including rule-based, feature-based, and deep-learning-based methods. A recent study proposed a deep-learning-based approach for facial sentiment analysis that outperformed traditional methods [68]. The proposed model utilised a CNN for feature extraction and a long short-term memory (LSTM) network for sequence modelling. Another study proposed a method for facial sentiment analysis that utilised a set of hand-crafted features, including facial action units and their combinations, to train an SVM classifier [69]. The proposed method achieved a high accuracy of 89.5% on the AffectNet dataset [70]. These studies highlight the effectiveness of deep-learning-based and feature-based approaches for face-level sentiment analysis.

(b)

Region-Level Analysis: Region-level facial sentiment analysis involves analysing the emotional expressions of specific regions of the face, such as the eyes, mouth, or eyebrows. This approach allows for a more fine-grained analysis of emotional expression and can provide insights into the conveyed emotions. Various region-level facial sentiment analysis techniques have been proposed, including deep-learning-based methods such as CNN and RNN [71]. These techniques have been shown to achieve high accuracy in detecting emotions from specific regions of the face, such as the eyes or mouth. Other approaches include the use of geometric features and hand-crafted features, such as local binary patterns (LBPs) and histogram of oriented gradients (HOGs), which have also been shown to be effective in region-level sentiment analysis [72]. However, region-level analysis is still challenging due to occlusion and variations in expression intensity, which can impact emotion recognition accuracy [73]. Overall, region-level facial sentiment analysis has shown promise in improving the accuracy and granularity in the recognition of facial expressions’ emotions. Further research is needed to address the remaining challenges, such as subtle emotion recognition, subjectivity, and individual variance and contextual understanding.

(c)

Landmark-Level Analysis: Facial landmark detection is another approach used for FSA. This approach is based on extracting facial landmarks, defined as critical points on the face, such as the corners of the mouth, nose, and eyes. These landmarks are then used to extract features for emotion recognition. This technique has been used in recent studies for FSA. For example, in a study, the authors used facial landmarks to recognise emotions from YouTube videos to develop an effective feature selection algorithm to determine the optimal features for further improving the performance of multimodal sentiment analysis [74]. Another study utilised facial landmarks for emotion recognition in the context of social robotics [75]. The landmark-level approach is considered more accurate than the face- or region-level approaches as it captures more subtle changes in facial expressions that can be missed at the higher levels. However, it requires more computational resources and may not be suitable for real-time applications such as driver monitoring systems and emotion recognition in video conferencing.

2.2.2. Facial Sentiment Analysis Techniques

Computer vision and machine learning algorithms are typically employed in FSA approaches to analyse facial expressions and extract emotional information from human faces. Various methodologies are applied in these techniques, such as facial landmark identification to identify significant facial characteristics, facial expression recognition to classify emotions, and intensity estimate to quantify the strength of displayed emotions. Deep learning models, such as CNNs, RNNs, or hybrid architectures, are commonly used in these methodologies. These algorithms are trained on massive datasets of labelled facial expressions to understand patterns and generate reliable predictions. Additionally, some techniques analyse other facial characteristics such as eye movements, head pose, or micro-expressions to improve sentiment analysis accuracy. With advances in computer vision and machine learning, FSA techniques are evolving, allowing applications in domains as diverse as market research, customer service, education, and human–computer interaction. Several techniques can be applied at each level of FSA. Some of the commonly used techniques are given below in

Table 4. Facial sentiment analysis techniques used at different levels.

Technique	Description	Level	Reference(s)
Viola–Jones Algorithm	Detects faces using Haar-like features	Face level	[76,77,78]
Eigenfaces	Projects face images into a lower-dimensional space and uses Principal Component Analysis (PCA) to classify emotions	Face level	[79,80]
Local Binary Patterns (LBPs)	Texture descriptor that extracts information about local patterns of pixel intensities	Face level	[81,82,83]
Fisherfaces	Projects face images into a lower-dimensional space and uses Fisher discriminant analysis (FDA) to classify emotions	Face level	[84]
Convolutional Neural Networks (CNNs)	Multi-layer neural networks that can automatically learn features for classifying emotions	Face level	[85,86]
Active Shape Models (ASMs)	Statistical models of the shape and appearance of objects, used to detect facial features	Landmark level	[87,88]
Active Appearance Models (AAMs)	Extension of ASMs that also models texture information to track facial expression changes	Landmark level	[89,90]
Constrained Local Models (CLMs)	Combines an ASM with a texture model to track facial expressions and improve accuracy	Landmark level	[91]
Facial Action Coding System (FACS)	System for analysing and describing facial expressions based on the activation of individual muscles	Landmark level	[92,93]
Histograms of Oriented Gradients (HOGs)	Descriptor that extracts information about the distribution of gradient directions in an image	Region level	[94]
Scale-Invariant Feature Transform (SIFT)	Descriptor that extracts features invariant to scaling, rotation and translation	Region level	[95,96]
Speeded Up Robust Features (SURF)	Descriptor similar to SIFT but faster and more robust to changes in image scale and orientation	Region level	[97,98]

.

2.3. Other Modalities

While our work primarily focuses on text-based and facial sentiment analysis, it is indeed important to acknowledge that other physiological modalities, such as EEG and heart rate monitoring, have also been utilised to assess students’ cognitive and emotional states [99,100]. These modalities can also provide direct and objective indicators of students’ emotional and mental processes and complement the information obtained from facial expressions and text. Therefore, it is important to consider the potential benefits of using multiple physiological modalities for sentiment analysis and engagement detection in education.

EEG and heart rate monitoring-based methods help measure students’ emotions and engagement because they can capture the changes in their brain activity and physiological arousal associated with different emotional states and cognitive load levels. The EEG contains a broader range of information about a subject’s mental state concerning other biosignals [9,101]. It measures various parameters that reflect the cognitive and emotional engagement of students. Similarly, heart rate monitoring can measure the variations in students’ heartbeats that indicate their stress, excitement, boredom, or frustration. When combined with facial expressions and text, these methods might offer a good amount of data, which may be influenced by students’ self-reporting bias, social desirability, or cultural differences. Moreover, these methods can offer teachers and students more continuous and real-time feedback, which can help them adjust their teaching and learning strategies accordingly. Therefore, EEG and heart rate monitoring-based methods can benefit sentiment analysis and engagement detection in education.

Our work complements the existing literature by providing a comprehensive overview of sentiment analysis in online STEM learning. While focusing on textual and facial sentiment analysis, we recognise the value of integrating physiological measures. Future research could combine sentiment analysis with EEG data to better understand emotional and cognitive engagement processes. By encouraging interdisciplinary exploration, these approaches promise richer insights into student experiences, enhancing online learning environments and enriching discussions on improving STEM engagement.

3. Method

We conducted a systematic literature review using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) technique [102]. PRISMA is a widely recognised and acknowledged guideline for comprehensive reporting of systematic reviews and meta-analyses. Following PRISMA guidelines, the systematic literature review consists of two main stages: (i) the planning stage, which includes search and inclusion/exclusion strategies of research articles; and (ii) data extraction and analysis. These approaches provide a holistic view of the research landscape through different perspectives and form relationships between the diverse layers of multifaceted research questions provided in Section 1.

3.1. Search Strategy

A search was conducted using the terms “Sentiment Analysis”, “Text-Based Sentiment Analysis”, “Facial Sentiment Analysis”, and “Sentiment Analysis in Online Learning/Education” to retrieve different studies on the use of sentiment analysis in online learning. A thorough search was performed, retrieving data from various databases such as ACM, ScienceDirect, IEEE Xplore, Medline, Scopus, and Google Scholar, resulting in time-relevant sources from 2015 to 2023. The logical operators “OR” and “AND” were used to identify duplicates. The search was based on the metadata (e.g., title, abstract, and keywords). We tried certain synonyms as well to cover all possible content.

3.2. Inclusion/Exclusion Criteria

The authors scrutinised the returned articles and excluded surveys, short papers without proper explanation, not-peer-reviewed articles, pre-prints, and articles not written in English. The keywords listed above resulted in articles and technical reports published in various publication databases. The database searches generated a total of 236 articles. After initial scrutiny based on the articles’ relevance and state-of-the-art methodologies used, 57 articles were selected and categorised into three categories, i.e., sentence-based sentiment analysis, aspect-based sentiment analysis in online learning/education, and facial sentiment analysis in online learning. Figure 4 depicts the inclusion/exclusion criteria based on the PRISMA technique.

4. Results and Analysis

This section explores, analyses, and provides the answers to the study research questions outlined in Section 1, Table 1. The solutions provided are based on the data available in the current studies and the authors’ analysis of each research question. The studies included in the investigation have utilised diverse datasets collected from various sources such as social media posts, online forums, and learning management systems (LMSs). These datasets encompass a range of educational subjects and grade levels. However, the studies have yet to specifically examine the generalisation of sentiment analysis algorithms to different academic domains and grade levels. Most research has focused on performance within a particular subject area or grade level. The ability of sentiment analysis models to maintain accuracy when applied to new subjects and grade groups remains an open question that could be explored in future work. The section is structured into four distinct subsections, addressing each research question. We start by focusing on text-based sentiment analysis, showcasing sentence-based and aspect-based sentiment analysis studies, highlighting the datasets, employed techniques, and resultant findings. Moreover, in the second subsection, the challenges and limitations encountered when capturing learners’ emotions through text-based sentiment analysis are addressed, and a summary is provided in the form of a table. Afterwards, the utilisation of FSA in the educational domain is addressed in the third subsection, and an overview of the relevant studies is provided, presenting the methodology, extracted emotion classes, and resultant findings. Finally, the fourth subsection discusses the challenges and limitations associated with capturing learners’ emotions using FSA. Different aspects, including cultural differences in facial expressions and dependence on lighting and camera quality, are also explored. Overall, this section will provide readers with insights from the authors about text-based SA and FSA in the educational domain.

4.1. RQ1: Use of Text-Based Sentiment Analysis in Educational Domain

Text-based sentiment analysis techniques have played a significant role in online learning environments, providing valuable insights into students’ emotions and sentiments expressed through text. NLP and machine learning algorithms are used in these techniques to analyse and categorise text documents based on their sentiment, which can be positive, negative, or neutral. To extract sentiment information from online learning materials, discussion forums, social media platforms, and student feedback, approaches such as bag-of-words, sentiment lexicons, and machine learning models such as NB, SVM, and RNN have been used.

The authors analysed the current literature for text-based emotion classification of learners’ responses to this research question. Moreover, we have identified the datasets and techniques used for text-based sentiment analysis in the educational domain. Between 2015 and 2023, 37 papers were published on sentiment analysis of text. Of these, 25 papers focused on analysing sentiment at the sentence-level, while the remaining 12 concentrated on analysing sentiment at the aspect-level.

Table 5. Sentence-based sentiment analysis studies in educational domain.

S. No	Article Title	Published Year	Article Type	Dataset/Sample Size	Study Methodology	Emotion Classes	Results/Findings
1	Variants of Long Short-Term Memory for Sentiment Analysis on Vietnamese Students’ Feedback Corpus [103]	2018	Conf.	16,175 sentences from students’ feedback	LSTM, Dependency Tree-LSTM (DT-LSTM), L-SVM, D-SVM, and LD-SVM, NB	Positive, negative, neutral	LD-SVM: Negative—92.52, Neutral—43.37, Positive—93.06, Accuracy—90.20, F1 score—90.74
2	Opinion mining and emotion recognition applied to learning environments [104]	2020	Journal	—	EvoMSA, Multinomial NB, KNN, BERT, SVC, Linear SVC	sentiTEXT: (positive and negative), eduSERE (engaged, excited, bored, and frustrated)	Accuracy: 93 percent sentiTEXT, and 84 percent eduSERE
3	What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach [105]	2020	Journal	249 randomly sampled MOOCs and 6393 students’ perceptions of these MOOCs	Boosting tree model with TextBlob3	Positive, negative, neutral	F1 score: structure (0.7780), video (0.8832), instructor (0.8570), content and resources (0.7625), interaction and support (0.8375), assignment and assessment (0.8138)
4	Sentiment analysis on massive open online course evaluations: A text mining and deep learning approach [106]	2020	Journal	66,000 MOOC reviews	Machine learning, ensemble learning methods, and deep learning methods with Word2Vec embedding	Positive, negative	Accuracy of 95.80 percent
5	Social Network and Sentiment Analysis: Investigation of Students’ Perspectives on Lecture Recording [107]	2020	Journal	1435 students reacted to Facebook question via emojis, 220 likes and 65 comments were generated from 150 unique students	Google Natural Language API	Positive, negative, neutral	Sentiment score: positive (39.4 percent), negative (33.3 percent), neutral (27.3 percent)
6	Investigating Learning Experience of MOOCs Learners Using Topic Modeling and Sentiment Analysis [108]	2021	Conf.	8281 reviews scraped from five courses within the field of data science are analysed from Coursera	Topic modelling (LDA) with VADER for sentiment analysis	Positive, negative, neutral	Sentiment score: positive, 67.9; negative, 17.4; neutral, 14.7
7	Sentiment Analysis and Topic Modeling on Tweets about Online Education during COVID-19 [109]	2021	Journal	Twitter dataset containing 17,155 tweets about e-learning	TextBlob, VADER, and SentiWordNet—For comparison: SVM, LR, DT, RF, SGD, KNN, GNB, CNN, LSTM, CNN-LSTM, and Bi-LSTM	Positive, negative, neutral	SVM achieves 0.95 accuracy using TF-IDF with SMOTE
8	Sentiment Analysis for Distance Education Course Materials: A Machine Learning Approach [110]	2020	Journal	6059 feedbacks	Machine learning techniques (decision tree, MLP, XGB, SVC, multinomial logistic regression, Gaussian NB, and k-neighbours)	Positive, negative, neutral	Highest accuracy: LR (0.775)
9	E-learning course recommendation based on sentiment analysis using hybrid Elman similarity [111]	2023	Journal	10,000 tweets, short texts, and comments from social websites	Feature extraction: TF-IDF, Word2Vec, hybrid N-gram Classification: Elman minimal redundancy maximum relevance model and enhanced aquila optimisation (EMRMR_EAO) model	Positive, negative, neutral	Accuracy: 99.98 percent
10	Emotional Variance Analysis: A new sentiment analysis feature set for Artificial Intelligence and Machine Learning applications [112]	2023	Journal	37 individual DEEP students journals	Emotional variance analysis	Positive, negative, neutral	Accuracy: 88.7 percent
11	Deep-learning-based user experience evaluation in distance learning [113]	2022	Journal	160,000 tweets	LSTM with Word2Vec embedding	Positive, negative, neutral	Accuracy: 76 percent
12	AOH-Senti: Aspect-Oriented Hybrid Approach to Sentiment Analysis of Students’ Feedback [114]	2023	Journal	—————	SVM, MNB, LR, RFC, DTC, and KNN	Positive, negative, neutral	98.7 percent aggregate accuracy using the RFC algorithm
13	Language to Completion: Success in an Educational Data Mining Massive Open Online Class [115]	2015	Conf.	320 students, 50 words in discussion	NLP tools (WAT, TAALES, TAAS)	Positive, negative	Accuracy: 67.8 percent, F1 score: 0.650
14	A Learning Analytics Methodology for Detecting Sentiment in Student Fora: A Case Study in Distance Education  [116]	2015	Journal	64 students, 371 messages	NioSto opinion word extraction algorithm	Positive, negative, neutral	27.27 percent positive, 55.56 percent neutral, and 17.17 percent negative
15	An Enhanced Decision Support System through Mining of Teachers Online Chat Data [117]	2018	Journal	6650 in-service K12 academics in China had participated, in 17,624 distinctive posts	Single-label naïve mathematician classification rule	Positive, negative, neutral	Classified: technical description (961), technical analysis (1638), technical critique (2235), personal description (613), personal analysis (5875), and personal critique (1166)
16	A machine-learning-based approach for sentiment analysis on distance learning from Arabic Tweets [118]	2022	Journal	Twitter dataset, 14,000 tweets	Logistic regression model	Positive, negative, neutral	Accuracy, F1 score, precision, and recall, obtaining scores of 91 percent, 90 percent, 90 percent, and 89 percent, respectively
17	Analysis of Student Feedback using Deep Learning [119]	2019	Journal	—-	CNN, SVM with Word2Vec	Positive, negative, neutral	—-
18	Lexicon-Based Sentiment Analysis of Teachers’ Evaluation [120]	2016	Journal	1748 students’ feedback	Knime	Positive, negative, neutral	Accuracy: 91.2 percent
19	Sentiment Analysis of Student Feedback Using Machine Learning and Lexicon Based Approaches [121]	2017	Conf.	1230 comments extracted from our institute’s educational portal	TF-IDF, N-grams with SVM and RF	Positive, negative, neutral	Accuracy: 0.93, F-measurement: 0.92
20	Improving international attractiveness of higher education institutions based on text mining and sentiment analysis [122]	2018	Journal	1938 reviews from 65 different business schools	NLP	Positive, negative, neutral	The satisfaction of the students towards HE institutions is significantly varied and depends on the topic being discussed in their opinions shared online
21	Student Feedback Sentiment Analysis Model using Various Machine Learning Schemes: A Review [123]	2019	Journal	950 posts	Multinomial naive Bayes (MNB), stochastic gradient descent, SVM, random forest and multilayer perceptron (MLP)	Positive, negative, neutral	83 percent, 79 percent, 80 percent, 72 percent, and 83 percent for classifier MNB, SGD, SVM, random forest, and MLP
22	Sentiment mining in a collaborative learning environment: capitalising on big data [124]	2019	Journal	12,300 tweets, 10,500 Facebook comments, and 8450 Moodle feedback messages	NB and SVM	Positive, negative, neutral	SM approaches can be used to understand students’ sentiment in a collaborative learning environment
23	Learning in massive open online courses: Evidence from social media mining [125]	2015	Journal	402,812 tweets	Opinion finder tool and social media mining approaches	Positive, negative, neutral	Social media SA provide a comprehensive understanding of MOOC learning trends
24	Sentiment Analysis of Students’ Comment Using Lexicon Based Approach [126]	2017	Conf.	Sentiment word database: 745 words	Lexicon-based approach	Strongly positive, moderately positive, weakly positive, strongly negative, moderately negative, weakly negative, or neutral.	8.5 + (−2.5) + 6 = 12 by (5) and divided by total number of opinion words in all comments. The result is 12/9 = 1.3333
25	Student Opinion Mining regarding Educational System using Facebook group [127]	2017	Conf.	Comments of master’s students from the Facebook Academic group: 250 comments	Bayesian network probabilistic model	Positive, negative, neutral	Sentiment score, we have found 56 percent positive, 32 percent neutral, and 12 percent negative comments

illustrates the studies on sentence-based sentiment analysis, including the datasets utilised, techniques employed, extracted emotion classes, and the resultant findings.

Table 6. Aspect-based sentiment analysis studies in educational domain.

S. No	Article Title	Published Year	Article Type	Sample Size	Study Methodology	Word Embedding	Aspects Extracted	Results/Findings
1	Aspect-Based Opinion Mining on Student’s Feedback for Faculty Teaching Performance Evaluation [128]	2019	Journal	Dataset constructed from the last five years of students’ comments from Sukkur IBA University as well as on a standard SemEval-2014 dataset	Two-layered LSTM model	Academic Domain, OpinRank, Glove.6B.100D	Teaching pedagogy, behaviour, knowledge, assessment, experience, general	Aspect extraction (91 percent) and sentiment polarity detection (93 percent)
2	Weakly Supervised Framework for Aspect-Based Sentiment Analysis on Students’ Reviews of MOOCs [129]	2020	Journal	105 k students’ reviews collected from Coursera and a dataset comprising 5989 students’ feedback	LSTM, CNN	FastText, GloVe, Word2Vec, MOOC	Course, instructor, assessment, technology	F1 score: weakly supervised LSTM (domain embedding: 92.5, GloVe: 93.3), weakly supervised CNN (domain embedding: 90.1, GloVe: 91.5)
3	Aspect-Based Sentiment Analysis of Arabic Tweets in the Education Sector Using a Hybrid Feature Selection Method [130]	2020	Conf.	7943 Arabic tweets related to Qassim University in KSA	SVM	One-way ANOVA	Teaching, environment, electronic services, staff affairs, academic affairs, activities, student affairs, higher education, miscellaneous	F-score: aspect detection 60 percent (0.76)
4	Multi-Attention Fusion Modeling for Sentiment Analysis of Educational Big Data [131]	2020	Journal	Education dataset: 5052; course dataset: 705	TD-LSTM, AE-LSTM, ATAE-LSTM, IAN, RAM	——————–	Difficulty, content, practicality, and teacher	Education dataset: Multi-AFM: 94.6; course dataset: Multi-AFM: 81.4
5	Aspect-Based Opinion Mining of Students’ Reviews on Online Courses [132]	2020	Conf.	21 thousand manually annotated student reviews that are collected from Coursera	1D-CNN, decision tree, naïve Bayes, SVM, boosting	FastText, GloVe, Word2Vec, own dataset	Instructor, structure, content, design, general	FastText: precision—86.78, recall—89.52, F1 score—88.13; Word2Vec: precision—87.08, recall—89.34, F1 score—88.20; GloVe: precision—86.75, recall—88.89, F1 score—87.81; Own dataset: precision—86.70, recall—89.54, F1 score—88.10
6	Online Reviews Evaluation System for Higher Education Institution: An Aspect Based Sentiment Analysis Tool  [133]	2017	Conf.	Twitter and Facebook data	Apache OpenNLP, Stanford NLP library	POS (part of speech) tags	Opinions about institution touch upon many intrinsic aspects and qualities and analysing each of these aspects	Accuracy of 72.56 percent
7	Aspect-based Sentiment Analysis for Improving Online Learning Program Based on Student Feedback [134]	2022	Conf.	162 new graduates from BINUS’s (Bina Nusantara University) online program	Stanford NLTK library	AFINN standard polarity of English words for each token with noun POS tagging	Cheating punishment, class facilities, college management, learning material, learning guide, education system benefit, happy learning experiences	Successful classification of aspects in positive, negative, and neutral sentiment classes
8	Knowledge-enabled BERT for aspect-based sentiment analysis [135]	2021	Journal	MOOC offerings on two Chinese university MOOC platforms 9123 posts by 7590 different online learners in different advanced language programming course	KNEE, CG-BERT, R-GAT+BERT, BERT+Liner	SKG	—-	BERT + SKG model outperforms all the baseline methods in accuracy and macro-F1 accuracy < 0.80, macro-F1 0.75
9	Aspect-based Sentiment Analysis for University Teaching Analytics [136]	2022	Journal	Two different surveys: (i) COVID-19 specific student survey (1805); (ii) semester-based student course evaluations (9348)	TextBlob, NLTK, spaCy package and flair	—-	Flexibility, teaching, pace, misc, technology, motivation, information	Findings reveal that students disliked online teaching due to insufficient information and unadjusted teaching methods. However, students liked its flexibility and possibility to learn at an individual pace.
10	Weakly Supervised Models of Aspect-Sentiment for Online Course Discussion Forums [137]	2015	Conf.	MOOC dataset of m different disciplines (business, technology, history, and the sciences)	Joint probabilistic model (PSL-Joint)	Seed words and weighted logical rules	Lecture, quiz, certificate, social	3–5 times improvement in F1 score in most cases over a system using only seeded LDA
11	Aspect based sentiment analysis of students opinion using machine learning techniques [138]	2017	Conf.	2000 Tweets	Naive Bayes (NB), complementary naive Bayes (CNB), and PART algorithm	—-	Teaching, placement, facilities, fees, sports, organising events, transport	PART algorithm precision: POS (1), Neg (1); recall POS (1), Neg (0.994); F-measure POS (1), Neg (0.997)
12	Triggers and Tweets: Implicit Aspect-Based Sentiment and Emotion Analysis of Community Chatter Relevant to Education Post-COVID-19 [139]	2022	Journal	Twitter chat data	Linear support vector classifier (SVC), logistic regression, multinomial naïve Bayes, random forest	TF-IDF-BoW	Safety, education quality and educational right, financial security	ASBA: logistic regression (81 percent), overall SA: linear SVC (91 percent)

shows the aspect-based sentiment analysis studies published using educational datasets while using different techniques.

4.2. RQ2: Challenges/Limitations to Capture Learners’ Emotions Using Text-Based Sentiment Analysis

There has been a growing interest in using sentiment analysis to capture learners’ emotions and improve the efficacy of educational systems in recent years. Text-based sentiment analysis, which can automatically analyse enormous volumes of data from multiple sources such as social media, discussion forums, and chat logs is a potential strategy for achieving this goal. However, accurately detecting and interpreting learners’ emotions from textual data poses significant challenges and limitations.

The authors investigated some of the challenges and limitations characterised in recent studies and provided suggestions for possible solutions. A comprehensive review of the relevant literature, including recent articles on sentiment analysis in education, supports our findings. Some of the significant challenges are shown in

Table 7. Text-based sentiment analysis challenges in capturing learners’ emotions.

S. No	Challenge(s)	Reference(s)
1	Classification of students’ textual utterances	[140]
2	Emotion classes overlapping (classes ambiguity)	[141,142,143]
3	Dealing with bipolar words	[144]
4	Fake comments/responses	[145,146]
5	Lack of reliable ground truth data for training and evaluation	[147]
6	Difficulty of accurately identifying and interpreting sarcasm and irony in chat-based data	[148]
7	Unstructured data	[149,150,151]

and briefly discussed below:

4.2.1. Classification of Students Textual Utterances

Student utterances are spontaneous words such as “Wow! that topic was interesting” while performing a task or providing feedback about a specific event [140]. Student utterances are considered one of the effective ways of recording student feedback about the pedagogical activity and helping to understand students’ emotions [152]. However, recording and classifying students’ utterances are challenging tasks. These utterances are usually very short and need to provide more factual information. For textual utterances, chat data from an online learning platform has been used to detect emotion using text-based sentiment analysis techniques for emotion detection [153]. Several studies used automatic chatbots to record and analyse student chat data using lexicon-based (e.g., corpus-, dictionary-based approaches) and machine learning techniques (rule-based and linear classifiers) [154,155]. These sentiment analysis techniques (lexicon-based/machine learning approaches) provide substantial results in emotion detection. However, they require further enhancement to provide an error-free classification of utterances in emotion classes (i.e., happy, sad, angry).

4.2.2. Emotion Classes Overlapping

Class imbalance and overlapping are well-known classification challenges in machine learning that have piqued academics for over a decade. The classes overlapping problem usually occurs due to ambiguous meanings/regions in the data where data qualifies for more than one class based on its reasonable probability [156]. Minor changes between samples of two different classes are typically challenging to capture solely using data characteristics suggested by domain experts. Unbalanced learning has recently received extensive attention from the scientific community, resulting in several dedicated approaches and algorithms such as synthetic minority oversampling technique (SMOTE), and overfitting and underfitting techniques [157]. The research community previously neglected the issue of unbalanced data in the context of sentiment classification since most datasets analysed were deliberately designed to have an equal amount of positive and negative reviews [158]. However, where good ratings often outnumber negative ones, the balanced data assumption does not apply. The authors in [159] were among the first to recognise the issue and demonstrate that utilising realistic imbalanced datasets resulted in classifier construction that performed significantly better in reality. SMOTE was recently applied to over-sampled text representations built by a recursive neural tensor network to utilise an imbalanced dataset for emotion classification [160]. Class overlapping of different emotions makes it challenging to classify learner emotions accurately, and learners’ sentences such as “Oh, I made a mistake” depict characteristics of “sad” and “angry” classes, which become difficult for the machine learning algorithm to classify.

4.2.3. Dealing with Bipolar Words

Sentiment analysis is the field of classification that extracts polarity and classifies them into different sentiment classes (i.e., happy, sad, angry, etc.). When two words with conflicting polarities are combined, they produce an overall contradiction that can exhibit positive and negative behaviour, known as bipolar phrases. Considering the phrase “passing marks”, which generally has a positive polarity, it may also be negative regarding student progress. Salas-Zárate et al. [161] worked in the health domain and used N-grams to find the polarity of aspects by using aspect-level sentiment analysis techniques. They used Twitter data using N-grams, resulting in 81.93% precision in sentiment classification into different classes. Another study used “SentiWordNet” to handle bipolar words in sentiment analysis to analyse statistical techniques for affective computing as knowledge [162]. Bipolarity is a common problem in learners’ sentiment analysis as the learners’ feedback consists of double-meaning words. For example, a learner feedback sentence, “The exam was quite easy, but I was short of time to attempt all questions”, is a bipolar sentence and conventional machine learning algorithms to classify emotions underperform with such sentences. Another problem with learners’ sentiment classification is the confusing sentences as learners express their feelings about any event or topic of discussion. Therefore, we expect an unstructured answer/response consisting of certain bipolar words, such as “challenging”, that must be adequately addressed before classification into emotion classes.

4.2.4. Fake Comments/Responses

One of the limitations of existing student response systems (SRSs) is fake responses from the learners during course/teaching evaluation. Most SRSs are predesigned with objective questions such as “rate the instructor’s teaching methodology (1 to 5)” about the teaching content and methods and their satisfaction with both. Learners select the first available option to answer those questions due to their lack of interest and motivation. Educational institutes rely on these responses to evaluate a course or instructor’s performance. Several studies worked on identifying fake responses in terms of fake news and fabricated product reviews. E. Tacchini et al. [163] used a dataset of 15,500 posts from 909,236 users to detect fake news on Facebook. They used the logistic regression (LR) and Boolean label crowdsourcing algorithm (BLC) techniques for fake news detection, resulting in 94.3% and 90% detection accuracy, respectively. S. Aphiwongsophon [164] discussed the impact of fake news in today’s technologically advanced society while considering that detecting fake news is a critical task. Their study aimed to identify fake news by using machine learning classifiers using three different methods: NN, SVM, and NB. Fake news detected by NB showed an accuracy of 96.08%. Fake news detected by NB showed an accuracy of 96.08%. At the same time, the SVM and the NN have an accuracy of 99.90% for detecting fake news. All of the above studies worked on fake news detection; however, detecting fake responses from the learners while providing feedback needs substantial work.

4.2.5. Lack of Reliable Data for Training and Evaluation

Researchers have pointed out certain challenges that sentiment analysis models face regarding chat-based data. One of the critical challenges they identified is the need for labelled data, which can significantly impact the accuracy of the models. Labelled data is essential for training supervised machine learning models to classify text data into sentiment categories accurately. However, in chat-based scenarios, acquiring labelled data can be challenging due to the dynamic nature of conversations and the need for human annotators to label the data manually. Moreover, the quality of the labelled data can also impact the performance of the models. Patil and Nandi [147] emphasised that the labelled data should be diverse, representative of the target population, and annotated by experts to ensure accuracy and reliability.

There is a growing need to collect and utilise real-time utterances data from learners to address these challenges. Real-time data can capture the context of the conversations, the dynamic nature of the interactions, and the changes in the sentiment over time. This data can then be used to train sentiment analysis models that are more accurate and robust. However, the collection and analysis of real-time data also pose significant ethical and privacy concerns, which must be addressed appropriately. In conclusion, the scarcity and quality of labelled data remain a significant challenge for sentiment analysis models, especially in chat-based scenarios. Collecting real-time utterances data from learners can help to overcome these challenges and improve the accuracy and effectiveness of sentiment analysis models in educational contexts.

4.2.6. Identification of Sarcasm and Irony in Chat Data

Student chat can often include sarcasm and irony, which pose significant challenges for sentiment analysis. As noted by the authors of [148], online discussions frequently employ linguistic terms, such as acronyms/abbreviations and emoticons, and failing to account for them can lead to inaccurate sentiment analysis results. The difficulty is compounded by the absence of contextual information in chat-based data, making it challenging to identify when a message is sarcastic or ironic. This lack of context can result in sentiment analysis models misclassifying messages and failing to capture students’ nuanced emotions and attitudes in educational settings. Therefore, sentiment analysis models need to account for the impact of sarcasm and irony in student chat to improve the accuracy of sentiment analysis in educational contexts. Several studies have explored the impact of sarcasm and irony on sentiment analysis and considered it one of the main challenges in text-based sentiment analysis, such as in [165].

4.2.7. Unstructured Data

Online learning platforms create massive amounts of unstructured data, such as student essays, forum posts, and social media comments. Incorporating these data sources into sentiment analysis models has the potential to provide a more comprehensive understanding of student emotions. However, as the authors in [149] pointed out, this approach faces various obstacles, including data cleansing and integration. The unstructured nature of the data can also make it challenging to distinguish emotions and attitudes appropriately.

Other studies have pointed out the drawbacks of using unstructured data for sentiment analysis. For example, the authors in [150] stated that the variety of language use and context complexity might lead to sentiment analysis models delivering inaccurate or unreliable results. Furthermore, the authors of [151] stated that the subjective nature of sentiment analysis makes establishing a criterion for evaluating model accuracy challenging.

Despite these challenges, sentiment analysis of unstructured data in educational contexts remains a promising area of research.

4.3. RQ3: Use of Facial Sentiment Analysis in Educational Domain

FSA techniques have emerged as a beneficial tool in online learning environments, allowing for analysing students’ emotions and sentiments based on facial expressions. Using computer vision and machine learning algorithms, these techniques identify and analyse facial cues, such as changes in facial muscle movements, to infer emotional states, such as happiness, sadness, confusion, or engagement. To analyse video recordings or real-time video interactions in online learning platforms, methods such as facial landmark detection, facial emotion recognition, and deep learning models have been used. Using FSA, educators and researchers can gather insights about students’ emotional responses, level of engagement, and comprehension. This data can be utilised to improve teaching tactics, give personalised assistance, and create more inclusive and effective online learning environments. As technology advances, FSA algorithms evolve, opening new avenues for improving online learning environments and student outcomes.

The authors comprehensively reviewed existing literature on classifying learners’ facial emotions in this research question. Several research studies have been conducted to investigate FSA in educational settings. Our findings revealed that nine papers were published between 2015 and 2023, focusing on sentiment analysis of facial expressions.

Table 8. Facial sentiment analysis studies in educational domain.

S. No	Article Title	Published Year	Article Type	Dataset/Sample Size	Study Methodology	Emotion Classes	Results/Findings
1	Modeling adaptive E-Learning environment using facial expressions and fuzzy logic [166]	2020	Journal	Corpora of 12 learners contain 72 learning activities and 1735 data points of distinct emotional states	CNN	Anger, disgust, fear, happiness, sadness, surprise, neutral	Proposed approach provides adaptive learning flows that match the learning capabilities of all learners in a group
2	Use of facial emotion recognition in E-learning systems [167]	2017	Journal	Frontal face images of participants recorded through Skype, size of 11,680 × 10	kNN, random forest, CART, SVM	Happiness, fear, sadness, anger, surprise, and disgust	Highest accuracy: SVM 98.24
3	An E-learning System With Multifacial Emotion Recognition Using Supervised Machine Learning [168]	2015	Conference	Yale Face Database (YFD) for training and Face Detection Dataset and Benchmark (FDDB) and Labelled Faces in the Wild (LFW) for evaluation	SVM	7 major emotions	Accuracy of 89 to 100% with respect to different datasets
4	Mood Extraction Using Facial Features to Improve Learning Curves of Students in E-Learning Systems [169]	2016	Journal	Cohn–Kanade AU-coded facial expression database consists of 486 sequences from 97 faces	Radial basis function NN algorithm, SVM	Happy, sad, confused, disturbed, surprised	Proposed algorithm showed a success rate of over 70% in assessing the student’s mood
5	Non-intrusive Identification of Student Attentiveness and Finding Their Correlation with Detectable Facial Emotions [170]	2020	Conference	Dataset of the raw images consisted of 3500 images	CNN	Attentiveness, calm, confused, disgusted, fear, happy, sad, surprised	93% accuracy
6	Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order [171]	2017	Journal	Extended Cohn–Kanade (CK þ), Japanese Female Facial Expressions (JAFFE), and Binghamton University 3D Facial Expression (BU-3DFE) database	CNN	Angry, disgust, fear, happy, sad, and surprise	96.76% accuracy on the CKþ database
7	Toward Automated Classroom Observation: Predicting Positive and Negative Climate [172]	2019	Conference	241 class-labelled videos	CNN and Bi-LSTM	Positive climate and negative climate	Accuracy: 0.40 and 0.51, respectively
8	Model Proposal on The Determination of Student Attendance in Distance Education with Face Recognition Technology [173]	2021	Journal	Face gestures captured through LMS camera	Eigenfaces recognition algorithm with Gaussian filters	More than 80% accuracy achieved
9	A New Deep Learning Model for Face Recognition and Registration in Distance Learning [174]	2022	Journal	Faces94, Faces95, Faces96, and Grimace datasets contain 7873 images	CNN	Eigenfaces emotions	Accuracy of 100% for the Faces94 and Grimace datasets and achieves 99.86% for Faces95. Faces96 model achieves accuracy of 99.54%
10	Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks [175]	2019	Journal	8000 single face in a single image frame and 12,000 multiple faces in a single image frame	Hybrid CNN	Engaged, boredom, and neutral	Accuracy of 86% and 70% for posed and spontaneous affective states of classroom data, respectively.

provides an overview of these studies, presenting the techniques utilised, the emotion classes extracted, and the resulting findings.

4.4. RQ4: Challenges/Limitations to Capture Learners’ Emotions Using Facial Sentiment Analysis

The utilisation of FSA has emerged as a highly promising method for capturing the emotions of learners. This approach, however, has several major challenges. One major challenge is the accurate recognition and interpretation of facial expressions, as emotions can be subtle and context-dependent [176]. Additionally, variations in cultural norms and individual differences in facial expressions pose challenges to universal emotion recognition [177]. Furthermore, the presence of confounding factors such as occlusions, variations in lighting conditions, and the complexity of dynamic facial expressions can significantly impact the accuracy of FSA models [178].

Through a systematic literature analysis, several challenges have been identified in capturing learners’ emotions. These challenges encompass various aspects of the process and highlight the complexities of accurately capturing and interpreting learners’ emotional states. Firstly, we examine the limitations in accurately identifying emotions through facial analysis. We then delve into the challenge posed by cultural differences in interpreting emotions. The subsequent point highlights the limited effectiveness of FSA in detecting subtle emotions. Additionally, we address the influence of factors such as lighting conditions and camera quality on the analysis quality. Lastly, we explore the challenges such as with real-time analysis of emotions, primarily due to computational and processing limitations. These challenges are summarised in

Table 9. Facial sentiment analysis challenges in capturing learners’ emotions.

S. No	Challenge(s)	Reference(s)
1	Limited accuracy in identifying emotions	[179,180,181]
2	Cultural differences in facial expressions	[182,183,184,185]
3	Limited effectiveness in identifying subtle emotions	[186,187,188,189]
4	Dependence on lighting and camera quality	[190,191,192,193]
5	Lack of real-time analysis	[194,195]

and discussed in subsequent paragraphs.

4.4.1. Limited Accuracy in Identifying Emotions

Due to the complexity and subjectivity of emotions, face-gesture-based sentiment analysis models face significant challenges in accurately recognising emotions. Emotions are multifaceted and can be influenced by various external and internal factors, challenging their interpretation through facial expressions. Facial expressions do not always adequately reflect an individual’s true feelings [179]. Students, for example, may present facial expressions that may not correspond to their actual emotional states or exhibit emotions that differ from the expected facial expressions generally associated with those emotions.

Further research revealed new challenges in face-gesture-based sentiment analysis. Researchers [180] investigated the shortcomings of facial expression detection algorithms in recognising subtle emotional nuances. They discovered that these algorithms might struggle to detect subtle changes in facial expressions, potentially leading to errors in emotion recognition. In a study by Kaminska et al. (2021) [181], the authors evaluated the impact of cultural variances on the recognition of facial emotions. Their findings revealed that variations in facial expression across cultures could reduce the effectiveness of emotion identification models when applied across different cultural contexts.

4.4.2. Cultural Differences in Facial Expressions

Face-gesture-based sentiment analysis models encounter challenges in accounting for cultural differences in facial expressions, leading to inaccurate analysis of student emotions. Cultural differences in facial expressions have been identified as a critical aspect influencing the perception of emotions and recognition. Individuals from diverse cultural origins may express emotions via different facial expressions, making considering the cultural context in sentiment analysis essential. For example, in some cultures, nodding from left to right is often employed to show agreement or affirmation, similar to saying “yes”. While in Western societies, nodding from left to right usually indicates disagreement or negation, equivalent to saying “no”.

Li Y. et al. [182] evaluated the impact of culture on facial expression recognition and discovered substantial differences across cultures. Their findings emphasised the significance of combining cultural knowledge and context when developing accurate and culturally sensitive emotion identification systems. In another study, the authors explored the function of cultural background in emotion perception. They emphasised the importance of considering cultural differences to improve the reliability and validity of emotion analysis models [183].

Furthermore, Zhang et al. [184] explored cross-cultural facial emotion detection and discovered that cultural factors considerably influence the perception of facial emotions. Their findings highlighted the importance of tailoring FSA models to various cultural contexts to produce more accurate and reliable emotion recognition results.

In addition to cultural differences, other studies have identified challenges related to the dynamic nature of facial expressions. Wang et al. [185] highlighted the limitations of static facial analysis and emphasised the importance of considering temporal dynamics in emotion recognition models. They proposed incorporating temporal information to capture the evolving nature of facial expressions and improve the accuracy of sentiment analysis. Overall, these studies highlight the challenges FSA models face in accounting for cultural differences in facial expressions and effectively analysing student emotions.

4.4.3. Limited Effectiveness in Identifying Subtle Emotions

Face-gesture-based sentiment analysis algorithms encounter difficulties in recognising subtle emotions, particularly those familiar to online learning, such as boredom or confusion. These emotions are often conveyed through subtle facial expressions that might be challenging for automated systems to identify and analyse. The traditional facial expression recognition methods struggle to effectively capture and analyse these subtle emotions [186]. The studies conducted between 2015 and 2023 revealed new challenges in detecting subtle emotions using face-gesture-based sentiment analysis. For example, Wang Y. et al. [187] investigated the limitations of facial expression recognition algorithms in collecting micro-expressions such as narrowing of the eyes or slight wrinkling of the nose that shows anger or frustration, which are brief and temporary facial expressions that reflect hidden emotions. Their findings highlighted the need for more sophisticated techniques to detect and analyse these subtle emotional cues.

Furthermore, another study focused on the difficulties in recognising emotions such as confusion and frustration in the educational context [188]. The authors discovered that these emotions are frequently accompanied by slight changes in facial expressions, making them difficult to identify solely from visual clues. The findings emphasise the necessity of multimodal techniques integrating facial expressions alongside other modalities, such as voice, text, or physiological data, to improve emotion recognition accuracy.

In addition, another study investigated the effect of facial occlusions, such as face masks or partial obstructions, on emotion recognition [189]. They discovered that occlusions could considerably impact the performance of FSA models, especially when learners’ faces are not completely visible. Overcoming these obstacles requires the development of robust systems capable of handling occlusions while still effectively capturing and interpreting facial expressions.

In conclusion, face-gesture-based sentiment analysis algorithms encounter difficulties reliably recognising subtle emotions, especially those observed in online learning contexts. The complexity associated with effectively analysing these emotions is further complicated by limitations in capturing micro-expressions, recognising subtle changes in facial expressions, and accounting for facial occlusions. To address these issues, multimodal methodologies and more sophisticated ways of gathering and analysing subtle emotional cues must be investigated.

4.4.4. Dependence on Lighting and Camera Quality

Face-gesture-based sentiment analysis models’ accuracy can be affected by factors such as lighting quality and the camera used to collect facial emotions. According to a study, inadequate lighting or low-quality cameras might substantially impact the analysis of facial expressions and emotions, resulting in incorrect findings [190].

Further research examined the effect of lighting and camera quality on face-gesture-based sentiment analysis. For example, a study evaluated the impacts of varying lighting conditions on facial expression recognition [191]. The authors discovered that lighting variations could affect the visibility and intensity of facial characteristics, affecting the accuracy of emotion detection.

Furthermore, a research study investigated the impact of the camera quality on facial expression analysis [192]. The study discovered that low-resolution or noisy images from low-quality cameras might affect accurate emotion recognition. They emphasised the need for high-quality cameras to capture facial expressions to produce consistent and accurate outcomes.

Another study evaluated the effect of head positioning changes on facial emotion detection, lighting, and camera quality. They discovered that abnormal head positions could make capturing facial expressions difficult, resulting in lower performance of the sentiment analysis model [193].

These studies highlight the importance of considering lighting, camera quality, and head poster variations when designing and deploying face-gesture-based sentiment analysis models.

4.4.5. Lack of Real-Time Analysis

Face-gesture-based sentiment analysis models may face challenges in providing real-time analysis of student emotions and sentiment due to the time required for image processing and analysis. The computational challenges of analysing facial expressions might cause a delay in acquiring results, restricting these models’ ability to provide real-time feedback to students and instructors in online learning environments. Extensive research conducted has explored the issue of real-time analysis in face- gesture-based sentiment analysis. Deshmukh et al. [194] evaluated the real-time performance of facial expression recognition algorithms. They discovered that computational demands could slow down real-time analysis, especially when dealing with massive datasets or complicated algorithms. Furthermore, another study investigated real-time emotion recognition using deep learning approaches [195]. Due to the enormous processing requirements of deep learning models, their research emphasised the difficulties in achieving real-time analysis. These studies highlight the difficulties that face-gesture-based sentiment analysis algorithms encounter while delivering real-time analysis of student emotions. While advances in hardware and algorithms continue to improve analysis speed, computational complexity remains a crucial barrier to providing rapid feedback in online learning environments.

Considering the above-discussed text-based and facial sentiment analysis challenges, combining facial expressions, voice, written words, and physiological signals, including EEG signals, could help us better understand an individual’s emotions. These various perspectives on emotions can be combined to detect subtle sentiments such as irritation, boredom, or lack of interest. Combining different modalities could be challenging; however, combining multiple methods for analysing emotions correlates effectively with how complex human emotions are. Making better technology that understands emotions could aid in making educational resources more understandable and beneficial. They could provide personalised help tailored to each individual’s emotional and cognitive needs.

5. Conclusions and Future Directions

This PRISMA-based systematic literature review and meta-analysis provided valuable insights into applying text-based sentiment analysis and FSA in education, specifically STEM subjects, to assist learners and educators. Our review addressed four primary research questions regarding the application and challenges of sentiment analysis in online learning by reviewing time-relevant articles published between 2015 and 2023.

Regarding RQ1, the review revealed that text-based sentiment analysis had been frequently used in education to improve student’s learning experiences. Several research studies have shown that sentiment analysis can help with student engagement, feedback generation, and personalised learning. Textual data analysis allowed instructors to better understand their students’ emotions, perceptions, and learning requirements, allowing for more targeted interventions and assistance.

Regarding RQ2, the review identified several issues and limitations related to text-based sentiment analysis in the context of online learning. Significant challenges were identified, including the lack of labelled data, the impact of sarcasm and irony in text, and the requirement for real-time analysis of learners’ utterances. These difficulties demand further research and the development of new techniques to improve the accuracy and efficacy of text-based sentiment analysis in online learning environments.

Moving on to RQ3, the review shed light on the growing popularity of FSA in education and online learning. Studies have shown that analysing facial expressions can help us understand learners’ emotions, engagement levels, and cognitive processes. FSA provides valuable insights for adaptive learning systems, learner modelling, and affective computing, resulting in more personalised and successful educational experiences.

Finally, RQ4 investigated the challenges and limitations of FSA in online learning. Concerns raised in the review highlighted the accurate identification and interpretation of facial expressions, the impact of cultural variations, the detection of subtle emotions, and the influence of lighting and camera quality. These challenges highlight the importance of robust algorithms, standardised datasets, and cultural diversity considerations when implementing face-gesture sentiment analysis.

Overall, this systematic literature review provides an in-depth understanding of text-based and FSA applications, challenges, and limitations in the educational domain. The findings demonstrate the potential of sentiment analysis approaches to improve learning experiences and identify areas that require further study and development to address the identified limitations. By addressing these issues, sentiment analysis can continue to play a vital part in improving online learning environments and increasing student engagement, satisfaction, and academic performance.

Despite the insights presented by this review, certain important research gaps regarding sentiment analysis in online educational environments need to be addressed. Currently, no studies are exploring real-time sentiment analysis of chat data to provide interventions for learners. Most sentiment analysis models rely on post hoc analysis of textual data, but real-time analysis of chat could allow for dynamic adaptation of instruction and support. Integrating real-time facial expression or gesture recognition with chat sentiment analysis may further enrich the understanding of learner states. However, developing accurate real-time models that respond appropriately remains an open challenge. More research is required to create larger annotated datasets covering various educational contexts and can be used to train more robust sentiment analysis models. More research is also needed on multimodal sentiment analysis, which combines textual, visual, and auditory inputs to achieve greater precision.

Furthermore, more research needs to be conducted to investigate advanced neural network techniques such as transformers and how they might improve sentiment classification accuracy for informal textual data. Research must also explore the cultural nuances of showing and interpreting sentiments to increase the contextual understanding skills of sentiment analysis technologies. Such studies can provide more evidence on how these tools impact key outcomes such as learner motivation, engagement, and academic achievement. Ultimately, addressing these research gaps through interdisciplinary efforts will be vital to unlocking the full potential of sentiment analysis for providing personalised and emotionally aware learning on a large scale.