TasksZE: A Task-Based and Challenge-Based Math Serious Game Using Facial Emotion Recognition

Abstract

Serious games play a significant role in the teaching and learning process by focusing on educational objectives rather than purely on entertainment. By addressing specific educational needs, these games provide targeted learning experiences. The integration of emotion recognition technology into serious games can further enhance teaching and learning by identifying areas where students may need additional support, The integration of emotion recognition into a serious game facilitates the learning of mathematics by allowing the identification of emotional impact on learning and the creation of a tailored learning experience for the student. This study proposes a challenge-based and task-based math serious game that integrates facial emotion recognition named TasksZE. TasksZE introduces a novel approach by adjusting gameplay based on detected emotions, which includes real-time emotion analysis and the cross-validation of emotions. We conducted a usability evaluation of the game using the System Usability Scale (SUS) as a reference, and the results indicate that the students feel that TasksZE is easy to use, the functions are well integrated, and most people can learn to use it very quickly. The students answered that they would use this system frequently since they felt motivated by game attributes, rewards, and level progression. These elements contributed to a more engaging and effective learning experience.

1. Introduction

Math skills are important since they provide a framework for understanding, modeling, and solving problems related to everyday aspects of life, such as space, structure, and change [1]. Hence, mathematical reasoning is integrated into all levels of study throughout education. In basic education, there is an evident need to teach math through engaging content and using age-appropriate pedagogical skills [2,3]; otherwise, it can be perceived as a monotonous, difficult, or boring subject [4]. Major aspects influencing underperformance in math-related subjects include the teaching approach, lack of motivation, lack of previous knowledge, lack of understanding, and lack of confidence [5].

Developing new tools to support math fluency in modern educational environments is essential. This need has led to the growing popularity of both game-based learning and serious games. Game-based learning is an immersive approach that incorporates game elements into educational settings to boost motivation and learning [6]. Conversely, serious games (SGs) are designed primarily for educational purposes through engaging gameplay. These games are applicable across various fields, including the military, business, arts, sports, government, healthcare, and, notably, education [7]. Serious games have an educational purpose and serve to increase knowledge retention. They are used as educational and research tools in many educational environments [8]. With gamification in mind, serious games can integrate game-like elements and user experience to add educational value to the games themselves and digitally attract and motivate people to achieve their learning outcomes [9]. However, for serious games and gamification to be effective, it is important to analyze and understand which game technologies better suit each learning outcome [10,11].

The implementation of game technologies in education can drive significant changes in the classroom, from the way teachers approach their lessons to the way students are keener to learn to the way they all interact and remain engaged [12,13,14], ultimately contributing to knowledge retention while playing [15]. In game-based learning, emotion recognition can be integrated into serious games to enhance the learning experience by allowing learners and teachers to obtain real-time feedback on the learners’ emotions to better identify learning difficulties. This work describes the development of TasksZE, a serious game for math learning in basic education. The game uses facial emotion recognition to provide instructors feedback on students’ emotions during learning experiences, thus supporting the pedagogical decisions for math teaching, such as refining learning materials, redefining teaching approaches, and reinforcing practice.

This research proposes TasksZE: a task-based and challenge-based math serious game using facial emotion recognition. TasksZE implements a procedure to integrate facial emotion recognition in a serious game, the procedure comprises three phases: Game Analysis, Game Implementation, and facial emotion recognition implementation. This procedure allows incorporating facial emotion recognition, which is valuable for improving the user’s experience in the human–computer interaction, especially in education [16]. Recognizing emotions through facial cues enables more personalized and responsive systems to help identify emotional states, increase engagement, and provide tailored responses in educational environments [17]. TasksZE includes a layer-based software architecture that is independent of any programming language or technology. This research hypothesizes that the implementation of a novel approach by adjusting the gameplay based on detected emotions, including real-time emotion analysis and a cross-validation of these emotions, allows for the assessment of the emotional impact on learning by identifying the topics that are more difficult for the student to understand in a serious game and allows them to develop their reasoning skills by adjusting the gameplay to suit their abilities.

The remainder of this paper is structured as follows: Section 2 describes the state of the art regarding both the development of serious games and the implementation of facial emotion recognition in math serious games. Section 3 introduces the design of TasksZE, our math serious game based on facial and emotion recognition, while Section 4 describes its development. Section 5 presents a case study where TasksZE is used to teach fractions to a sixth-grade group. Section 6 discusses the evaluation of TasksZE and its results. Finally, Section 7 presents our conclusions and remarks for future work.

2. State of the Art

This section analyzes the current literature on serious game development. Based on the scope of this work, the reviewed literature can be categorized into two groups: (1) Serious games for math learning and (2) serious games for math learning based on facial emotion recognition.

2.1. Serious Games for Math Learning

Brauner et al. [18] presented an overview and metaperspective of serious games as a human-centered methodology. They discussed how serious games can have multiple purposes: (1) learning how industry operators react to complex situations and how they deal with incomplete, fuzzy, or confusing information; (2) understanding how operators of production systems and production networks can be supported by human-centric industrial user interfaces; and (3) enabling operators to act appropriately in relation to complex and dynamic cyber–physical production systems. The work concludes with the authors presenting an actionable, adaptable, and iterative process model for designing serious games and exemplifying this model using a supply chain and quality management system serious game. Gauthier et al. [19] proposed an Activity Theory-based Model of Serious Games (ATMSG) to provide a visual framework through which designers and researchers can explicitly map the gaming, learning, and instructional design of their learning game mechanics and game flow. The model was used to redesign Stop & Think, an existing learning game for math and science problem-solving. In turn, Díaz et al. [20] discussed the use of educational games across different chemical engineering subjects at graduate and undergraduate levels. The use of board games for the subjects Process Control and Particle Technology is illustrated and the combination of simple games and escape room activities can help to develop both low-level and high-level skills. Pan et al. [21] conducted a systematic review of the use of learning games for K-12 math education and identified the trends of implementing learning games in math education and the ways for designing and integrating math content in relation to gameplay. Kacmaz et al. [22] introduced a systematic review of math game studies and evaluated them concerning three characteristics: (a) the type of pedagogical foundations inherent in games, (b) the type of math knowledge being facilitated, and (c) the effects of the central learning outcomes. The findings of this review indicate that procedural and conceptual knowledge are included in games using three types of pedagogical approaches: experiential, discovery, and constructivism. Overall, this study concludes that behaviorally oriented pedagogies are still dominant in math games, and each pedagogical approach’s effectiveness varies as a function of knowledge type.

Chorianopoulos et al. [23] conducted a literature review of child-oriented video games and identified three principles behind the design of these games: (1) promote child engagement through a story and a hero, (2) implement familiar game mechanics from popular video games, (3) and provide constructive trials and errors. These principles were implemented in a serious game that teaches students how to add and subtract numbers. Fraga-Valera et al. [24] analyzed the impact of the use of serious games in primary education classrooms, specifically on students’ mathematical fluency, taking into account gamification variables and teaching experience. Then, the authors followed these principles for the development of a serious game for teaching additions and subtractions. The results showed a significant improvement in math fluency with the use of the serious game in the different courses and classroom groups studied. Authors Price et al. [25] developed a serious game in the form of a quiz that makes learners use math skills to obtain rewards. In this game, student competitiveness increased as well as their numeric ability. The game demonstrated to be a potential breakthrough to balance current technological developments and student numerical ability. When a game has a rewards system, students engage with it and are more likely to accomplish their learning outcomes. Malvasi et al. [26] conducted a study in high schools in Italy to evaluate the use of gamification, specifically for math, and the use of a serious game as a resource for learning this discipline. The authors found that teachers unconsciously applied gamification in their classrooms—even without being trained—yet they did not use the chess game, despite recognizing its benefits. The study confirmed that gamification deserves more attention in schools, especially to support math fluency, as it can positively impact motivation and learning.

Fraga-Varela et al. [27] conducted a quasi-experimental study to (1) understand the possible differences in the implementation of serious games for math fluency in primary schools concerning gender, and (2) examine the extent to which these games contribute to overall school performance. Their findings confirmed the positive impact of serious games in school environments and highlighted the opportunity to address performance differences based on gender. In their work, Cerqueira et al. [28] reported the results obtained from an experiment conducted using an Augmented Reality (AR) serious game for learning mathematical functions in middle school. The game was created specifically for this purpose and allowed the researchers to conduct an exploratory study using a qualitative and quantitative approach on two groups of teachers of different subjects: math and informatics. The study found that FootMath offers a promising approach to learning mathematical functions by providing an innovative and engaging method of interaction. It was noted to be a valuable complementary tool in educational settings, increasing students’ interest and motivation. Ibarra et al. [29] studied the literature on serious games for math learning concerning their potential positive impact on student motivation. Calleros et al. [30] applied the first level of a serious game that includes eight civilizations in a timeline for learning different math concepts. The study sought to measure student attention using an electroencephalogram (EEG) and took into account student age, gender, behavior, and mood. The study corroborated how serious games impact children’s attention during their execution. Barbieri et al. [31] conducted a case study to enhance mathematical competencies using serious games using both empirical qualitative and quantitative methods to highlight the effectiveness of serious games as a teaching methodology when motivating students to learn math. Malvasi et al. [32] evaluated the level of gamification in secondary schools, specifically in mathematics, and the use of the serious game of chess as a resource for learning this discipline. The study confirmed that gamification had several benefits, including greater motivation. Mendoza et al. [33] studied the role of serious games in the development of mathematical skills to improve learning. The authors followed a quantitative methodology with a quasi-experimental design. After comparing the pre-test and post-test of the experimental group, the findings revealed an increase of 80% in the expected learning achievement, thus indicating an improvement in learning.

2.2. Math Serious Games Using Facial Emotion Recognition

Blas et al. [34] introduced an educational tool for teachers that is based on the Kinect sensorization system. The tool proactively and dynamically interacts with students to promote knowledge in a certain area using games and physical activity. Sajjadi et al. [35] conducted a systematic review of the literature on serious games and provided an overview of the empirical evidence on the effect of individualization on player experience, learning, and other outcomes. Ninaus et al. [36] employed a machine-learning approach to facial emotion detection data to predict whether individual participants were engaged in game-based and non-game-based tasks. The task classification reached an accuracy significantly above the chance level. Maronidis et al. [37] explored the robustness of subspace learning techniques in geometric transformations. Several of these techniques were tested using four databases of facial expressions. A strong correlation was observed between recognition accuracy and image registration error. Mostefai et al. [38] proposed a generic and efficient emotion-driven approach for personalized assessment and adaptation in serious games, where two main methods and their algorithms were proposed. Ezzameli et al. [39] analyzed multimodal emotion recognition to survey affective computing and emotion recognition, examine deep learning and its main techniques, and review emotion recognition based on traditional features, deep learning features, and datasets.

Pacella et al. [40] developed Emodiscovery, an online multiplatform serious game that evaluates whether 8–11-year-old children tend to use different adaptive and maladaptive regulation strategies to change others’ negative emotions with the use of 3D animated characters. Pumphrey et al. [41] relied on a clinically available measure to assess multimodal emotion recognition abilities among individuals with multiple sclerosis. The results could provide information for clinicians amidst calls to include social cognition measures in standard multiple sclerosis assessments. Almulla et al. [42] presented a multimodal emotion recognition system capable of recognizing emotions from audio, video, and text using deep convolution neuronal networks. According to their results, the system could successfully recognize emotions of happiness, anger, sadness, fear, disgust, surprise, and neutrality. Llurba et al. [43] developed a facial emotion recognition program that transfers the collected data into a database for further analysis, which consists of establishing the first approximations of the relationship between students’ emotions and other conditions, including the subject, time of the day, and academic performance.

Chul et al. [44] reviewed the literature on facial emotion recognition. The review begins by describing conventional approaches to facial emotion recognition and summarizing the representative categories of facial emotion recognition systems and their main algorithms. Then, the authors present a summary of deep learning facial emotion recognition approaches that use deep neuronal networks to enable end-to-end learning. De Pinho et al. [45] conducted a correlational study to understand the correlation between the performance of a patient while playing a serious game with the collected data, at the same time, by an eye tracker, and the emotions detected by the Face API in the Microsoft Azure suite after processing the photos collected by a webcam. Through collected eye movements and facial expressions, the study determined that children with memory deficits have a slower processing speed, fix more times, make more regressions, and are more disposed to express an emotion of sadness or surprise. Vural et al. [46] proposed a deep machine-learning-based method for stress detection from facial images of children. The method uses a modified deep network architecture, VGG-Face, for facial emotion recognition, and three machine learning models for stress recognition. Grossard et al. [47] proposed JEMImE, a serious game to help children with Autism Spectrum Disorder (ASD) learn how to express happiness, sadness, and anger in a 3D environment in social situations that should be resolved by producing the correct facial expression. Dewhirst et al. [48] conducted a scoping review of the use of serious games for mental health. The review was conducted using a pre-defined set of criteria and a relevant word combination of three medical research databases. The obtained findings suggested that serious games could support medical interventions to reduce mental health symptoms and improve people’s outcomes. Hassan et al. [49] conducted a review of the literature to design serious games to improve social behavior in individuals with ASD. The findings of this review suggest that significant attention has been paid to children with high-functioning ASD and Asperger syndrome, yet greater focus is needed on facial emotion recognition.

In this work, we developed a task-based and challenge-based serious game named TasksZE for math learning. TasksZE implements a procedure to integrate facial emotion recognition in a serious game. The difference between such initiatives and our work lies in the fact that TasksZE introduces a novel approach by adjusting gameplay based on detected emotions, which includes:

• Real-time emotion analysis: The game uses facial recognition to identify the player’s predominant emotion in key moments, such as during tasks or challenges.
• Cross-validation of emotions: A comparison is made between the automatically detected emotions and the emotions self-reported by the student, which allows for a more precise analysis of the student’s emotional experience.

Using these components in a serious game for the teaching-learning process is not only innovative in the educational field, but also opens up new possibilities for using facial recognition technology as a learning adaptation and enhancement tool.

3. TasksZE—Design of a Task-Based and Challenge-Based Math Serious Game Using Facial and Emotion Recognition

3.1. Description of TasksZE

Learning math through serious games based on tasks and challenges offers a unique opportunity for students to learn while engaging in gamified activities that enhance their educational experience. TasksZE incorporates various elements of gameplay, including game mechanics, objectives, strategies, and progression levels. Throughout the game, the player (i.e., the student) engages in activities such as running, jumping, choosing objects, and interacting with game characters. These mechanics add depth to the learning experience by encouraging exploration. As for the game dynamics, the game continuously requires the player to answer a series of multiple-choice questions to move forward, which necessitates both math skills and strategic thinking. The game also follows a branching dialog approach to allow the player to make choices that influence the direction of the narrative and the dialog between the characters.

TasksZE’s purpose is a task-based and challenge-based math serious game for children to gain math fluency, especially fraction fluency, while playing. TasksZE relies on the face-api.js library for facial emotion recognition. Face-api.js is an API that automatically identifies a person’s face by analyzing its facial features extracted from an image or video keyframe, and then compares these features with a set of models using convolutional neural networks [50]. The motivation for implementing TasksZE is that it facilitates facial emotion recognition throughout the serious game and allows teachers to track how these emotions evolve in response to the variation in content presented during the game. The approach of this research is to provide the teacher with a tool, such as a serious game with facial emotion recognition, which allows the evaluation of the emotional impact on learning by identifying the topics that are most difficult for the student to deal with in the serious game, and the facial emotion recognition of TasksZE allows the teacher to provide feedback on the topics based on the obtained emotions. The TasksZE context is the intersection of education and technology, intending to improve math learning through a serious game using facial emotion recognition. In the technological context, facial emotion recognition is used to monitor the emotions of players while they solve mathematical problems, and then these emotions allow the teacher to provide feedback on the topics. In the educational context, the use of serious games can offer a more interactive, engaging, and motivating approach to math learning. This motivation can be increased with the use of technologies, such as facial emotion recognition, because the student can identify their emotions in real-time during the tasks and challenges. The target audience is students 10-to-12-year olds who, in their basic education, need to know the basic concepts about fractions, visualize fractions, and perform exercises with fractions. The implementation of a serious game that incorporates facial emotion recognition in math learning allows us to detect emotions expressed in each task or challenge to identify the most complete or most difficult topics for students based on their emotions and to provide feedback or detect areas of opportunity to reinforce these topics by adjusting the gameplay.

3.2. Procedure to Integrate Facial Emotion Recognition in a Serious Game

TasksZE incorporates a procedure to structure, plan, and control the procedure of learning math through a serious game. TasksZE integrates technical and pedagogical aspects, incorporating user interfaces, device resources, and permissions for alerts, sounds, location, and internet connectivity. Additionally, TasksZE accounts for the data exchanged in its facial recognition module. Figure 1 illustrates the procedure to integrate facial emotion recognition in a serious game.

• Game Analysis—The learner (i.e., player) is introduced to the topic of the game. To this end, TasksZE introduces key concepts and narratives for the topic and explains how to play the game.

  o

  Generate a Serious Game Interface: TasksZE generates a serious game interface depending on the tasks or challenges selected by the user.

• Game Implementation—The learner must overcome the challenges presented. Also, to obtain rewards, they must answer correctly a series of questions about the topic being learned.

  o

  Serious Game Interaction Interface: This component is responsible for interacting with the user to overcome challenges and answer questions.

• Facial Emotion Recognition Implementation—The facial emotion recognition implementation module detects and captures the player’s emotions at different times throughout the game. Next, the same module analyzes the collected data to look for the dominant emotion during each task.

  o

  Face Detection: Detects the input frame.

  o

  Preprocessing: Identifies the student’s face.

  o

  Feature Extraction: The features for emotion recognition are selected.

  o

  Classification: This implements the convolutional networks provided by Face-Api.js to identify emotions.

  o

  Recognized Emotion: This identifies the emotions resulting from the previous component.

  o

  Emotion Evaluation: This recognizes the predominant emotion when playing serious games.

3.3. Architecture of TasksZE

TasksZE is designed with a multi-layered architecture for high scalability and easy maintenance. As depicted in Figure 2, the responsibilities in the serious game are distributed evenly across each game component in each layer; thus, each layer fulfills a particular role.

• Presentation Layer. Provides an interaction between the serious game and the player through the user interface (developed in HTML5, JavaScript, and CSS). In the presentation layer, the player (i.e., the user) authenticates themselves, chooses the lesson to be practiced in the game, and can explore the multiple resources of the serious game.
• App Layer. Acts as an intermediary between the presentation layer and the Services Layer. It is responsible for accessing the Services Layer and simplifying service information by converting it to understandable data for the interface.
• Services Layer. Provides a set of services (modules) representing different functions in the serious game. It is also responsible for interacting with the repositories. For instance, the Serious Game Module is responsible for executing the serious game once requested by the player in the presentation layer. To this end, the Serious Game Module moves the serious game to the Serious Game Repository and handles the execution of the elements needed to run the serious game. On the other hand, the Face Identity Module performs facial emotion recognition by consuming de face-api.js and stores the data in the Face Identity Repository for further analysis.
• Data Layer. Contains the repositories with the data needed to run the serious game, i.e., the Serious Game Repository and the Facial Identity Repository.
• The architecture of TasksZE contains the following components:
• User Interface. Responsible for communicating with the user (i.e., the player) and allows them to modify the game’s themes, consult media resources, and select the game to play.
• Serious Game Request. Processes user interface requests.
• Serious Game Response. Builds the answers for the user interface.
• Serious Game Module. Runs the serious game as selected by the user in the presentation layer. To this end, the Serious Game Module requests the game from the repository and manages the necessary elements for the serious game to run. The Serious Game Module has three elements:

  o

  Level Component. Delivers the different levels of complexity of the game.

  o

  Tasks Component. Handles the tasks that are presented to the player throughout the serious game.

  o

  Challenges Component. Handles the different challenges that are presented to the user at each level of the serious game.

• Face Identify Module. This module performs facial emotion recognition using the face-api.js library. The data are stored in the Face Recognition Repository for further analysis. The Face Identity Module comprises the following elements:

  o

  Face Identify Component. Processes the results of the user’s facial recognition, including facial descriptions, and eliminates image noise.

  o

  Face Identify Processor. Retrieves information related to user age and gender and identifies the facial emotion expressions.

  o

  Emotion Processor. Recognizes the user’s emotions during the game based on the facial expression processed by the Face Identity Processor.

• Face-api.js. Open source library for face recognition on the browser side. It includes a series of lightweight, pre-trained neural networks that can be efficiently executed in web environments [51]. Each neural network is designed for a specific purpose, such as single-face or multi-face detection, facial point generation, facial recognition based on facial descriptors, facial emotion recognition, and age/gender recognition, among others.
• Serious Game Repository. This is where serious games with facial recognition are stored.
• Face Identity Repository. This is where the facial data recognized during the game are stored.

3.4. Facial Emotion Recognition

Emotion recognition involves estimating the player’s predominant emotion by using facial recognition technology to identify key features, such as the location and scale of the face, as well as reference points (e.g., eyes, nose, mouth, and cheeks). Based on these data, an artificial intelligence algorithm from the emotion analysis library determines the player’s emotional state. The emotion recognition procedure in TasksZE thus comprises two components. First, the facial recognition component is used to gather data points about the player, such as gender and age. The second component focuses on recognizing the player’s emotional state during the game. This data are stored and later compared with the results from a player-emotion-identification questionnaire to analyze the correlation between the results. The player-emotion-identification questionnaire was developed to identify players’ emotions when performing specific actions in the game. This questionnaire was developed with input from educators and aims to pinpoint emotional responses during particular scenarios, such as answering a question. The questions from the player-emotion-identification questionnaire are described as follows.

• How did you feel when you detected an improper fraction?
• How did you feel when you detected a proper fraction?
• How did you feel about answering the following question: What is the lower number of a fraction called?
• How did you feel about answering the following question: What is the result of reducing 75/135 to the lesser term?
• How did you feel about answering the following question: What is the name of the method used when multiplying the numerator of the first fraction by the denominator of the second fraction, then writing that number on top of the final fraction?
• How did you feel about answering the following question: Is 2/5 a fraction?
• How did you feel about solving the following problem: Karen spent 1/3 of her pocket money on candy and 1/9 on soda. If represented in fractions, how much money did she spend?
• How did you feel about answering the following question: Is 4/6 a fraction?
• How did you feel about solving the following problem: There are 13 pieces of fruit in a box, five of which are oranges. If represented in fractions, how many pieces of fruit are left in the box?

Each question was accompanied in the answer section by a panel of options according to the emotions detected by face-api.js, which are neutral, happy, sad, disgusted, angry, surprised, and fearful; for each option, a Likert scale [52] was presented for the student to mark the emotion intensity, as shown in Figure 3.

The player-emotions-identification questionnaire was developed based on the qualitative exploration of the experiences of players involved in experiments with adaptive serious games [53] because qualitative exploration analyzes feedback data and players’ perspectives on both performance and enjoyment. On the other hand, the qualitative evaluation was also taken into account to explore the users’ perceptions about their satisfaction. We took as a basis the method they used with an evaluation at the time followed by formal interviews for each participant [54]. The questions were prepared based on the guidelines for qualitative inquiry [55] to ensure rigor and credibility for the player-emotion-identification questionnaire.

Figure 4 illustrates the flowchart of the emotion recognition technology implemented in TasksZE.

4. Development of TasksZE

As a game based on tasks and challenges, TasksZE provides a clear objective to the player: to solve fractions correctly as they are displayed. To this end, the player embarks on a series of missions that outline the tasks and challenges to be completed. Completing these missions allows the player to unlock new skills and levels of interaction, enhancing their learning experience as they advance through the game. As the player progresses from one level to the other, the complexity of the operations increases.

Game scenarios in serious games must be carefully designed to ensure a logical connection between the gameplay experience and the learning outcome. TasksZE supports six different but familiar scenarios, and there are no time restrictions in the serious game, allowing the player to progress at their own pace. TasksZE also enables the player to interact with several non-playable characters (NPCs), who guide the player through the different game missions by explaining to them how to solve each math problem; this allows the student to gain knowledge and an understanding of the topic. The main scenario in TasksZE is designed to resemble a sixth-grade public elementary school classroom (see Figure 5). This familiar setting helps students identify with elements they encounter throughout the school year in their classrooms, thus reinforcing the concept that they can learn through an educational game. Task-based and challenge-based serious games must contain the following attributes: NPC-based task description, progress bars, multiple choices to select, major tasks, branching tasks, puzzles, research points, study, and requirements (see Figure 6) [56].

The player earns rewards once they complete a challenge in the serious game. These rewards are points that accumulate and help the player advance through the serious game’s five levels; however, these points can also be used to purchase accessories in the Coffee House scenario. In each level of the serious game, the player is assigned a task, and the Stage Manager introduces the challenge to be completed. The player progresses through the serious game by solving fractions of increasing complexity as they move from one level to another and engage in different scenarios inspired by elementary school facilities to create a familiar and confidence-boosting environment (see Figure 7 and Figure 8).

TasksZE has a facial emotion recognition module that runs in the background to capture the player’s emotions at different moments in the serious game. The module then analyzes these data to determine the dominant emotion during the game. An example of emotion detection is presented in Figure 9. Figure 9 shows an adult playing the game, which is used to protect the identity of students for child protection purposes.

5. Case Study

This section discusses the case study in which TasksZE was implemented.

5.1. Game Scope and Implementation

The case study was conducted among 20 sixth-grade students (10 males and 10 females) in a middle school. The students had basic computer skills and a basic knowledge of serious games, and their ages ranged from 11 (6: male and 5: female) to 12 (4: male and 5: female) years old. To define the learning scope of the game, we administered a questionnaire to the 20 sixth-grade students to pinpoint math topics they found particularly challenging or were interested in learning. We complemented the questionnaire with a series of interviews and diagnostic assessments. Following the analysis of the collected data, we defined the scope of TasksZE as a math serious game for supporting fraction fluency using facial emotion recognition.

TasksZE was implemented in compliance with the course syllabus after the topic had been studied in class. The goal was for the game to be introduced smoothly, without particular emphasis, to allow the students to react to it as naturally as possible. The students were organized into groups of five, and we scheduled four 10–15-min sessions with each group. The groups were supervised by both the class teacher and a technical assistant. Each student had the opportunity to engage with the game at their own pace, allowing them to explore the presented topics and experience the surprise of emotion recognition through facial analysis. After completing the game, each student was asked to return to their seat and refrain from discussing the game to maintain the integrity of the data collection.

Results Analysis

Table 1. Comparative results: player self-reported emotions vs. Face-api.js emotion recognition.

Math Topic	Self-Reported Emotion			Emotion Recognition by Face-api.js
	Emotion	Mean	Deviation	Emotion	Mean	Deviation
Add Fractions	Happiness	3.65	2.35	Happiness	4.57	0.43
Divide Fraction	Anger	4.78	1.22	Anger	5.00	0.00
Multiply Fraction	Neutral	3.61	2.39	Anger	4.37	0.63
Proper Fractions	Happiness	4.40	1.60	Happiness	5.00	0.00
Improper Fractions	Neutral	4.23	1.77	Happiness	4.89	0.11
Equivalent Fractions	Surprise	4.47	1.53	Anger	5.00	0.00

comparatively presents the results of the analysis of the emotions self-reported by the students (i.e., players) in the player-emotion-identification questionnaire against those detected by face-api.js for each of the six topics covered by the game. We provided two statistical measures: mean and standard deviation.

Using the mean and standard deviation are essential because these statistical measures offer a balanced and straightforward way to summarize and understand both the central tendency of and the variability in the data. While other metrics, like the median and interquartile range, are available, the mean and standard deviation are particularly valuable when the data follow a normal distribution, enabling a more straightforward interpretation and facilitating more accurate comparisons and analyses in various statistical contexts.

Table 2. Results of Face-api.js emotion recognition following serious game questions.

Serious Game Question	Emotion	Mean	Deviation
How did you feel when you detected an improper fraction?	Neutral	3.41	0.59
How did you feel when you detected a proper fraction?	Happiness	3.68	0.32
How did you feel about answering the following question: What is the lower number of a fraction called?	Neutral	4.01	0.99
How did you feel about answering the following question: What is the result of reducing 75/135 to the lesser term?	Happiness	3.62	0.38
How did you feel about answering the following question: What is the name of the method used when multiplying the numerator of the first fraction by the denominator of the second fraction, then writing that number on top of the final fraction?	Anger	4.18	0.82
How did you feel about answering the following question: Is 2/5 a fraction?	Happiness	3.29	0.71
How did you feel about solving the following problem: Karen spent 1/3 of her pocket money on candy and 1/9 on soda. If represented in fractions, how much money did she spend?	Sadness	3.97	1.02
How did you feel about answering the following question: Is 4/6 a fraction?	Happiness	3.45	0.55
How did you feel about solving the following problem: There are 13 pieces of fruit in a box, five of which are oranges. If represented in fractions, how many pieces of fruit are left in the box?	Neutral	3.23	1.77

lists the questions featured in TasksZE. These questions were formulated with the help of the class teachers and were expected to trigger a reaction (emotion) from the players. These emotions were identified with the help of face-api.js and stored in a database to later determine the dominant emotion in each question.

The results in Table 2 provide an opportunity to pinpoint areas that may deserve special attention in class. By analyzing which serious game questions elicited negative emotions, such as anger or sadness, and which caused little interest (resulting in a neutral emotion), teachers can better understand and address students’ emotional and engagement needs.

TasksZE also allowed each student to rate each serious game challenge. As previously mentioned, the challenges increased in difficulty as the student (player) moved from one level to another. The results of this analysis are summarized in

Table 3. Results of Face-api.js emotion recognition in the serious game challenges.

Game Challenge	Emotion	Difficulty	Median	Deviation
Challenge 1	Happiness	Medium	4.12	0.88
Challenge 2	Happiness	Easy	4.07	0.93
Challenge 3	Anger	Medium	3.52	1.48
Challenge 4	Happiness	Medium	3.62	0.38
Challenge 5	Anger	Difficult	4.82	0.18

.

The results presented in this section demonstrate that facial emotion recognition is successfully implemented in TasksZE using the face-api.js library. Based on the data collected by face-api.js, as the students (players) progressed through the game, their emotional responses changed. This corroborated the effectiveness of integrating facial emotion recognition in TasksZE.

6. Usability Evaluation

We used the System Usability Scale (SUS) [57] to guide the usability evaluation of TasksZE. The System Usability Scale (SUS) is a questionnaire that is used to evaluate the usability of products and services. The SUS is a highly robust and versatile tool for usability professionals [58]. It is renowned for its simplicity, speed, flexibility, and ease of interpretation [59]. Moreover, it can be customized to reflect the specific needs of a project.

6.1. Data Collection and Analysis

The teachers were asked to play a game to evaluate its usability. Once finished, they completed the usability questionnaire, which comprised 10 questions, using a five-point Likert scale [59] to rate their agreement with each statement as follows: (1) Totally agree, (2) Agree, (3) Neither agree nor disagree, (4) Disagree, and (5) Totally disagree.

6.2. Usability Questionnaire

The statements in the usability questionnaire are as follows:

• Q1. I think that I would like to use this system frequently.
• Q2. I found the system unnecessarily complex.
• Q3. I thought the system was easy to use.
• Q4. I think that I would need the support of a technical person to be able to use this system.
• Q5. I found the various functions in this system were well-integrated.
• Q6. I thought there was too much inconsistency in this system.
• Q7. I would imagine that most people would learn to use this system very quickly.
• Q8. I found the system very cumbersome to use.
• Q9. I felt very confident using the system.
• Q10. I needed to learn a lot of things before I could get going with this system.

6.3. Results

Figure 10 presents the results of TasksZE’s usability evaluation. The evaluation yielded positive results. As regards the ease of use (Q3), the serious game scored 4.6 out of 5. Specifically for this question, 70.23% of the responses fell into the “Strongly agree” category, while 29.77% indicated “Agree.” Such results demonstrate that the usability of TasksZE exceeds the expectations according to the SUS.

As shown in Figure 10, TasksZE exhibits satisfactory levels of user acceptance, scoring above the standard threshold of 4. This suggests that the serious game was successfully developed for the purpose for which it was intended. The usability questionnaire for TasksZE had three key questions that played a crucial role in the evaluation results, as they are directly related to its usability.

• Q1. I think that I would like to use this system frequently.
• Q3. I thought the system was easy to use.
• Q7. I would imagine that most people would learn to use this system very quickly.

Additionally, the questionnaire results confirm that TasksZE is a serious game with easy-to-use facial emotion recognition to identify players’ emotions throughout the game. Overall, the results show positive data for the usability of TasksZE as a task-based and challenge-based math serious game for fraction fluency using facial emotion recognition. With a usability score of 70.23%, which exceeds expectations, this study demonstrates that TasksZE is effective and meets its proposed objectives, successfully satisfying the usability requirements.

7. Discussion

Serious games offer immersive learning experiences that promote conceptual and procedural learning, in addition to motivating and engaging students. Brauner et al. [18] highlighted their usefulness for handling incomplete information and developing critical skills, such as decision making. Gauthier et al. [19] proposed a framework based on the activity theory to align games with educational objectives, such as mathematical problem-solving. Authors, such as Díaz et al. [20] and Pan et al. [21], highlighted the personalization of these games in technical areas, while Kacmaz et al. [22] highlighted pedagogical approaches, such as experiential and constructivist learning. Some research [24,25] showed that serious games improve mathematical fluency and numerical skills, promoting better performance and greater motivation. Meanwhile, the use of technologies, such as Kinect and facial recognition, in serious games seeks to improve educational interaction by personalizing and monitoring emotions and behaviors in real time. Blas et al. [34] presented a tool that combines physical activity and learning, promoting a dynamic and practical environment. Sajjadi et al. [35] highlighted individualization, while some studies [36,38] explored how emotional analysis can adjust the learning experience. In addition, authors such as Grossard et al. [47] and Hassan et al. [49] showed that these technologies help to improve social skills in children with ASD.

For these reasons, we developed a proposed procedure and architecture software to integrate facial emotion recognition in a serious game. To perform a proof of concept of our procedure, we developed TasksZE. TasksZE is a task-based and challenge-based math serious game using facial emotion recognition designed for math learning, particularly on the topic of fractions. TasksZE was developed as a case study to test the integration of the architecture and the proposed procedure. As part of this case study, we conducted an evaluation to verify its proper function and to show that the serious game serves as a tool for teachers to detect emotions, and, based on these results, improve their teaching–learning process. This study was conducted on 20 students with an age range of 11-to-12-years old using questionnaires and pre-interviews in which the most difficult areas of mathematics for the students were identified. The results showed emotional changes in students according to the difficulty of the content, which allowed teachers to identify areas of opportunity to improve learning. The students with the best results were females at 12 years old and boys at 11 years old. The best-rated topic for happiness was proper fractions and the topic that generated the most anger in the students was divide fractions. This was due to the difficulty of dividing fractions. Otherwise, the topic that generated surprise was equivalent fractions, because it was the topic that presented the most dynamic content, thus generating the interest and motivation of the students.

We conducted a qualitative evaluation of the TasksZE using the System Usability-Scale (SUS) questionnaire. The usability evaluation was applied to a task-based and challenge-based math serious game using facial emotion recognition to measure whether the system was easy to use, or whether functions in this system were well integrated. Data were collected using an SUS questionnaire. The results presented in Figure 10 are overwhelmingly positive, with an overall average score of 4.6 out of 5, indicating satisfactory usability and surpassing expectations. Additionally, the usability index exceeded the benchmark of 4, demonstrating that TasksZE was well-received and fulfilled its intended purpose. Overall, the integration of technologies, such as serious game using facial emotion recognition, was regarded as innovative, and students found the game appropriate for classroom use. These findings suggest that the game is an effective and user-friendly tool for enhancing math learning.

8. Conclusions

Serious games are a valuable tool for math fluency, as they allow students and teachers to refine and redefine teaching–learning experiences. Moreover, by integrating technologies, such as facial emotion recognition, these games have the potential to become more powerful motivational tools in education. In this work, we proposed TasksZE, a task-based and challenge-based serious game, for math learning with a novel approach by adjusting the gameplay based on the detected emotions. The underlying motivation is to provide a tool that easily and quickly guides the integration of facial emotion recognition into a serious game. To this end, TasksZE incorporates a procedure to integrate facial emotion recognition in a serious game and a layered software architecture. The architecture is loosely coupled because the components are weakly associated, and programming language and software development technologies are non-dependent. Its only goal is to guide users in the process of using facial emotion recognition in a serious game, thus showing users the components to be developed and how to interact with them. We developed TasksZE, a challenge-based and task-based math serious game for fraction fluency, which uses facial emotion recognition. The game was implemented among a group of 20 sixth-grade students from a public middle school. The evaluation of TasksZE provided both quantitative and qualitative data on the educational impact of serious games, and also yielded a commendable score of 70.23% in terms of the game’s ease of use. Additionally, this study found that face.api.js was effectively integrated with TasksZE, as the players’ emotions changed as they progressed in the game. This suggests that serious games can help students increase their understanding of challenging topics through engaging gameplay, providing an additional learning tool for students and a teaching aid for teachers. The analysis of the data collected by face.api.js indicates that a player’s emotional response throughout the game can vary significantly, depending on the game’s challenge or level of difficulty. This allows teachers to identify areas for improvement in their learning activities. The results obtained in the SUS evaluation show that TasksZE is a serious game that is easy to use and facial emotion recognition functions in this system are well integrated. Also, students responded that they would use TasksZE frequently. The results of the SUS questionnaire confirm the aim of this research. Moreover, the results obtained from TasksZE’s implementation allow us to evaluate the emotional impact on learning by identifying the topics that are most difficult for the student to address through the emotions detected.

This study has some limitations. For example, the sample size was very small, which led to the lower reliability of the statistical results. Another limitation was that the tests were carried out by students in the same middle school and with a very reduced age range (11–12 years old). Because the learning environment is intended to be used by students with different middle-school ages, more experiments should be conducted with a larger number of test subjects, including more math-related topics, and with a wider age range. Some technological limitations are the range of emotions detected, like neutral, happy, anger, and sadness. Another limitation was the emotion identification API used, which is Face-api.js. From this perspective, we are considering using other APIs, such as Face Recognition.js or OpenCV. The last limitation is that the serious game runs only as a desktop version, but we are developing a mobile app version.

For future work, we are considering extending TasksZE to other subjects, such as chemistry and physics, to identify the emotional impact on learning and to create a tailored learning experience for students. Finally, we are considering identifying additional emotions, such as frustration, discouragement, or boredom, among others, which are considered in academic environments. Furthermore, future work on TasksZE will focus on evaluating both user experience and learning experience using a different, yet-to-be-defined mathematical topic. Finally, TasksZE could be analytically compared with similar tools both reported in the literature and commercially available.