Multimodal Technol. Interact., Volume 8, Issue 11 (November 2024) – 12 articles

In recent years, the integration of technology into education has significantly transformed teaching methods, especially in science education. Tools like PhET simulations have proven highly effective in enhancing student engagement and comprehension. Research has highlighted the value of simulation-based learning in fostering critical thinking and problem-solving skills. This study aimed to explore the impact of simulations, with a focus on PhET, on improving elementary students’ learning outcomes, an area that remains under-researched. The study compared the performance of two groups of third-grade students: one group learned about solubility using PhET simulations, while the other relied on traditional textbook instruction. Each group comprised fifty students. The study lasted for a two-month period. The instructional approach was investigative learning. Data were gathered through student responses to materials science questions aligned with Bloom’s Taxonomy, allowing for a detailed evaluation of their understanding and application of scientific concepts. Responses were assessed for accuracy and scored accordingly. We ran an independent-sample t-test to decide whether the difference in the mean score in science achievement between the two research groups was significant. The results showed that students using PhET simulations not only achieved significantly higher scores but also demonstrated their ability to explain their reasoning during problem-solving tasks. These findings emphasize the substantial advantages of incorporating digital tools like PhET simulations into elementary science education, as they enhance conceptual understanding and better equip students to tackle future scientific challenges. The present research results complement the previous research on using technology in the chemistry elementary classroom and add the issue of simulations to this research. The results of this study are centered on the topic of solubility. To broaden the generalizability of these findings, future research should examine the effects of simulations on student achievement in a wider range of elementary science topics. Full article

One of the most crucial aspects of the user experience in virtual reality (VR) is the sense of presence. To evaluate this, both subjective and objective methods can be employed. While subjective methods are easy to implement and interpret, they may not fully capture user feedback, and the results can sometimes lack consistency. In contrast, using objective methods, such as electroencephalography (EEG), can provide more reliable insights. To investigate the influence of realism on the sense of presence, we conducted an EEG study with 21 participants who experienced two VR environments—one realistic and one non-realistic. During the study, we continuously measured their brain activity using an EEG device. Our findings showed that alteration in the level of realism in an environment can be detected through changes in brain activity. Notably, we observed that users take longer to adapt to a non-realistic environment when transitioning from a realistic scene, compared to the reverse. Although our study has limitations, such as the total number of participants, we gained valuable initial insights into how realism may influence brain activity. These findings suggest that higher realism may lead to reduced cognitive load, increased attention, improved decision-making, and suppression of irrelevant information. Full article

Serendipitous learning, characterized by the discovery of new insights and unexpected connections, is recognized as a valuable educational experience that stimulates critical thinking and self-regulated learning. While there have been limited efforts to develop serendipity-oriented recommender systems in education, these systems often fall short in supporting learners’ agency, that is, the sense of ownership and control over their learning journey. In this paper, we introduce an Interactive Evolutionary Computation (IEC)-driven recommender system designed to empower learners by granting them control over their learning experiences while offering recommendations that are both novel and unexpected yet aligned with their interests. Our proposed system leverages an Interactive Genetic Algorithm in conjunction with Knowledge Graphs to dynamically recommend learning content, with a focus on the history of scientific discoveries. We conducted both numerical simulations and experimental evaluations to assess the effectiveness of our content optimization algorithm and the impact of our approach on inducing serendipity in informal learning environments. The results indicate that a significant number of participants found certain recommended learning materials to be engaging and surprising, providing evidence that our system has the potential to facilitate serendipitous learning experiences within informal learning contexts. Full article

Understanding document layouts is vital for enhancing document exploration and information retrieval for sighted individuals. However, for blind and visually impaired people, it becomes challenging to have access to layout information using typical assistive technologies such as screen readers. In this paper, we examine the potential benefits of presenting documents on two-dimensional (2D) refreshable tactile displays. These displays enable the tactile perception of 2D data, offering the advantage of dynamic and interactive functionality. Despite their potential, the development of user interfaces (UIs) for such displays has not advanced significantly. Thus, we propose a design of an intelligent tactile user interface (TUI), incorporating touch and audio feedback to represent documents in a tactile format. Our exploratory study for evaluating this approach revealed satisfaction from participants with the experience of directly viewing documents in their true form, rather than relying on screen-reading interpretations. Additionally, participants offered recommendations for incorporating additional features and refining the approach in future iterations. To facilitate further research and development, we have made our dataset and models publicly available. Full article

Interactive conversational agents, also known as chatbots, have the potential to increase the success rate of digital technology interventions to promote healthy behaviors. However, due to their newness and limited use, little is known about their integration, usefulness, and effectiveness in promoting smoking and vaping cessation. The aim of this mixed-methods systematic review was to assess the effectiveness and characteristics of current interactive conversational agents in promoting and supporting smoking and vaping cessation. A mixed-methods systematic review was conducted to identify studies published in the last 20 years in five relevant databases. Eight studies, including seven on smoking cessation and one on smoking and vaping cessation, were included. The results showed that, compared to other smoking cessation methods, chatbots can lead to better engagement in treatment, resulting in higher rates of sustained abstinence and improved quality of life. In addition, chatbots can be perceived as empathetic and establish a decent therapeutic alliance thanks to their communication skills. This knowledge could be useful for the development of interactive conversational agents to support smoking and/or vaping cessation. Alternative intervention tools targeting younger generations, such as chatbots, may offer an additional way for public health professionals to reach them. Full article

Managing complex behaviors in disability support settings requires competent de-escalation skills. However, the current training methods often lack sufficient opportunities for realistic practice. This study details a three-stage development and evaluation of a virtual reality (VR) application for disability support staff to safely build de-escalation skills through simulated interactions. The first phase involved creating VR prototype scenarios depicting escalations with adolescent clients. Next, 12 disability support experts conducted content validation by refining the scenarios to confirm appropriateness and realism. Finally, a pilot study tested the tool’s usability and examined the initial construct validity in 20 participants. The prototype achieved high usability ratings (average 81.0 (SD 10.7) on the System Usability Scale). Additionally, a positive correlation between VR performance and empathy levels, as measured using the Toronto Empathy Questionnaire, was found (Pearson’s r = 0.487, p = 0.035). The promising results highlight the VR application’s potential as a transformative training tool. Future research should expand scenario diversity and compare VR with traditional methods to establish its efficacy in diverse settings and offer a path to enhance staff and student capabilities in challenging environments. Full article

Spatial abilities, which are sources of our capacity to understand visual and spatial relations among objects, as well as the abilities to generate, retain, retrieve, and transform well-structured visual information are important in several scientific fields and workplaces. Various tests have already been prepared to measure these abilities, including the Mental Cutting Test, which is considered the golden standard of measurement. However, much less attention is paid to how to prepare students for this test, as well as how to develop these skills most effectively. The purpose of this research is to study the multimodal approach of improving these skills and its effectiveness, i.e., the mixed use of 2D tools similar to the paper-based test, and 3D tools, including augmented reality and web-based interfaces in training students for those kinds of tasks. We want to demonstrate and prove with tests that multimodal modes of training can significantly affect the effectiveness of developing these skills. Moreover, through appropriately combining these methods, they can reinforce each other to form a multimodal approach, which is the most effective way for developing spatial skills and improving students’ performance related to the Mental Cutting Test. Full article

This scoping review examines the broad applications, risks, and ethical challenges associated with Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), within the context of Metaverse. XR is revolutionizing fields such as immersive learning in education, medical and professional training, neuropsychological assessment, therapeutic interventions, arts, entertainment, retail, e-commerce, remote work, sports, architecture, urban planning, and cultural heritage preservation. The integration of multimodal technologies—haptics, eye-, face-, and body tracking, and brain–computer interfaces—enhances user engagement and interactivity, playing a key role in shaping the immersive experiences in the Metaverse. However, XR’s expansion raises serious concerns, including data privacy risks, cybersecurity vulnerabilities, cybersickness, addiction, dissociation, harassment, bullying, and misinformation. These psychological, social, and security challenges are further complicated by intense advertising, manipulation of public opinion, and social inequality, which could disproportionately affect vulnerable individuals and social groups. This review emphasizes the urgent need for robust ethical frameworks and regulatory guidelines to address these risks while promoting equitable access, privacy, autonomy, and mental well-being. As XR technologies increasingly integrate with artificial intelligence, responsible governance is essential to ensure the safe and beneficial development of the Metaverse and the broader application of XR in enhancing human development. Full article

Many disciplines—from science to art and education—engage with the postdigital concept, where human activities transform into digital activities. The post-COVID-19 era has involved new consequences for societies, where education has increasingly utilized online platforms. Having said that, online pedagogy, with the directions and discourses of the senses, particularly visual and aural, remains under-explored. Studies have addressed the educational implications of the aural sense, but this study explores the nexus between digital aesthetics and sound pedagogy and delves deeper into students’ primarily descriptive learning outcomes. A multi-disciplinary class covered aesthetic terminologies, followed by a series of digital arts that envisioned digital, aesthetic, and listening practices. Considering the global agenda’s recent “new norms”, this study contributes to the postdigital era of sound and acoustic-related sciences. Full article

Sustainable natural resource management is a high priority in the 21st century as it plays a crucial role in mitigating climate change and preventing some of its consequences like loss of biodiversity, land degradation, desertification, and the exhaustion of natural resources. This concern is reflected in the United Nations Sustainable Development Goals (SDGs), which emphasize, among other factors, sustainable cities and communities, responsible production and consumption, and climate action. Achieving sustainable natural resource management begins with raising awareness and educating the next generation. Therefore, it is essential to develop educational initiatives that prepare young people to become responsible and proactive adults in promoting environmental sustainability across industries and communities. Additionally, these initiatives should develop critical and analytical thinking skills, nurture innovative mindsets for creating environmentally sound solutions, and enhance the ability to collaborate within multidisciplinary teams. The NATURE project addressed these needs by designing and developing a serious educational game that fosters this set of skills. The results of the pilot testing show that the game is an effective tool and contributes to the education and awareness of the younger generation. Full article

Addressing human trafficking is crucial due to its severe impact on human rights, dignity, and well-being. Serious games refer to digital games that are designed to entertain while also accomplishing at least one additional objective, such as learning or health promotion. Serious games play a significant role in raising awareness, training professionals, fostering empathy, and advocating for policy improvements related to human trafficking. In this study, we systematically examine and assess the current landscape of serious games addressing human trafficking to unveil the existing state, pinpoint gaps, and propose future research avenues. Our investigation encompassed academic publications, gray literature, and commercial games related to human trafficking. Furthermore, we conducted a thorough review of evaluation criteria and heuristics for the comprehensive assessment of serious games. Subsequently, incorporating these evaluation metrics and heuristics, the games were subjected to evaluation by both players and experts. Following a combined qualitative and quantitative analysis, the results were deliberated upon, and their implications were expounded. Five serious games related to human trafficking were identified and evaluated using the SGES and EGameFlow scales, along with both game-specific and serious game heuristics. Player and expert evaluations ranked “(Un)TRAFFICKED” and “Missing” as the best-performing games, while “SAFE Travel” received the lowest ratings. Players generally rated the games higher than experts, particularly in usability, feedback, and goal clarity, although the games scored poorly in audiovisual quality and relevance. Experts highlighted deficiencies in motivation, challenge, and learning outcomes. The lack of personalization and the absence of social gaming elements point to the need for more targeted human trafficking games adapted to different demographics, cultures, and player types. Full article

The integration of automated vehicle (AV) technology in public transportation systems offers promising opportunities to improve the flexibility and safety of the traffic environment. However, user acceptance remains a critical challenge in the field of human-machine interaction for the effective deployment of shared autonomous vehicles (SAVs). This study presents a design framework aimed at enhancing user acceptance through human-machine interface (HMI) design tailored to SAVs. The framework is developed in adherence to relevant interaction design principles, following a systematic approach encompassing three key steps: analysis, synthesis, and evaluation. It integrates user acceptance factors into the design process, providing a structured method for designers. The framework was iteratively refined through interviews with three international domain experts; a focus group discussion with 10 researchers and professionals specializing in automotive interaction designers; and a workshop with 30 students and designers. The results demonstrate the framework’s ability to guide the development of user-acceptable HMI solutions. The paper concludes by emphasizing the need for further exploration into how user acceptance factors evolve over time and how real-world testing can validate the framework’s effectiveness in promoting user acceptance and satisfaction. Full article