Multimodal Technologies and Interaction

18 pages, 12564 KiB

Open AccessArticle

The Presenter in the Browser: Design and Evaluation of Human Interactive Overlays with Web Content

by Maxime Cordeil, Anais Servais, Guillaume Truong, Tim Dwyer, Dhaval Vyas and Christophe Hurter

Multimodal Technol. Interact. 2025, 9(2), 10; https://doi.org/10.3390/mti9020010 - 27 Jan 2025

Abstract

This research explores the design and evaluation of a webcam-based presentation tool that enables presenters to directly interact with web content via free-hand gestures. Our approach consists of overlaying the webcam video feed on top of web browser content to enable live presentations [...] Read more.

This research explores the design and evaluation of a webcam-based presentation tool that enables presenters to directly interact with web content via free-hand gestures. Our approach consists of overlaying the webcam video feed on top of web browser content to enable live presentations of any webpage. To support interactive presentations, we designed free-hand gesture interactions with the webpage to enable pointing, clicking, panning, and zooming interactions. We propose three alternatives to enable free-hand clicking: dwell time, modal key control, and a pinching interaction technique. We conducted an exploratory user study of these alternative designs to gather insights on the usability of such systems from a presenter point of view, with a focus on understanding the impact of the three techniques on flow interruptions. The results indicate that the system we propose can be used to deliver presentations effectively and that natural gestures do not disturb the flow of the presentation. Full article

► Show Figures

Figure 1

13 pages, 874 KiB

Open AccessArticle

Intention to Work with Social Robots: The Role of Perceived Robot Use Self-Efficacy, Attitudes Towards Robots, and Beliefs in Human Nature Uniqueness

by Jean-Christophe Giger, Nuno Piçarra, Grzegorz Pochwatko, Nuno Almeida and Ana Susana Almeida

Multimodal Technol. Interact. 2025, 9(2), 9; https://doi.org/10.3390/mti9020009 - 21 Jan 2025

Abstract

Recent studies have enlightened the crucial role of perceived robot use self-efficacy in human robot interaction. This paper investigates the interplay between perceived robot use self-efficacy, attitudes towards robots, and beliefs in human nature uniqueness (BHNU) on the intention to work with social [...] Read more.

Recent studies have enlightened the crucial role of perceived robot use self-efficacy in human robot interaction. This paper investigates the interplay between perceived robot use self-efficacy, attitudes towards robots, and beliefs in human nature uniqueness (BHNU) on the intention to work with social robots. Participants (N = 117) first filled out a questionnaire measuring their BHNU and attitudes towards robots. Then, they were randomly exposed to a video displaying a humanoid social robot (either humanlike or mechanical). Finally, participants indicated their robot use self-efficacy and their intention to work with the displayed social robot. Regression and serial mediation analyses showed the following: (1) the intention to work with social robots was significantly predicted by robot use self-efficacy and attitudes towards robots; (2) BHNU has a direct influence on attitudes towards robots and an indirect influence on the intention to work with social robots through attitudes towards robots and robot use self-efficacy. Our findings expand the current research on the impact of perceived robot use self-efficacy on intention to work with social robots. Implications for human robot interaction and human resource management are discussed. Full article

► Show Figures

Figure 1

26 pages, 2661 KiB

Open AccessSystematic Review

Managing ADHD Symptoms in Children Through the Use of Various Technology-Driven Serious Games: A Systematic Review

by Aikaterini Doulou, Pantelis Pergantis, Athanasios Drigas and Charalampos Skianis

Multimodal Technol. Interact. 2025, 9(1), 8; https://doi.org/10.3390/mti9010008 - 16 Jan 2025

Abstract

Children with attention deficit hyperactivity disorder (ADHD) frequently experience impairments in a range of abilities. Due to their poor attention and concentration, they find it challenging to stay focused when learning. They need help to retain the directions given by teachers and are [...] Read more.

Children with attention deficit hyperactivity disorder (ADHD) frequently experience impairments in a range of abilities. Due to their poor attention and concentration, they find it challenging to stay focused when learning. They need help to retain the directions given by teachers and are very animated. Focus issues, hyperactivity, and attention problems may hamper learning. The needs and challenges of children with ADHD have been addressed by numerous digital solutions over the years. These solutions support a variety of needs (e.g., diagnosing versus treating), aim to address a variety of goals (e.g., addressing inattention, impulsivity, working memory, executive functions, emotion regulation), and employ a wide range of technologies, including video games, PC, mobile, web, AR, VR, tangible interfaces, wearables, robots, and BCI/neurofeedback, occasionally even in tandem. According to studies on the psychological impacts of serious games, immersive games can potentially be valuable tools for treating ADHD. This research investigates using PC, mobile/tablet applications, augmented reality, virtual reality, and brain–computer interfaces to develop executive functions and metacognitive and emotional competencies in children with ADHD through serious games. Following PRISMA 2020 criteria, this systematic review includes a comprehensive search of the PubMed, Web of Science, Scopus, and Google Scholar databases. The database search provided 784 records, and 30 studies met the inclusion criteria. The results showed that serious games assisted by multiple technologies could significantly improve a wide range of cognitive and socioemotional meta-competencies among children with ADHD, including visuospatial working memory, attention, inhibition control, cognitive flexibility, planning/organizing, problem-solving, social communication, and emotional regulation. The results of this review may provide positive feedback for creating more inclusive digital training environments for the treatment of ADHD children. Full article

► Show Figures

Figure 1

10 pages, 1157 KiB

Open AccessArticle

The Effectiveness of a Virtual Reality-Based Exergame Protocol in Improving Postural Balance in Older Adults During the COVID-19 Pandemic

by Valeska Gatica-Rojas, María Isabel Camoglino-Escobar, Hernán Carrillo-Bestagno and Ricardo Cartes-Velásquez

Multimodal Technol. Interact. 2025, 9(1), 7; https://doi.org/10.3390/mti9010007 - 15 Jan 2025

Abstract

Background: The COVID-19 pandemic significantly reduced physical activity levels, particularly among older people, negatively impacting their postural balance and increasing the risk of falls and hip fractures. This study aims to assess the effect of a virtual reality-based exergame physical activity protocol at [...] Read more.

Background: The COVID-19 pandemic significantly reduced physical activity levels, particularly among older people, negatively impacting their postural balance and increasing the risk of falls and hip fractures. This study aims to assess the effect of a virtual reality-based exergame physical activity protocol at home on improving postural balance in older people. Materials and Methods: A quasi-experimental design was employed with 10 older people (71 ± 9 years) who participated in a virtual reality-based exergame physical activity protocol consisting of eighteen 25 min sessions conducted at home. The protocol incorporated 3D movement tracking using a sensor attached to the participants’ bodies to monitor postural sway in real time. Clinical measurements included the Timed Up and Go test and posturographic measures of center-of-pressure, including sway area, velocity, and standard deviation in the mediolateral and anteroposterior directions under four conditions: static with the eyes open and eyes closed and dynamic voluntary sway in the mediolateral direction following a 30 Hz metronome with the eyes open and eyes closed. Paired t-tests were used to compare pre- and post-intervention data. Results: The intervention led to significant improvements in postural balance as measured using both posturographic measures (p < 0.05) and the Timed Up and Go test (p = 0.04). Conclusion: The virtual reality-based exergame physical activity protocol conducted at home, comprising eighteen 25 min sessions, effectively improves postural balance in older people. Full article

(This article belongs to the Special Issue 3D User Interfaces and Virtual Reality—2nd Edition)

► Show Figures

Figure 1

32 pages, 475 KiB

Open AccessReview

Multimodal Interaction, Interfaces, and Communication: A Survey

by Elias Dritsas, Maria Trigka, Christos Troussas and Phivos Mylonas

Multimodal Technol. Interact. 2025, 9(1), 6; https://doi.org/10.3390/mti9010006 - 14 Jan 2025

Abstract

Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in [...] Read more.

Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in various applications, including virtual assistants, intelligent environments, healthcare, and accessibility technologies. This survey concisely overviews recent advancements in multimodal interaction, interfaces, and communication. It delves into integrating different input and output modalities, focusing on critical technologies and essential considerations in multimodal fusion, including temporal synchronization and decision-level integration. Furthermore, the survey explores the challenges of developing context-aware, adaptive systems that provide seamless and intuitive user experiences. Lastly, by examining current methodologies and trends, this study underscores the potential of multimodal systems and sheds light on future research directions. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

12 pages, 2155 KiB

Open AccessArticle

Human–Robot Interactions: A Pilot Study of Psychoaffective and Cognitive Factors to Boost the Acceptance and Usability of Assistive Wearable Devices

by Margherita Bertuccelli, Stefano Tortora, Edoardo Trombin, Liliana Negri, Patrizia Bisiacchi, Emanuele Menegatti and Alessandra Del Felice

Multimodal Technol. Interact. 2025, 9(1), 5; https://doi.org/10.3390/mti9010005 - 9 Jan 2025

Abstract

Robotic technology to assist rehabilitation provides practical advantages compared with traditional rehabilitation treatments, but its efficacy is still disputed. This controversial effectiveness is due to different factors, including a lack of guidelines to adapt devices to users’ individual needs. These needs include the [...] Read more.

Robotic technology to assist rehabilitation provides practical advantages compared with traditional rehabilitation treatments, but its efficacy is still disputed. This controversial effectiveness is due to different factors, including a lack of guidelines to adapt devices to users’ individual needs. These needs include the specific clinical conditions of people with disabilities, as well as their psychological and cognitive profiles. This pilot study aims to investigate the relationships between psychological, cognitive, and robot-related factors playing a role in human–robot interaction to promote a human-centric approach in robotic rehabilitation. Ten able-bodied volunteers were assessed for their anxiety, experienced workload, cognitive reserve, and perceived exoskeleton usability before and after a task with a lower-limb exoskeleton (i.e., 10 m path walking for 10 trials). Pre-trial anxiety levels were higher than post-trial ones (p < 0.01). While trait anxiety levels were predictive of the experienced effort (Adjusted-r² = 0.43, p = 0.02), the state anxiety score was predictive of the perceived overall workload (Adjusted-r² = 0.45, p = 0.02). High–average cognitive reserve scores were predictive of the perception of exoskeleton usability (Adjusted-r² = 0.45, p = 0.02). A negative correlation emerged between the workload and the perception of personal identification with the exoskeleton (r = −0.67, p-value = 0.03). This study provides preliminary evidence of the impact of cognitive and psychoaffective factors on the perception of workload and overall device appreciation in exoskeleton training. It also suggests pragmatic measures such as familiarization time to reduce anxiety and end-user selection based on cognitive profiles. These assessments may provide guidance on the personalization of training. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

30 pages, 8717 KiB

Open AccessArticle

Development and Evaluation of a Low-Jitter Hand Tracking System for Improving Typing Efficiency in a Virtual Reality Workspace

by Tianshu Xu, Wen Gu, Koichi Ota and Shinobu Hasegawa

Multimodal Technol. Interact. 2025, 9(1), 4; https://doi.org/10.3390/mti9010004 - 8 Jan 2025

Abstract

Virtual reality technology promises to transform immersive experiences across various applications, particularly within office environments. Despite its potential, the challenge of achieving efficient text entry in virtual reality persists. This study addresses this obstacle by introducing a novel machine learning-based solution, namely, the [...] Read more.

Virtual reality technology promises to transform immersive experiences across various applications, particularly within office environments. Despite its potential, the challenge of achieving efficient text entry in virtual reality persists. This study addresses this obstacle by introducing a novel machine learning-based solution, namely, the two-stream long short-term memory typing method, to enhance text entry performance in virtual reality. The two-stream long short-term memory method utilizes the back-of-the-hand image, employing a long short-term memory network and a Kalman filter to enhance hand position tracking accuracy and minimize jitter. Through statistical analysis of the data collected in the experiment and questionnaire results, we confirmed the effectiveness of the proposed method. In addition, we conducted an extra experiment to explore the differences in users’ typing behavior between regular typing and virtual reality-based typing. This additional experiment provides valuable insights into how users adapt their typing behavior in different environments. These findings represent a significant step in advancing text entry within virtual reality, setting the stage for immersive work experiences in office environments and beyond. Full article

► Show Figures

Figure 1

47 pages, 6401 KiB

Open AccessReview

A Comprehensive Review of Dropout Prediction Methods Based on Multivariate Analysed Features of MOOC Platforms

by Saad Alghamdi, Ben Soh and Alice Li

Multimodal Technol. Interact. 2025, 9(1), 3; https://doi.org/10.3390/mti9010003 - 7 Jan 2025

Abstract

Massive open online courses have revolutionised the learning environment, but their effectiveness is undermined by low completion rates. Traditional dropout prediction models in MOOCs often overlook complex factors like temporal dependencies and context-specific variables. These models are not adaptive enough to manage the [...] Read more.

Massive open online courses have revolutionised the learning environment, but their effectiveness is undermined by low completion rates. Traditional dropout prediction models in MOOCs often overlook complex factors like temporal dependencies and context-specific variables. These models are not adaptive enough to manage the dynamic nature of MOOC learning environments, resulting in inaccurate predictions and ineffective interventions. Accordingly, MOOCs dropout prediction models require more sophisticated artificial intelligence models that can address these limitations. Moreover, incorporating feature selection methods and explainable AI techniques can enhance the interpretability of these models, making them more actionable for educators and course designers. This paper provides a comprehensive review of various MOOCs dropout prediction methodologies, focusing on their strategies and research gaps. It highlights the growing MOOC environment and the potential for technology-driven gains in outcome accuracy. This review also discusses the use of advanced models based on machine learning, deep learning, and meta-heuristics approaches to improve course completion rates, optimise learning outcomes, and provide personalised educational experiences. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

26 pages, 380 KiB

Open AccessReview

How Architecture Builds Intelligence: Lessons from AI

by Nikos A. Salingaros

Multimodal Technol. Interact. 2025, 9(1), 2; https://doi.org/10.3390/mti9010002 - 27 Dec 2024

Abstract

The architecture in the title refers to physical buildings, spaces, and walls. Dominant architectural culture prefers minimalist environments that contradict the information setting needed for the infant brain to develop. Much of world architecture after World War II is therefore unsuitable for raising [...] Read more.

The architecture in the title refers to physical buildings, spaces, and walls. Dominant architectural culture prefers minimalist environments that contradict the information setting needed for the infant brain to develop. Much of world architecture after World War II is therefore unsuitable for raising children. Data collected by technological tools, including those that use AI for processing signals, indicate a basic misfit between cognition and design. Results from the way AI software works in general, together with mobile robotics and neuroscience, back up this conclusion. There exists a critical research gap: the systematic investigation of how the geometry of the built environment influences cognitive development and human neurophysiology. While previous studies have explored environmental effects on health (other than from pathogens and pollutants), they largely focus on factors such as acoustics, color, and light, neglecting the fundamental role of spatial geometry. Geometrical features in the ancestral setting shaped neural circuits that determine human cognition and intelligence. However, the contemporary built environment consisting of raw concrete, plate glass, and exposed steel sharply contrasts with natural geometries. Traditional and vernacular architectures are appropriate for life, whereas new buildings and urban spaces adapt to human biology and are better for raising children only if they follow living geometry, which represents natural patterns such as fractals and nested symmetries. This study provides a novel, evidence-based framework for adaptive and empathetic architectural design. Full article

36 pages, 16215 KiB

Open AccessArticle

Smart City Products and Their Materials Assessment Using the Pentagon Framework

by Pedro Ponce, Mario Rojas, Juana Isabel Mendez, Brian Anthony, Russel Bradley and Aminah Robinson Fayek

Multimodal Technol. Interact. 2025, 9(1), 1; https://doi.org/10.3390/mti9010001 - 25 Dec 2024

Abstract

Smart cities are complex urban environments that rely on advanced technology and data analytics to enhance city services’ quality of life, sustainability, and efficiency. As these cities continue to evolve, there is a growing need for a structured framework to evaluate and integrate [...] Read more.

Smart cities are complex urban environments that rely on advanced technology and data analytics to enhance city services’ quality of life, sustainability, and efficiency. As these cities continue to evolve, there is a growing need for a structured framework to evaluate and integrate products that align with smart city objectives. This paper introduces the Pentagon Framework, a comprehensive evaluation method designed to ensure that products and their materials meet the specific needs of smart cities. The framework focuses on five key features—smart, sustainable, sensing, social, and safe—collectively called the Penta-S concept. These features provide a structured approach to categorizing and assessing products, ensuring alignment with the city’s goals for efficiency, sustainability, and user experience. The Smart City Pentagon Framework Analyzer is also presented, a dedicated web application that facilitates interaction with the framework. It allows product data input, provides feedback on alignment with the Penta-S features, and suggests personality traits based on the OCEAN model. Complementing the web application, the Smart City Penta-S Compliance Assistant API, developed through ChatGPT, offers a more profound, personalized evaluation of products, including the life cycle phase recommendations using the IPPMD model. This paper contributes to the development of smart city solutions by providing a flexible framework that can be applied to any product type, optimizing its life cycle, and ensuring compliance with the Pentagon Framework. This approach improves product integration and fosters user satisfaction by tailoring products and their materials to meet specific user preferences and needs within the smart city environment. The proposed framework emphasizes citizen-centric design and highlights its advantages over conventional evaluation methods, ultimately enhancing urban planning and smart city development. Full article

► Show Figures

Figure 1

23 pages, 776 KiB

Open AccessSystematic Review

Performance of Commercial Deep Learning-Based Auto-Segmentation Software for Breast Cancer Radiation Therapy Planning: A Systematic Review

by Curtise K. C. Ng

Multimodal Technol. Interact. 2024, 8(12), 114; https://doi.org/10.3390/mti8120114 - 20 Dec 2024

Abstract

As yet, no systematic review on commercial deep learning-based auto-segmentation (DLAS) software for breast cancer radiation therapy (RT) planning has been published, although NRG Oncology has highlighted the necessity for such. The purpose of this systematic review is to investigate the performances of [...] Read more.

As yet, no systematic review on commercial deep learning-based auto-segmentation (DLAS) software for breast cancer radiation therapy (RT) planning has been published, although NRG Oncology has highlighted the necessity for such. The purpose of this systematic review is to investigate the performances of commercial DLAS software packages for breast cancer RT planning and methods for their performance evaluation. A literature search was conducted with the use of electronic databases. Fifteen papers met the selection criteria and were included. The included studies evaluated eight software packages (Limbus Contour, Manteia AccuLearning, Mirada DLCExpert, MVision.ai Contour+, Radformation AutoContour, RaySearch RayStation, Siemens syngo.via RT Image Suite/AI-Rad Companion Organs RT, and Therapanacea Annotate). Their findings show that the DLAS software could contour ten organs at risk (body, contralateral breast, esophagus-overlapping area, heart, ipsilateral humeral head, left and right lungs, liver, and sternum and trachea) and three clinical target volumes (CTVp_breast, CTVp_chestwall, and CTVn_L1) up to the clinically acceptable standard. This can contribute to 45.4%–93.7% contouring time reduction per patient. Although NRO Oncology has suggested that every clinical center should conduct its own DLAS software evaluation before clinical implementation, such testing appears particularly crucial for Manteia AccuLearning, Mirada DLCExpert, and MVision.ai Contour+ as a result of the methodological weaknesses of the corresponding studies such as the use of small datasets collected retrospectively from single centers for the evaluation. Full article

► Show Figures

Figure 1

13 pages, 1025 KiB

Open AccessArticle

Virtual Reality-Based Approach to Evaluate Emotional Everyday Scenarios for a Digital Health Application

by Valentin Wunsch, Effi Freya Picka, Hanna Schumm, Joshua Kopp, Tamer Abdulbaki Alshirbaji, Herag Arabian, Knut Möller and Verena Wagner-Hartl

Multimodal Technol. Interact. 2024, 8(12), 113; https://doi.org/10.3390/mti8120113 - 20 Dec 2024

Abstract

Social interactions are a part of our everyday lives. This can be challenging for individuals who experience social interactions as demanding, such as persons with autism spectrum disorder (ASD). Therefore, different types of training exist to help individuals affected by ASD practice in [...] Read more.

Social interactions are a part of our everyday lives. This can be challenging for individuals who experience social interactions as demanding, such as persons with autism spectrum disorder (ASD). Therefore, different types of training exist to help individuals affected by ASD practice in challenging situations. Digital applications offer advantages over traditional training because they can better address the individual needs of people with ASD. The development of a therapeutic application initially requires identifying appropriate emotion-relevant scenarios of social interaction. Based on a previous study evaluating text-based scenarios with different levels of complexity, a virtual reality (VR) environment was developed to assess the applicability of the scenarios in VR. Therefore, an experimental study was conducted. Two different scenarios of social interaction, each with four different levels of complexity, were presented and evaluated by 18 participants (10 males, eight females). A multidimensional approach was used to combine subjective assessments and psychophysiological measures (ECG and EDA). The results showed that the implementation of the scenarios in VR was able to differentiate between different levels of complexity. As the long-term target is to implement the findings in a therapeutic application for people with ASD, the results of the study are promising for the achievement of this goal. Full article

► Show Figures

Figure 1

26 pages, 10443 KiB

Open AccessArticle

Metaverse-Based Evacuation Training: Design, Implementation, and Experiment Focusing on Earthquake Evacuation

by Hiroyuki Mitsuhara

Multimodal Technol. Interact. 2024, 8(12), 112; https://doi.org/10.3390/mti8120112 - 20 Dec 2024

Abstract

Virtual reality (VR) can realize evacuation training in an immersive, interactive, safe, three-dimensional virtual world. Many VR-based evacuation training systems have been developed; however, they typically notify participants explicitly or implicitly before the evacuation training; thus, participants are mentally ready for successful evacuation. [...] Read more.

Virtual reality (VR) can realize evacuation training in an immersive, interactive, safe, three-dimensional virtual world. Many VR-based evacuation training systems have been developed; however, they typically notify participants explicitly or implicitly before the evacuation training; thus, participants are mentally ready for successful evacuation. To satisfy a prerequisite where participants do not have mental readiness, this study proposes a prototype of a metaverse-based evacuation training system called “Metavearthquake”. The main characteristic of the proposed prototype system is that evacuation training begins unexpectedly due to a sudden earthquake in the metaverse (virtual world); participants are then required to evacuate to a safe place while making decisions under difficult earthquake-caused situations. The prototype system introduces scenarios and nonplayable characters to express difficult situations that may occur after an earthquake occurrence. To heighten training effects, the prototype system supports reflection (reflection-on-action) by replaying the evacuation of participants. An experiment implied that a sudden earthquake is indispensable for realistic simulated evacuation experiences. In summary, Metavearthquake is a metaverse-based evacuation training system that provides realistic simulated earthquake evacuation experiences in terms of evacuation behaviors, emotions, and training effects. Full article

(This article belongs to the Special Issue Effectiveness of Serious Games in Risk Communication of Natural Disasters)

► Show Figures

Graphical abstract

22 pages, 7210 KiB

Open AccessArticle

Unlocking Trust and Acceptance in Tomorrow’s Ride: How In-Vehicle Intelligent Agents Redefine SAE Level 5 Autonomy

by Cansu Demir, Alexander Meschtscherjakov and Magdalena Gärtner

Multimodal Technol. Interact. 2024, 8(12), 111; https://doi.org/10.3390/mti8120111 - 17 Dec 2024

Abstract

As fully automated vehicles (FAVs) advance towards SAE Level 5 automation, the role of in-vehicle intelligent agents (IVIAs) in shaping passenger experience becomes critical. Even at SAE Level 5 automation, effective communication between the vehicle and the passenger will remain crucial to ensure [...] Read more.

As fully automated vehicles (FAVs) advance towards SAE Level 5 automation, the role of in-vehicle intelligent agents (IVIAs) in shaping passenger experience becomes critical. Even at SAE Level 5 automation, effective communication between the vehicle and the passenger will remain crucial to ensure a sense of safety, trust, and engagement. This study explores how different types and combinations of information provided by IVIAs influence user experience, acceptance, and trust. A sample of 25 participants was recruited for the study, which experienced a fully automated ride in a driving simulator, interacting with Iris, an IVIA designed for voice-only communication. The study utilized both qualitative and quantitative methods to assess participants’ perceptions. Findings indicate that critical and vehicle-status-related information had the highest positive impact on trust and acceptance, while personalized information, though valued, raised privacy concerns. Participants showed high engagement with non-driving-related activities, reflecting a high level of trust in the FAV’s performance. Interaction with the anthropomorphic IVIA was generally well received, but concerns about system transparency and information overload were noted. The study concludes that IVIAs play a crucial role in fostering passenger trust in FAVs, with implications for future design enhancements that emphasize emotional intelligence, personalization, and transparency. These findings contribute to the ongoing development of IVIAs and the broader adoption of automated driving technologies. Full article

(This article belongs to the Special Issue Cooperative Intelligence in Automated Driving-2nd Edition)

► Show Figures

Figure 1

35 pages, 5660 KiB

Open AccessArticle

“Warning!” Benefits and Pitfalls of Anthropomorphising Autonomous Vehicle Informational Assistants in the Case of an Accident

by Christopher D. Wallbridge, Qiyuan Zhang, Victoria Marcinkiewicz, Louise Bowen, Theodor Kozlowski, Dylan M. Jones and Phillip L. Morgan

Multimodal Technol. Interact. 2024, 8(12), 110; https://doi.org/10.3390/mti8120110 - 5 Dec 2024

Abstract

Despite the increasing sophistication of autonomous vehicles (AVs) and promises of increased safety, accidents will occur. These will corrode public trust and negatively impact user acceptance, adoption and continued use. It is imperative to explore methods that can potentially reduce this impact. The [...] Read more.

Despite the increasing sophistication of autonomous vehicles (AVs) and promises of increased safety, accidents will occur. These will corrode public trust and negatively impact user acceptance, adoption and continued use. It is imperative to explore methods that can potentially reduce this impact. The aim of the current paper is to investigate the efficacy of informational assistants (IAs) varying by anthropomorphism (humanoid robot vs. no robot) and dialogue style (conversational vs. informational) on trust in and blame on a highly autonomous vehicle in the event of an accident. The accident scenario involved a pedestrian violating the Highway Code by stepping out in front of a parked bus and the AV not being able to stop in time during an overtake manoeuvre. The humanoid (Nao) robot IA did not improve trust (across three measures) or reduce blame on the AV in Experiment 1, although communicated intentions and actions were perceived by some as being assertive and risky. Reducing assertiveness in Experiment 2 resulted in higher trust (on one measure) in the robot condition, especially with the conversational dialogue style. However, there were again no effects on blame. In Experiment 3, participants had multiple experiences of the AV negotiating parked buses without negative outcomes. Trust significantly increased across each event, although it plummeted following the accident with no differences due to anthropomorphism or dialogue style. The perceived capabilities of the AV and IA before the critical accident event may have had a counterintuitive effect. Overall, evidence was found for a few benefits and many pitfalls of anthropomorphising an AV with a humanoid robot IA in the event of an accident situation. Full article

(This article belongs to the Special Issue Cooperative Intelligence in Automated Driving-2nd Edition)

► Show Figures

Figure 1

27 pages, 15486 KiB

Open AccessArticle

Mixed-Presence Collaboration with Wall-Sized Displays: Empirical Findings on the Benefits of Awareness Cues

by Valérie Maquil, Adrien Coppens, Lou Schwartz and Dimitra Anastasiou

Multimodal Technol. Interact. 2024, 8(12), 109; https://doi.org/10.3390/mti8120109 - 5 Dec 2024

Abstract

Collaborative decision-making increasingly involves wall-sized displays (WSDs), allowing teams to view, analyse and discuss large amounts of data. To enhance workspace awareness for mixed-presence meetings, previous work proposes digital cues to share gestures, gaze, or entire postures. While several isolated cues were proposed [...] Read more.

Collaborative decision-making increasingly involves wall-sized displays (WSDs), allowing teams to view, analyse and discuss large amounts of data. To enhance workspace awareness for mixed-presence meetings, previous work proposes digital cues to share gestures, gaze, or entire postures. While several isolated cues were proposed and demonstrated useful in different workspaces, it is unknown whether results from previous studies can be transferred to a mixed-presence WSD context and to what extend such cues can be used in a combined way. In this paper, we report on the results from a user study with 24 participants (six groups of four participants), testing a mixed-presence collaboration scenario on two different setups of connected WSDs: audio-video link only vs. full setup with seven complementary cues. Our results show that the version with cues enhances workspace awareness, user experience, team orientation and coordination, and leads teams to take more correct decisions. Full article

► Show Figures

Figure 1

24 pages, 9111 KiB

Open AccessReview

Bi-Directional Gaze-Based Communication: A Review

by Björn Rene Severitt, Nora Castner and Siegfried Wahl

Multimodal Technol. Interact. 2024, 8(12), 108; https://doi.org/10.3390/mti8120108 - 4 Dec 2024

Abstract

Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. [...] Read more.

Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. In this review, we explore the state of the art in gaze-based communication, focusing on both directions: From user to system and from system to user. First, we examine how eye-tracking data is processed and utilized for communication from the user to the system. This includes a range of techniques for gaze-based interaction and the critical role of intent prediction, which enhances the system’s ability to anticipate the user’s needs. Next, we analyze the reverse pathway—how systems provide feedback to users via various channels, highlighting their advantages and limitations. Finally, we discuss the potential integration of these two communication streams, paving the way for more intuitive and efficient gaze-based interaction models, especially in the context of Artificial Intelligence. Our overview emphasizes the future prospects for combining these approaches to create seamless, trust-building communication between users and systems. Ensuring that these systems are designed with a focus on usability and accessibility will be critical to making them effective communication tools for a wide range of users. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Graphical abstract

47 pages, 3641 KiB

Open AccessReview

Innovative and Interactive Technologies in Creative Product Design Education: A Review

by Ioanna Nazlidou, Nikolaos Efkolidis, Konstantinos Kakoulis and Panagiotis Kyratsis

Multimodal Technol. Interact. 2024, 8(12), 107; https://doi.org/10.3390/mti8120107 - 4 Dec 2024

Abstract

When discussing the Education 4.0 concept and the role of technology-based learning systems along with creativity, it is interesting to explore how these are reflected as educational innovations in the case of design education. This study aims to provide an overview of interactive [...] Read more.

When discussing the Education 4.0 concept and the role of technology-based learning systems along with creativity, it is interesting to explore how these are reflected as educational innovations in the case of design education. This study aims to provide an overview of interactive technologies used in product design education and examine their integration into the learning process. A literature search was conducted, analyzing scientific papers to review relevant articles. The findings highlight several categories of technologies utilized in design education, including virtual and augmented reality, robotics, interactive embedded systems, immersive technologies, and computational intelligence systems. These technologies are primarily integrated as supportive tools throughout different stages of the design process within learning environments. This study suggests that integrating such technologies alongside pedagogical methods positively impacts education, offering numerous opportunities for further research and innovation. In conclusion, this review contributes to ongoing research in technological advancements and innovations in design education, offering insights into the diverse applications of interactive technologies in enhancing learning environments. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Graphical abstract

22 pages, 3907 KiB

Open AccessArticle

Science Mapping of AI as an Educational Tool Exploring Digital Inequalities: A Sociological Perspective

by Isotta Mac Fadden, Elena-María García-Alonso and Eloy López Meneses

Multimodal Technol. Interact. 2024, 8(12), 106; https://doi.org/10.3390/mti8120106 - 21 Nov 2024

Abstract

This study aims to explore the evolution of the literature on the sociological implications of integrating artificial intelligence (AI) as an educational tool, particularly its influence on digital inequalities. While AI technologies, such as AI-based language models, have begun transforming educational practices by [...] Read more.

This study aims to explore the evolution of the literature on the sociological implications of integrating artificial intelligence (AI) as an educational tool, particularly its influence on digital inequalities. While AI technologies, such as AI-based language models, have begun transforming educational practices by personalizing learning, fostering student autonomy, and supporting educators, concerns remain regarding access disparities, ethical implications, and the potential reinforcement of existing social inequalities. To address these issues, a bibliometric analysis employing science mapping was conducted on 1515 studies sourced from the Web of Science Core Collection. This analysis traces the thematic evolution of social science perspectives on AI’s role in education and its relationship with digital inequalities. The results indicate a growing academic interest in AI in education, with a notable progression from understanding its basic impact to exploring complex themes such as vulnerability, disability, bias, and community. The studies show that AI’s application has expanded from isolated research on specific populations to broader discussions on inclusivity, equity, and the impact of AI on governance, policy, and community. However, the findings also reveal a significant gap in sociological perspectives, particularly regarding issues like digital illiteracy and socio-economic access disparities. Although AI holds promise for promoting more inclusive education, further research is essential to address these sociological concerns and to guide the ethical, equitable implementation of AI as its influence on governance, policy, and community impact continues to grow. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

Journal Description

Multimodal Technologies and Interaction

Latest Articles

Journal Menu

Journal Browser

Highly Accessed Articles

Latest Books

E-Mail Alert

News

Topics

Conferences

Special Issues

Further Information

Guidelines

MDPI Initiatives

Follow MDPI