Multimodal Technologies and Interaction

10 pages, 236 KiB

Open AccessArticle

Reproducibility: A Researcher-Centered Definition

by Sebastian Stefan Feger and Paweł W. Woźniak

Multimodal Technol. Interact. 2022, 6(2), 17; https://doi.org/10.3390/mti6020017 - 20 Feb 2022

Cited by 3 | Viewed by 3454

Recent years have introduced major shifts in scientific reporting and publishing. The scientific community, publishers, funding agencies, and the public expect research to adhere to principles of openness, reproducibility, replicability, and repeatability. However, studies have shown that scientists often have neither the right [...] Read more.

Recent years have introduced major shifts in scientific reporting and publishing. The scientific community, publishers, funding agencies, and the public expect research to adhere to principles of openness, reproducibility, replicability, and repeatability. However, studies have shown that scientists often have neither the right tools nor suitable support at their disposal to meet these modern science challenges. In fact, even the concrete expectations connected to these terms may be unclear and subject to field-specific, organizational, and personal interpretations. Based on a narrative literature review of work that defines characteristics of open science, reproducibility, replicability, and repeatability, as well as a review of recent work on researcher-centered requirements, we find that the bottom-up practices and needs of researchers contrast top-down expectations encoded in terms related to reproducibility and open science. We identify and define reproducibility as a central term that concerns the ease of access to scientific resources, as well as their completeness, to the degree required for efficiently and effectively interacting with scientific work. We hope that this characterization helps to create a mutual understanding across science stakeholders, in turn paving the way for suitable and stimulating environments, fit to address the challenges of modern science reporting and publishing. Full article

(This article belongs to the Special Issue HCI for and within Open and Reproducible Science)

18 pages, 1633 KiB

Open AccessArticle

Exploring Data-Driven Components of Socially Intelligent AI through Cooperative Game Paradigms

by Casey Bennett, Benjamin Weiss, Jaeyoung Suh, Eunseo Yoon, Jihong Jeong and Yejin Chae

Multimodal Technol. Interact. 2022, 6(2), 16; https://doi.org/10.3390/mti6020016 - 17 Feb 2022

Cited by 9 | Viewed by 3404

Abstract

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents [...] Read more.

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways. Full article

(This article belongs to the Special Issue Speech-Based Interaction)

► Show Figures

Figure 1

19 pages, 1219 KiB

Open AccessArticle

Key Ergonomics Requirements and Possible Mechanical Solutions for Augmented Reality Head-Mounted Displays in Surgery

by Renzo D’Amato, Fabrizio Cutolo, Giovanni Badiali, Marina Carbone, Hao Lu, Harm Hogenbirk and Vincenzo Ferrari

Multimodal Technol. Interact. 2022, 6(2), 15; https://doi.org/10.3390/mti6020015 - 10 Feb 2022

Cited by 5 | Viewed by 3989

Abstract

In the context of a European project, we identified over 150 requirements for the development of an augmented reality (AR) head-mounted display (HMD) specifically tailored to support highly challenging manual surgical procedures. The requirements were established by surgeons from different specialties and by [...] Read more.

In the context of a European project, we identified over 150 requirements for the development of an augmented reality (AR) head-mounted display (HMD) specifically tailored to support highly challenging manual surgical procedures. The requirements were established by surgeons from different specialties and by industrial players working in the surgical field who had strong commitments to the exploitation of this technology. Some of these requirements were specific to the project, while others can be seen as key requirements for the implementation of an efficient and reliable AR headset to be used to support manual activities in the peripersonal space. The aim of this work is to describe these ergonomic requirements that impact the mechanical design of the HMDs, the possible innovative solutions to these requirements, and how these solutions have been used to implement the AR headset in surgical navigation. We also report the results of a preliminary qualitative evaluation of the AR headset by three surgeons. Full article

(This article belongs to the Special Issue Feature Papers of MTI in 2021)

► Show Figures

Graphical abstract

20 pages, 2821 KiB

Open AccessArticle

Reducing the Gap between Mental Models of Truck Drivers and Adaptive User Interfaces in Commercial Vehicles

by Lasse Schölkopf, Camilla Gatto von der Heyde, Anna Sprung and Frank Diermeyer

Multimodal Technol. Interact. 2022, 6(2), 14; https://doi.org/10.3390/mti6020014 - 29 Jan 2022

Cited by 1 | Viewed by 3491

Abstract

Whether or not a product matches the user’s mental model and therefore his understanding of how it works influences the perceived usability. Therefore, it is beneficial if an interface is based on the user’s initial mental model, hence it works just as expected. [...] Read more.

Whether or not a product matches the user’s mental model and therefore his understanding of how it works influences the perceived usability. Therefore, it is beneficial if an interface is based on the user’s initial mental model, hence it works just as expected. If it contradicts preexisting models, operating errors and frustrated users are to be expected. This work proposes a method to increase the probability of correspondence between a developed product and the user’s mental model by addressing a common source of error in the product development process: Product designers assuming their own mental model matches the user’s. The process was demonstrated using the example of an adaptive user interface for commercial vehicles. A questionnaire was used to identify the underlying dimensions of the user group’s mental model of adaptive user interfaces. By conducting two expert workshops and a user survey with 75 truck drivers, a questionnaire consisting of 37 items and four dimensions was constructed. Thereby, the initial mental model of truck drivers regarding an adaptive user interface for commercial vehicles was determined. Full article

► Show Figures

Figure 1

11 pages, 1093 KiB

Open AccessArticle

Unplugged Activities in Cross-Curricular Teaching: Effect on Sixth Graders’ Computational Thinking and Learning Outcomes

by José Miguel Merino-Armero, José Antonio González-Calero, Ramón Cózar-Gutiérrez and Javier del Olmo-Muñoz

Multimodal Technol. Interact. 2022, 6(2), 13; https://doi.org/10.3390/mti6020013 - 28 Jan 2022

Cited by 11 | Viewed by 5515

Abstract

There is a debate about the way to introduce computational thinking (CT) in schools. Different proposals are on the table; these include the creation of new computational areas for developing CT, the introduction of CT in STEM areas, and the cross-curricular integration of [...] Read more.

There is a debate about the way to introduce computational thinking (CT) in schools. Different proposals are on the table; these include the creation of new computational areas for developing CT, the introduction of CT in STEM areas, and the cross-curricular integration of CT in schools. There is also concern that no student should be left behind, independently of their economic situation. To this effect, an unplugged approach is the most cost-effective solution. In addition, this topic is interesting in the context of a pandemic situation that has prevented the sharing of materials between students. This study analyzes an unplugged cross-curricular introduction of CT in the Social Sciences area among sixth grade students. A group of 14 students was selected to carry out an unplugged intervention design—where they were required to program an imaginary robot on paper—in the Social Sciences area. Their CT development and academic results were compared to those of 31 students from the control group who continued attending regular classes. Results showed that an unplugged teaching style of CT in Social Sciences lessons significantly increased CT (p < 0.001) and with a large effect size (d = 1.305) without differences in students’ academic achievement. The findings show that children can potentially develop their CT in non-STEM lessons, learning the same curricular contents, and maintaining their academic results. Full article

► Show Figures

Figure 1

2 pages, 152 KiB

Open AccessEditorial

Acknowledgment to Reviewers of Multimodal Technologies and Interaction in 2021

by Multimodal Technologies and Interaction Editorial Office

Multimodal Technol. Interact. 2022, 6(2), 12; https://doi.org/10.3390/mti6020012 - 28 Jan 2022

Viewed by 2081

Abstract

Rigorous peer-reviews are the basis of high-quality academic publishing [...] Full article

23 pages, 8869 KiB

Open AccessArticle

End-to-End Modeling and Transfer Learning for Audiovisual Emotion Recognition in-the-Wild

by Denis Dresvyanskiy, Elena Ryumina, Heysem Kaya, Maxim Markitantov, Alexey Karpov and Wolfgang Minker

Multimodal Technol. Interact. 2022, 6(2), 11; https://doi.org/10.3390/mti6020011 - 27 Jan 2022

Cited by 22 | Viewed by 4446

Abstract

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely “in-the-wild” data. [...] Read more.

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely “in-the-wild” data. This work investigates audiovisual deep learning approaches to emotion recognition in in-the-wild problem. Inspired by the outstanding performance of end-to-end and transfer learning techniques, we explored the effectiveness of architectures in which a modality-specific Convolutional Neural Network (CNN) is followed by a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) using the AffWild2 dataset under the Affective Behavior Analysis in-the-Wild (ABAW) challenge protocol. We deployed unimodal end-to-end and transfer learning approaches within a multimodal fusion system, which generated final predictions using a weighted score fusion scheme. Exploiting the proposed deep-learning-based multimodal system, we reached a test set challenge performance measure of 48.1% on the ABAW 2020 Facial Expressions challenge, which advances the first-runner-up performance. Full article

► Show Figures

Figure 1

20 pages, 1350 KiB

Open AccessArticle

The Pedagogical Value of Creating Accessible Games: A Case Study with Higher Education Students

by Carla Sousa, José Carlos Neves and Manuel José Damásio

Multimodal Technol. Interact. 2022, 6(2), 10; https://doi.org/10.3390/mti6020010 - 25 Jan 2022

Cited by 5 | Viewed by 3993

Abstract

The potential of games in empowering underrepresented groups is a central theme in the field of media studies. However, to ensure that everyone can benefit from them, it is necessary to ensure that they are inclusive and accessible. In the present work, we [...] Read more.

The potential of games in empowering underrepresented groups is a central theme in the field of media studies. However, to ensure that everyone can benefit from them, it is necessary to ensure that they are inclusive and accessible. In the present work, we have implemented a Participatory Action Research (PAR) approach to target this problem, right at the stage of training new game designers and game developers. Thus, through a game-design-based or experimental game design pedagogical approach, we intended to promote inclusive and accessibility-driven game design and development skills in students, while decreasing their negative attitudes towards people with Intellectual Disability (pwID). A protocol with a Non-Governmental Organization in the field was established and, during two semesters, students were challenged to develop 10 accessible games and physical interfaces for pwID and motor disabilities, through participatory processes. Pre and post assessment was conducted, through a mixed-method approach. After participating in this process, students reported satisfaction and increased knowledge of cognitive and motor accessibility, and inclusive game design in general. Moreover, the attitudinal assessment showed significantly lower levels of prejudiced beliefs towards pwID (p < 0.05). Full article

(This article belongs to the Special Issue Innovations in Game-Based Learning)

► Show Figures

Figure 1

22 pages, 1844 KiB

Open AccessArticle

Potentials and Challenges in Students’ Meaning-Making via Sign Systems

by Karoline Schnaider and Limin Gu

Multimodal Technol. Interact. 2022, 6(2), 9; https://doi.org/10.3390/mti6020009 - 21 Jan 2022

Cited by 4 | Viewed by 3328

Abstract

The relationship between sign systems and the meaning potentials and affordances of multimodal technologies has received increasing attention in research on digital technology use in education. Students constantly adhere to and engage with semiotic shifts in sign systems when they work with digital [...] Read more.

The relationship between sign systems and the meaning potentials and affordances of multimodal technologies has received increasing attention in research on digital technology use in education. Students constantly adhere to and engage with semiotic shifts in sign systems when they work with digital technologies for learning purposes. This study explores students’ use of digital technologies in Swedish schools. We trace the way semiotic activity systems and cognitive processes are transformed and realized when students engage with shifts in sign systems into various meaning-making strategies. Methodologically, the study is based on a data set of video recordings, interviews, and observations of classroom practice in three primary schools. An analysis that draws on quantitative ethnography was applied to process and analyse the data. The main findings revealed that sign systems prompted by the technologies and the social space compete to some extent for students’ attention, and that technology design is monotonously rendering lower levels of mediation. These findings show that various sign system prompts need to be balanced and streamlined to support students in their meaning-making. This article conveys the importance of understanding sign systems, as they are the most common resources for technology-assisted learning, and change the prerequisites for meaning-making. Full article

► Show Figures

Graphical abstract

13 pages, 1110 KiB

Open AccessArticle

Digital Escape Rooms as Game-Based Learning Environments: A Study in Sex Education

by Lena von Kotzebue, Joerg Zumbach and Anna Brandlmayr

Multimodal Technol. Interact. 2022, 6(2), 8; https://doi.org/10.3390/mti6020008 - 18 Jan 2022

Cited by 17 | Viewed by 8355

Abstract

Game-based learning is becoming increasingly popular in education. The playful experience especially promises a high degree of students’ motivation. In this research, we examine the influence of sequential scaffolding within a digital educational escape room game. Escape rooms are usually games where players [...] Read more.

Game-based learning is becoming increasingly popular in education. The playful experience especially promises a high degree of students’ motivation. In this research, we examine the influence of sequential scaffolding within a digital educational escape room game. Escape rooms are usually games where players have to escape from a room within a given time limit by completing different tasks and quests. Therefore, we developed an educational virtual escape room for biology classes, focusing on the topic of sex education. In an experiment, we modified this learning environment and developed two different conditions: in one escape room, scaffolding was implemented using sequential learning aids; in the other escape room, which was assigned to the control group, no additional learner support was provided. The main objective of this quantitative research is to measure the escape room’s impact on learning and cognitive load. In addition, motivation, flow experience and experience of immersion are analyzed. A comparison between the two escape rooms shows that additional scaffolding does not significantly increase cognitive load or have any effect on learning. Results show that motivation and knowledge acquisition can be successfully supported by using game-based learning with escape rooms. Full article

(This article belongs to the Special Issue Innovations in Game-Based Learning)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 6, Issue 2 (February 2022) – 10 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI