Integrating Virtual, Mixed, and Augmented Reality to Human–Robot Interaction Applications Using Game Engines: A Brief Review of Accessible Software Tools and Frameworks

Abstract

This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).

1. Introduction

Extended Reality (XR) is an umbrella concept encompassing the entire spectrum of methods able to alter reality through immersive technology, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) [1]. In robotics, XR applications can be developed on top of robot-specific simulators or game engines [2]. Robot-specific simulators are generally used for offline testing of robotic systems. As presented in [3,4], certain robot-specific simulators enable the creation of virtual reality environments. However, robot-specific simulators can limit the content, graphics quality, and supported hardware of developed applications. To explore the capabilities of XR technologies, researchers require software platforms compatible with new XR hardware that facilitates the generation of content for these devices as well as other programming and integration tasks [2]. In this context, game engines such as Unity and Unreal Engine provide several tools to facilitate the creation of 3D environments for the latest XR hardware devices. Moreover, a growing community is engaged in supporting and expanding these tools. Thus, in many cases game engines are considered a cost-effective and flexible solution for developing XR applications [5]. Nowadays, the ability of game engines to create advanced user experiences and facilitate usage of the latest XR hardware is motivating researchers and practitioners to extend the use of game engines for non-gaming applications [6], such as social and industrial robotics [7]. However, integrating game engines with robotic systems can be a complex task [8]. Moreover, many robotics researchers can have difficulties working with the novel concepts, frameworks, and tools required to use game engines in robotics applications. Therefore, the main objective of this article is to present relevant concepts, tools, and frameworks used or proposed in the literature for enabling the creation of human–robot XR experiences through general-purpose, accessible, and popular game engines, specifically Unity and Unreal Engine.

1.1. Related Surveys

Recently, several reviews presenting trends and challenges of XR technologies have been proposed for industrial areas; examples include product assembly, maintenance/repair [9], manufacturing training [10], quality control [11], and the automotive industry [12]. While robots can be relevant elements in these applications, human–robot interaction and game engines are not a primary focus of these reviews. Dianatfar et al. [13] presented a narrative review (i.e., one that does not follow a strict methodology to locate articles) of VR and AR solutions in human–robot collaboration. They summarized the main objectives, integration details, and results from 21 research articles proposing VR and AR solutions for collaborative and industrial robots published from 2012 to 2019, of which seven projects used the Unity game engine. Most recently, Suzuki et al. [14] presented a survey and taxonomy of augmented reality and robotics. The focus of [14] was to provide a holistic view that covers broader aspects of the Human–Computer Interaction (HCI) and Human–Robot Interaction (HRI) literature considering any robot-like or actuated system. Finally, Costa et al. [15] presented a systematic literature review of AR applications in HRC in industrial settings. These previous works mostly focus on identifying and classifying hardware devices, interaction modalities, metrics, or applications; to date, little attention has been paid to software aspects. Therefore, the present article attempts to fill this gap, specifically in social and industrial settings. Unlike previous works, this article focuses on reviewing research articles that use game engines as a keystone element in developing XR applications in robotics. The key differences between this article and previous works are summarized in

Table 1. Differences between this article and similar works recently presented in the robotics literature.

Reference	Year	Topics Focused on Each Paper	Differences with This Article
[13]	2021	Review of existing VR/AR solutions in HRC Solutions are presented in four main categories: operator support, instruction, simulation, and manipulation Referenced articles published from 2010 to 2019	Our article presents research questions and a systematic search methodology Our article presents articles published from 2015 to 2022 Our article focuses on solutions using game engines as a key technology
[15]	2022	Categorizes the recent literature on AR for HRC and industrial applications Identifies the main areas/sectors and employed AR technologies Highlights key benefits of AR	Our article reviews papers proposing a VR, MR, and/or AR solution Our article focuses on solutions using game engines as a key technology Our article reviews papers for industrial, social, and service applications Our article highlights reported hardware and software issues
[14]	2022	Synthesizes and categorizes AR solutions in the following dimensions: approaches to augmenting reality, characteristics of robots, purposes and benefits, classification of presented information, design components and strategies for visual augmentation, interaction techniques and modalities, application domains, and evaluation strategies The article reviews AR applications presenting any actuated or robot-like hardware (many of which are outside the scope of our review).	Our article reviews papers proposing VR solutions Our article focuses on solutions using game engines as a key technology Our article focuses on papers proposing a solution for industrial, social, and service applications Our article focuses on software aspects

.

1.2. Motivation and Contributions

Advances in emergent technologies such as XR are not static. The increasing popularity of XR technologies in recent years has motivated practitioners and researchers to develop new software artifacts to explore the capabilities of new hardware devices. Therefore, keeping this scenario of constant advancement in mind, in addition to the lack of a comprehensive study focused on the software aspects, there is a need to revisit the state-of-the-art solutions for social, service, and industrial robotics. Therefore, the main contribution of this article is:

The identification of novel and relevant software frameworks and tools for building advanced XR experiences in HRI using popular game engines.

2. Background and Article Organization

According to [16,17,18], Cyber-Physical Systems (CPSs) can be defined as multi-dimensional and complex systems that integrate computing and networks to govern physical actuators or machines, such as robots. CPSs are a core foundation of Industry 4.0 [19] based on the 3C (computing, communication, and control) approach. The main objective of CPSs is to provide different services (e.g., sensing, information feedback, and dynamic control) to enable reliable, safe, and efficient monitoring and control of physical entities [19]. In this context, digital twins represent an engineering category focusing on virtual models more than 3C capabilities [19]. In addition, ref. [20] demonstrated a proof of concept of digital twin usage in manufacturing. The proposed modules estimate a worker’s physical load in a parts-picking scenario and perform dynamic scheduling based on the predicted working progress under ergonomic constraints. In this context, the digital model and digital shadow are two related concepts. On the one hand, a digital model is described in [21] as a representation of reality constructed manually that remains static when reality changes. Therefore, there is no data exchange between the physical and virtual worlds. On the other hand, ref. [21] describes a digital shadow as “an automatically derived model that changes when reality changes”. However, in a digital shadow, “there is no automated real-time feedback loop”. This is because a digital shadow only provides a one-way data flow from the physical environment to the virtual model [22]. Digital twins go further than digital models and digital shadows, as their results directly trigger or impact reality. Therefore, digital twins provide a two-way data flow from the physical system, both to the virtual system and the other way around [22]. The primary interest of this review is to identify those software artifacts used in HRI applications where the virtual and physical systems can communicate with each other. Therefore, digital models and digital shadows are outside the scope of this article. Most recently, emergent human-centered paradigms, such as Society 5.0 and Industry 5.0, have been envisioned to promote the creation of Human–Cyber-Physical Systems (HCPs), where humans play a central position as “masters” of CPSs [23]. Therefore, the highest right to make decisions always comes from humans, no matter how powerful or sophisticated an intelligent system is. Novel perspectives on these human-centered paradigms have been described in [24,25]. The main focus of the present article is to review those human–robot interaction systems proposed in the literature that use game engines as keystone elements for the creation of complex systems that integrate software frameworks and libraries enabling computing (e.g., object and human recognition), communication between the different modules (e.g., using a socket library or middleware), and control of real robots.

2.1. Game Engines

A game engine is defined in [26] “as a collection of engines for graphics, physics, networking, artificial intelligence (AI), and scripting”. A more recent definition is proposed in [6] as an “integrated development environment (IDE) that enables game operators and software developers to create real-time and 3D rendering visualizations”. Therefore, at the most basic level, a game engine comprises a rendering side that enables the creation of 3D digital objects (composed of textures, materials, and lighting) and a software development side that allows developers to compile code and execute it on specific devices. While many game engines are available on the market [27], Unity and Unreal Engine in particular have emerged as relevant research tools for non-gaming applications. Below, we briefly review the differences between these game engines (see [28] for a more detailed comparison).

2.1.1. Unity

Unity is a popular and accessible cross-platform game engine enabling the creation of simulated 2D/3D environments and XR applications. The focus of Unity is to provide a general purpose game engine that supports different platforms and users [29]. Unity Projects are composed of assets that can be used to create a game or simulation, such as images, audio files, meshes (graphics primitives of Unity), or any other type of file supported in Unity. Assets can be imported to a Unity project using the Unity Package Manager (UPM) or from the Unity Asset Store. A game in Unity is composed of one or multiple scenes (game levels). Each scene contains a hierarchical attachment of basic containers, denoted as GameObjects. The functionality or behavior of GameObjects is defined by built-in components or custom components defined in C# scripts or external plugins [29].

2.1.2. Unreal Engine

Unreal Engine is a popular open and advanced 3D creation platform enabling developers to create immersive experiences and games using photorealistic visuals (i.e., 3D environments that look as realistic as possible). Similar to Unity, projects in Unreal Engine are composed of assets. Developers can import assets from other projects or download them from the Unreal assets marketplace. In Unity, Actors are the equivalent of GameObjects in Unreal Engine (i.e., elements that can be placed in the virtual world). Unlike Unity, which mainly uses C#, Unreal Engine uses C++. Therefore, developers require more programming experience in order to create applications. However, applications in Unreal Engine can be performed using a visual scripting tool denoted as Blueprints, which can be mixed with C++ scripts.

2.1.3. Paper Organization

Section 3 presents the methodology we followed to perform our systematic search for relevant research articles. Section 4 and Section 5 report the study’s results. On the one hand, Figure 1 shows a general system architecture summarizing the basic elements presented in the identified articles and an overview of the tools shown in Section 4. In this architecture, a set of sensors collect information from humans and the real-world environment. These sensors include RGB and 3D cameras, game and VR controllers, touch inputs, and wearable devices. Data are sent from the physical system to the virtual system and from the virtual system to the physical system using a communication mechanism (described in Section 4.1). In this review, we focus on those articles that explicitly describe the use of the Unity or Unreal game engines to build or execute virtual systems. A set of interaction tools (described in Section 4.2) process data obtained from sensors and change the real-world environment using robot control tools. Articles presenting this two-way data flow approach between the real and virtual world are the primary focus of this article. On the other hand, Section 5 presents an overview and brief description of applications using similar architectures. Section 6 presents challenges and research opportunities, followed by our conclusions.

3. Methodology

Scoping reviews are defined in [30] as “a type of knowledge synthesis, follow a systematic approach to map evidence on a topic and identify main concepts, theories, sources, and knowledge gaps”. This article follows the guidelines described in [31,32,33] for performing literature reviews in software engineering and scoping reviews [30]. Consequently, the main steps performed in this review are described below.

3.1. Definition of Research Questions

The research questions (RQs) guiding this study are:

1.

RQ1: Which type of XR applications or systems using real robots have been developed using game engines?

2.

RQ2: Which software frameworks or tools are commonly used or have been proposed to build these applications?

3.

RQ3: Which are the reported hardware and software limitations by the authors of these applications?

3.2. Study Selection, Quality Assessment, and Data Extraction

We limited the search to works published between 2015 and 2022 in order to focus on the most recent developments in this field. The search was performed in October 2022. A pilot search in the ACM and Elsevier databases indicated that after 2015 the interest of the robotics community in the use of game engines with robotic systems began to increase, surpassing a threshold of three papers per year. Keywords composing the search strings were defined using the PICO (Population, Intervention, Comparison, and Outcomes) method suggested in [31]. Several iterations and pilot searches were performed in order to determine an optimum set of primary studies. The strings and advanced settings defined for each database are shown in

Table 2. Advanced settings used for each database.

Database	Search String
IEEE Xplore	(Unity OR “Unreal engine”) AND Robot AND Human AND (Interaction OR Collaboration) AND (“Virtual reality” OR “Augmented reality” OR “Mixed reality”)
ACM Digital Library	[[All: unity] OR [All: “unreal engine”]] AND [Title: robot] AND [Abstract: human] AND [[Abstract: interaction] OR [Abstract: collaboration]]
Springer Link	(Unity OR “Unreal engine”) AND Robot AND Human AND (Interaction OR Collaboration) AND (“Virtual reality” OR “Augmented reality” OR “Mixed reality”) NOT Medicine NOT Surgery AND (Social OR Industry OR Manufacturing), Content type: Article
Science Direct	All: (Unity OR “Unreal engine”), Title and abstract: Robot AND Human AND (Interaction OR Collaboration OR Control), Subject areas: Engineering, Article type: Research articles
Web of Science	Robot AND Human AND (Interaction OR Collaboration) AND (“Virtual reality” OR “Augmented reality” OR “Mixed reality”), Citation topics: Robotics, and Human-Computer Interaction, Document Types: Article

.

In this article, the study selection is defined in two steps. In Step 1, we read the title and abstract of each article obtained from the search in the proposed databases. Then, an article was selected as a candidate to be included in the review if it met the following inclusion criteria:

The main focus of the article was to present an XR solution using a social or industrial robot and not other types of hardware artifacts such as drones, swarm robotics, autonomous vehicles, smartphones, intelligent speakers, gloves, or wearable devices.

In Step 2, we skimmed the full text of the articles obtained in Step 1. An article was excluded from the review if it met any of the exclusion criteria:

1.

The article focuses only on simulation (i.e., the robot is only displayed on a monitor) or there is no interaction between the user and a physical robot

2.

The article is a previous version of a more recent study or system proposing the same software architecture or framework by the same authors

3.

The article is a short paper, review article, thesis, technical report, or book chapter

4.

The article is a repeated article in different databases

5.

The article does not propose a solution using the Unreal Engine or Unity Engine

Additionally, quality assessment (QA) was defined using the following questions:

1.

Is the article published in a journal that is indexed on Web of Science?

2.

Was the article presented at a conference in the top twenty Google Scholar list for Robotics, Engineering and Computer Science, Human–Computer Interaction, or Manufacturing and Machines?

3.

Are the methods, hardware, and software architecture clearly described?

Articles that did not meet at least one of these quality criteria were excluded from the study.

Table 3. Number of studies per database and results after applying inclusion criteria (Step 1), exclusion criteria, and quality assessment criteria (Steps 2 and 3).

Database	Search Results	Articles Included	Articles Excluded	Articles Finally
		(Step 1)	(Step 2 and QA)	Included
IEEE Xplore	26	13	9	4
ACM Digital Library	29	6	1	5
Springer Link	137	11	8	3
Science Direct	99	16	8	8
Web of Science	117	23	18	4
Total	408	69	44	24

shows the number of articles processed in the proposed steps. To answer the proposed research questions, we used an Excel spreadsheet to extract relevant data from each reviewed article. The data extracted from the reviewed articles included year, venue, XR type, application context, software tools, hardware devices, main objective, and reported limitations.

3.3. Limitations of the Study

Factors limiting the validity of a literature review are described in [33,34]. These potential threats are difficult to eliminate entirely, as they involve human judgment [33]. For example, searches of the same research topic performed by different persons can end up with different sets of articles and extracted data due to researcher bias [33,35]. To reduce these problems, in this study researchers experienced in different areas of robotics, such as social and physical HRI, robot manipulation and control, and artificial intelligence, were involved in validation of the extracted data and conclusions.

4. Software Tools for Developing XR Applications in HRI

This section describes commonly used software tools reported to be integrated or used in the reviewed articles. We classified the identified software tools based on the 3C paradigm. However, we considered tools enabling computation (sensing and processing of data) and control of physical systems in the category of interaction. Figure 1 shows a general system architecture summarizing the main elements presented in the robotic systems reviewed in this article. In this architecture, a set of sensors collect information from interaction with the environment. Examples of data collected from sensors attached to robots include joint angles, physical interactions (from touch sensors), and forces, while examples of data obtained from humans include heart rate, inertial data, inputs from game controllers, and speech.

4.1. Communication Tools

Enabling the exchange of information produced in virtual worlds and generated by humans and robots in the real world is not a trivial task. In order for it to be successful, users must use different communication or networking tools. Below, we briefly describe the communication tools most frequently used in the reviewed articles.

4.1.1. Robotic Middleware and Bridging Mechanism

Communicating with game engines and external robotic modules often requires inter-process communication using socket connections. However, managing socket connections can become a cumbersome task in complex software architectures. Therefore, using robotic middleware is often recommended in order to reduce the management complexity and improve the reusability of complex robotic software architectures. We observed that the Robot Operating System (ROS) version 1.0 was the most popular middleware solution used by the authors of the reviewed articles. However, the connection between ROS1 modules and virtual environments developed in Unity and Unreal cannot be made directly due to differences in dependencies, programming languages, communication protocols, and/or operating systems. In order to deal with this issue, bridging mechanisms or modules are often used. These modules generally translate and transfer data from a game engine to a non-compatible robotic middleware that uses another message formatting or communication protocol. In the case of ROS, the most popular solution enabling the connection of non-ROS programs to an ROS network is rosbridge_suite [36], which provides a solution based on JavaScript object notation (JSON) and Websockets. However, ref. [37] reported performance limitations of rosbridge_suite when transferring large sensory messages (e.g., images and point cloud data), which specifically affects transfer speed. In this direction, ROS# [38] has been widely used by the reviewed articles for communicating ROS with Unity. This framework provides an ROS Bridge Client for C# applications, a Unified Robot Description Format (URDF) file parser that enables developers to import robot models to Unity, and a message generation tool for sending and receiving ROS messages from Unity. To overcome the performance limitations of rosbridge_suite, alternatives based on TCP/IP sockets instead of Websockets, such as [37], have been proposed. More recently, the TCP connector [39] framework for Unity has been released by the developers of ROS# (sharing many of the same tools) as an official Unity asset. However, none of the reviewed articles has reported the integration of this framework. Another alternative explored by [40,41] proposes alternatives for connecting Unity with external robotic systems using a middleware solution based on ZeroMQ [42]. Unlike conventional sockets, which present a synchronous interface, ZeroMQ sockets are asynchronous. Moreover, as proven in [40,41], tools based on ZeroMQ can be used to overcome the communication performance limitations of rosbridge_suite. Most recently, ros2-for-unit (https://github.com/RobotecAI/ros2-for-unity [access date: 12 December 2022]) has been proposed as an unbridged alternative to connect ROS2 and Unity. The authors of this tool state that it provides considerably lower latency than solutions using rosbridge_suite.

4.1.2. Multiplayer Networking

The use of game engines for developing XR experiences in HRI applications can benefit from the technology used to create online multiplayer games. In this type of game, users are not restricted to being located in the same local network, and can interact with other players around the world. In this context, game engines can enable the relatively easy integration of high-level multiplayer networking tools. These tools can handle communication and synchronization of objects presented in virtual environments and user authentication. Examples of these tools include Photon [43], Mirror [44], and the recently launched Netcode for GameObjects [45]. However, very few works, among which are included [37,46,47] have used this technology to enable remote and/or multi-party HRI in virtual environments.

4.2. Interaction Tools

Several works identified in this article integrate one or more software libraries and frameworks to detect human actions or states of the environment. In this context, ref. [48] integrated TouchScript [49], a multi-touch library for Unity, is used to enable multi-touch gesture recognition in a tangible user interface (TUI). Moreover, ref. [48] integrated Point Cloud Library (PCL) [50] to detect objects from a table using the Random Sample Consensus method [51]. Zhou et al. [52] used the PointNet [53] neural network library to classify objects from point cloud data. Examples of applications enabling PointNet include classification, part segmentation, and scene semantic parsing. In the case of AR projects, the Vuforia Engine SDK [54] has been used in a number of the reviewed articles. This software tool enables developers to add advanced computer vision functionalities to build AR applications for mobile devices and AR glasses, such as recognizing and tracking objects by shape using digital 3D models and attaching AR content to flat images or objects with cylindrical bodies, among others. An alternative solution for creating AR applications on mobile devices used in [55] is Google’s ARCore library for Unity [56]. In the case of AR glasses, specifically Hololens, the Mixed Reality Toolkit [57] is generally used for AR/MR development in Unity. Additionally, ref. [58] used the ZXing [59] barcode image processing library to link the world coordinates of a virtual environment with the physical world. For skeleton tracking, ref. [60] implemented the NuiTrack 3D tracking framework [61], which provides cross-platform solutions for body motion analysis. This framework is compatible with several depth sensors, such as Kinect Azure, Intel Realsense, and Asus Xtion. One of the most popular libraries used in the reviewed articles is the OpenCV [62] image-processing library, which has become quasi-standard in many computer vision applications. OpenCV is often used as a dependency with other software tools using computer vision algorithms, such as the Find_Object_2D [63] ROS package, which is used in [60] to perform feature extraction from images of objects. For motion planning and control of robots, many reviewed articles use the MoveIt Motion Planning Framework [64]. Other tools used in the reviewed articles include IAI Kinect [65] (for bridge Kinect and ROS), RobCog-IAI [66] (to develop robot applications using Unreal Engine), and Newton VR (to manipulate virtual objects using tracking controllers) [67]. Basic specifications, descriptions, and capabilities of the tools mentioned in this section are shown in

Table 4. Specifications and capabilities of identified interaction tools in the reviewed articles.

Tool	Programming Languages	General Description and Main Capabilities/Functionalities
TouchScript	C#	An open-source framework that enables the use of basic gestures (e.g., press, release, tap, long press, flick, pinch/scale/rotate) in devices with touch input
Point Cloud Library (PCL)	C++	An open-source library for 2D/3D image and point cloud processing. Some of its main modules include visualization, segmentation, and registration of point clouds.
PointNet	Python	A deep net architecture and library written on top of TensorFlow. It can be used to reason about 3D geometric data (e.g., point clouds and meshes). Its functionalities include part segmentation, object classification, and scene semantic parsing.
Vuforia Engine SDK	C++, Objective-C++, Java, and C#	A framework for building Augmented Reality applications in Android, iOS, and Windows applications able to be executed on mobile devices and AR glasses. It provides free, academic, and premium plans.
Google’s ARCore	Kotlin/Java, Unity/C#, Unreal/C++	A framework for building Augmented Reality applications in Android, iOS, Unreal, and Unity. It uses motion tracking, environmental understanding, and light estimation to integrate virtual models with the real world
Mixed Reality Toolkit	C#	Is a cross-platform framework providing a set of components and features for accelerating Mixed and Augmented reality application development. Its functionalities include eye and hand tracking, spatial awareness, speech dictation, and XR device and game control manager.
ZXing	Java	An open-source library for 1D/2D barcode image processing. Its functionalities include analysis and extraction of information from images containing barcodes or QR codes
NuiTrack	C++, C#, and Python	A 3D body tracking middleware. This framework can interpret depth maps as 3D point clouds, detect floor planes and background objects, detect and track persons, perform skeleton and hand tracking, and perform face analysis.
OpenCV	C++/C, Python, and Java	A cross-platform and general-purpose real-time computer vision library. Its functionalities include image acquisition from RGB cameras, image segmentation and filtering, object recognition, camera calibration, and changing image color spaces, among many others.
Find_Object_2D	C++	A ROS-based package using a Qt interface that implements different feature detectors and descriptors for calculating the 3D position of objects using OpenCV
MoveIt	C++ and Python	Robot motion planning framework enabling motion planning, manipulation, collision checking, 3D perception, kinematics, control, and navigation
IAI Kinect	C++	Collection of tools for enabling the use of Kinect with ROS using libfreenect2 (a driver for Kinect)
RobCog-IAI	C++	Provides Unreal Engine plugins for building applications for robotics
Newton VR	C#	A VR interaction system enabling to pick up, throw, and use objects using tracked controllers

.

5. Applications in Human–Robot Interaction

This section briefly describes the main contributions of the reviewed articles. We have classified these articles into four groups representing the main context or application area; therefore, this section answers RQ1. These articles are summarized in

Table 5. Articles presenting XR solutions for social robotics.

Article			Hardware Integrated		Software Tools Integrated
Year	Reference	XR Type	Robots	Devices	Communication	Interaction
2019	[68]	VR	Pepper	HTC VIVE	TCP/IP	NAO SDK
2021	[60]	VR	QTRobot	Oculus Rift, iphone	ROS and ROS bridge	NuiTrack, Find_Object_2D
2022	[69]	VR	NAO	Oculus Rift	ROS, ROS#	NAO SDK

,

Table 6. Articles presenting XR solutions for programming robots using game engines.

Article			Hardware Integrated		Software Tools Integrated
Year	Reference	XR Type	Robots	Devices	Communication	Interaction
2019	[48]	AR	UR5	Kinect 2, projector	ROS, ROS#	TouchScript, Point Cloud Library
2019	[73]	AR	GripperBot, CamBot, Armbot	Oculus Rift and IR-LED Sensors, ZED stereocamera	ROS, ROS#	N.A
2020	[74]	MR	KUKA iiwa, UR10e	HoloLens	ROS (no details on how Unity and ROS are connected)	MoveIt, Mixed Reality Toolkit
2020	[55]	MR/AR	OpenManipulator-X	Smartphone	Bluetooth	Google’s ARCore
2021	[75]	AR	UR5 and ABB IRB 2600	HoloLens 2	ROS, ROS#, TCP/IP sockets	N.A.
2021	[58]	AR	UR10	HoloLens	ROS, ROS#	ZXing, Mixed Reality Toolkit, MoveIt

,

Table 7. Articles presenting XR solutions for the teleoperation of industrial robots using game engines.

Article			Hardware Integrated		Software Tools Integrated
Year	Reference	XR Type	Robots	Devices	Communication	Interaction
2020	[79]	AR	Kuka KR6 r900	Realsense D435, Gamepad, HoloLens	ROS and ROS bridge	Mixed Reality Toolkit
2020	[52]	VR	Baxter	Kinect, HTC VIVE	ROS, ROS2Unity, ROS#	PointNet, IAI Kinect, libfreenect2, PCL
2021	[47]	VR	COMAU Dual Arm	Oculus Rift	ROS, ROS#, Mirror	Moveit, Open Motion Planning Library
2021	[80]	VR	UR5	Pro, ZED-mini, Intel Realsense, HTC VIVE	ROS and ROS bridge, TCP sockets	RobCog-IAI
2022	[81]	MR	UR5	Camera, Kinect, HTC VIVE	ROS and ROS bridge	OpenCV

and

Table 8. Articles presenting XR solutions supporting Human–Robot collaboration tasks.

Article			Hardware Integrated		Software Tools Integrated
Year	Reference	XR Type	Robots	Devices	Communication	Interaction
2018	[83]	AR	Turtlebot	Tablet	ROS (no details on how Unity and ROS are connected)	Vuforia SDK
2020	[46]	AR	COMAU AURA, COMAU Racer5 HoloLens	Smartwatch, Projector, Camera	ROS Bridge and Mirror	Microsoft Speech recognition library
2020	[84]	AR	KUKA KR6 R7000 Sixx	HoloLens	ROS and ROS bridge #,	Vuforia SDK
2021	[85]	VR	Franka Emika	N.A	ROS, ROS#	N.A.
2021	[86]	AR	Turtlebot2	Tablet	ROS, ROS#	N.A.
2021	[37]	VR/AR	Turtlebot2, Turtlebot3, TIAGo, among others	Oculus Rift	ROS, ROS Bridge, TCP/IP, Photon	NewtonVR
2022	[87]	VR	Scitos G5 mobile robot	HoloLens 2, Azure Kinect	ROS, ROS#	Mixed Reality Toolkit
2022	[88]	MR/AR	ABB industrial robot	HoloLens	FB network	N.A
2022	[89]	VR	Universal Robot	Xsens motion, HTC VIVE	ROS, ROS#	N.A
2022	[90]	AR	Mobile dual-arm platform	HoloLens 2	ROS, ROS#	Mixed Reality Toolkit

.

5.1. Social Robotics

The results of our review indicate that the use of game engines to build HRI applications using XR technologies for service and social robotics remains in its initial stages. In this context, a common application is to use a VR environment to replace a real robot, as presented in [70]. However, few works integrate real social robots with virtual environments. An exception is [68], which uses a VR environment to investigate the differences between a real Pepper robot and its virtual representation by using both a virtual and a real robot simultaneously. For this, Client/Server communication using traditional TPC/IP sockets in a local area network is used to connect Unity with Choregraphe [71] (the official visual programming environment [72] and simulation platform for Pepper and NAO social robots). The results of [68] suggest that users may prefer closer interaction with a physical robot than its virtual representation. The authors of [68] infer that this difference can be influenced by the potential discomfort that can be produced by a VR environment with technical limitations, such as a limited field of view, low visual fidelity, or a low-resolution display. Another exception is using VR environments to remotely control social robots. In this context, refs. [60,69] have proposed ROS-based systems for controlling NAO and QTRobot.

5.2. End User Programming of Industrial Robots

Robot programming is an emergent area where game engines and XR technologies are implemented for creating HRI systems. In these applications, the reduction of training effort available when using intuitive, simple, and usable interfaces is fundamental. In this context, ref. [73] have presented GhostAR. This time-space editor combines the Oculus Rift and the ZED Dual Cameras to enable users to program and edit the actions of robots through Programming by Demonstration (PbD). This system can display “ghosts”, images representing a timeline of captured human motions. These “ghosts” are used as time-space references, which can be used to edit HRC tasks using a virtual avatar. PbD solutions using the HoloLens have been explored using AR [58] and MR [74] methods, providing promising results compared to conventional programming through teaching pendant and kinesthetic programming. In this context, ref. [75] have proposed a markerless solution that records and translates the operator’s hand movements into robot motions and commands using Hololens 2. However, HoloLens is considered an expensive device [76,77,78]. In contrast, smartphone-based AR applications such as [55] can represent a more cost-effective solution. The authors of [75] claim that their proposed approach “is computationally lightweight compared to other Learning from Demonstration techniques, such as Machine Learning approaches”. However, they claim that this solution may not be suitable for applications that require high accuracy in robot motions. A projection-augmented tabletop interface, denoted as PATI, was proposed in [48] to overcome the perception limits (visual tracking and object referencing) of PbD. In PATI, the task-level specifications defined in the TUI are translated to robot commands and motion plans. For this, the PATI system can detect touch gestures and track objects in the working environment. The authors of PATI claim that their system can be learned in a shorter period of time than state-of-the-art PbD systemsm, and can enable users to be more efficient, producing less physical and mental workload.

5.3. Teleoperation

VR and AR systems are suitable solutions for enabling remote control of robots in scenarios that are inaccessible or dangerous for humans. Relevant considerations in the design of effective teleoperation systems highlighted by the reviewed articles include intuitiveness, low latency, and situational awareness [47,52]. In this context, ref. [47] propose an intuitive VR interface for enabling multiple users simultaneously immersed in a synchronized VR scene to define a robot’s positions and actions. A solution to increase the human operators’ situational awareness in teleoperation tasks is proposed in [52]. For this, the authors of [52] present the Telerobotic Operation based on an Auto-reconstructed Remote Scene (TOARS) algorithm, which uses deep learning methods to detect objects and their physical properties from point cloud information generated by a Kinect v2. This information is subsequently rendered as virtual objects with physical properties (e.g., friction and weight) in the Unity-based simulation to provide a more immersive interaction. In addition, the authors provide an alternative for reducing communication latency produced by rosbridge_suite. For this, they reduce the amount of transferred data over the network generated by Kinect v2. Moreover, a TCP/IP connection between Unity and ROS is implemented to improve the communication performance in comparison with ROS#. An AR solution using gamepads for the teleoperation of industrial robots is proposed and evaluated in [79]. Results from usability testing performed in [79] suggest that gamepads can provide higher usability for teleoperation tasks than conventional teaching pendant devices, regardless of the user’s previous experience in robotics and/or AR technology. Unlike most of the reviewed articles, which use Unity as the main game engine platform, ref. [80] proposes a VR solution for robot teleoperation developed with Unreal Engine version 4 for high-fidelity visualization, denoted as Vicarios. The Vicarios framework aims to improve situational awareness and user performance in remote robot manipulation tasks by implementing a method denoted as viewpoint-independent motion mapping. In this approach, operators can choose between different viewpoints without worrying about remapping their motions to each viewpoint. Vicarios uses the robcog-iai [66] tools to import 3D models of robots using the URDF, then connects the Unreal Engine to ROS. Two MR-based hybrid 3D/2D visualization paradigms, denoted as MR-3DSV (integrating 3D stereoscopic vision and monocular RGB cameras) and MR-3DPC (integrating 3D point cloud and monocular RGB cameras), are proposed in [81]. These paradigms aim to provide more immersive and intuitive teleoperation for non-skilled operators. The results from [81] indicate that using hybrid 3D/2D vision paradigms can reduce the training effort, completion time, and cognitive workload compared with conventional 2D-based teleoperation methods (i.e., only using a 2D camera stream to perceive the robot space). Finally, ROS Reality [82] is a VR Framework and ROS package enabling the teleoperation of ROS-enabled robots using Unity-compatible VR headsets. This framework provides different modules, including WebSocket communication with ROS, a URDF parser that allows 3D models of ROS-based robots to be imported to Unity, an RGB camera, and a point cloud and inverse kinematic status visualizer. Experimental validation of the ROS Reality framework was carried out in a pilot study that compared the efficacy against the kinesthetic handling of the robot. The results suggested that the proposed VR architecture was less effective than direct manipulation of the robots. However, the results additionally indicated that teleoperation solutions using VR environments can be more useful than directly manipulating a real robot when complex movements of the robot’s joints are required. The source code of ROS Reality is available on GitHub (https://github.com/h2r/ros_reality, accessed on 10 October 2022).

5.4. Human–Robot Collaboration

The creation of XR systems enabling robots to perform collaborative tasks with humans has been widely explored. In this context, ref. [85] presented a set of techniques for ad hoc HRC (i.e., collaborative activities where the robot works without pre-planned actions) using digital tabletops and mixed reality. However, restrictions presented by the COVID-19 pandemic constrained the experimental validation of the system in a VR simulation. A solution that links AR and eye-tracking technologies to enable 3D object detection in a collaborative way was presented in [87]. A key novelty that the authors of [87] presented is the use of gaze information to help detect and segment unknown objects. For this, they used the HoloLens 2 AR device, which has a built-in eye tracker over Unity, and for which information is connected with ROS using ROS#. The authors claimed to be the first to use AR in an HRC setting to segment unknown 3D objects without deep learning techniques, and their system was able deal with cases where existing deep neural learning techniques have failed. Other related works using Hololens and ROS (connected by ROS#) to facilitate HRC activities have been described in [84,91]. A novel approach using Hololens for empowering human-centric assembly (HCA) by combining a digital twin [92] developed in a Java-based 3D environment with AR methods (executed in Unity) and event-driven function blocks [93] has recently been proposed by [88]. Validation of their system was centered on performance evaluation in an engine assembly case. A framework denoted as Operator Support Module (OSM) was presented in [46] for facilitating collaboration by providing cognitive support to operators using AR technologies, voice commands, and gestures. The authors stated that OSM can recognize objects and provide feedback to users, e.g., when a wrong object is manipulated or when a step in the assembly task is completed. Due to the expensive deep learning methods used in [46], the system was distributed on different computers. A module in Unity denoted OSM Controller was proposed to handle communication between the different elements in the system. This module connects information generated by Unity with ROS (using rosbridge) and Unity with wearable and smart devices (using Mirror). We observe that studying ergonomic and performance aspects in HRC industrial tasks using XR systems is becoming an emergent topic. In this context, ref. [89] proposed a methodology to determine the accuracy and quality of a human motion capture system integrating an inertial measurement unit (IMU) and Vive-based HTC during physical human–robot interaction. For this, they created a digital twin of the actual system in which human and robot motions are replicated in real-time in the visual environment. To transfer the actual robot motion to the digital robot model, they used ROS#. An effort to understand the variables affecting the trust involved in HRC using AR technologies was presented in [83]. The presented results suggest that context awareness and human safety are key variables affecting trust in HRC scenarios. Though presented in [83] was performed with a Turtlebot mobile robot, the authors stated that similar results could be obtained with industrial cobots, which are heavy and strong. However, their authors note that further research involving a real industrial cobot is needed in order to better understand the factors affecting trust in cobots. Most recently, ref. [90] proposed a software suite for supporting the operator’s interaction with mobile and industrial robot co-workers. This suite proposes a solution for easy programming, supporting autonomous operations, assisting operators in assembly processes, and enhancing safety awareness and resilience of production processes. The experimental evaluation proposed in [90] suggests that the proposed system can reduce the time required by operators to recover from operational failures and safety violations. AR for Robots Collaborating with a Human (ARROCH) [86] is a system enabling communication between humans and multiple robots. With ARROCH, an operator can use a tablet to visualize the robots’ current states and planned actions (intentions). Finally, the SIGverse [37] cloud-based VR platform, built on top of Unity, proposes solutions for performing different HRI experiments. With SIGverse, one or more users can simultaneously interact with virtual robots at the same time using the Photon Unity Networking (PUN) asset for Unity, which provides multiplayer networking features. A bridging mechanism is used to ensure communication between VR scenes created in Unity and ROS modules. This bridging mechanism uses WebSockets and the JSON format to send small-size data from Unity to ROS, a solution based on TCP/IP sockets, and uses binary JavaScript object notation (BSON) format to send large data messages. The user cases reported in [37] include robotic competitions, collaborative clean-up of a room using human pointing gestures, evaluation of the subjective quality of HRI, and motion learning by demonstration. SIGVerse is available on GitHub (https://github.com/SIGVerse, accessed on 10 October 2022), and its documentation can be found on its website (http://www.sigverse.org/wiki/en/, accessed on 10 October 2022).

6. Challenges and Future Opportunities

This section summarizes a set of hardware and software issues reported by the authors of the reviewed articles. Therefore, this section answers research question RQ3.

Ergonomics and comfort issues of popular XR headsets have been repeatedly exposed as one of the most relevant limitations of the systems proposed in the reviewed articles. In this context, ref. [58] noted that HoloLens, the most popular AR device used in the reviewed articles, is considered uncomfortable after long usage periods due to its size and weight. Similar results have been presented in experiments conducted in [79]. Experiments presented in [91] indicate that the ergonomics of the HoloLens headset represent a relevant drawback, making AR solutions ineligible for full-scale usage in working environments despite the advanced features and functionalities they can provide. Aivaliotis et al. note that almost every VR and AR headset on the market is unsuitable for a regular eight-hour workday [90]. Moreover, a notable percentage of the population suffers cybersickness after long sessions using VR/AR headsets [94]. We observed that few reviewed articles using these XR devices reported any countermeasures provided to reduce cybersickness or build tolerance. Exceptions are [37,95], where an analysis of the elements or factors causing cybersickness (e.g., when motions in a virtual world are too fast compared with reality) is performed, with the offending element modified or deleted to improve performance and user experience. Furthermore, many VR/AR systems require the use of expensive hardware and computational components and complex software tools. For example, AR devices such as Hololens 2 cost at least USD 3500, representing an expense that many researchers, sectors, or users cannot afford [76,77,78]. Moreover, many VR/AR solutions require the acquisition of mid-range or high-end computers to run XR environments and perceptual and robot control algorithms. In this respect, the availability of more accessible hardware devices such as Oculus and more efficient algorithms such as the mediapipe framework can reduce the need to acquire high-performance graphics cards. On the other hand, ROS has become a popular solution in robotics research and industrial applications. However, “it requires significant training and experience before one can use it to competently program robot tasks” [48]. We observe that many of the projects presented in this review require the use of both Linux (for robot control) and Windows (for executing the virtual environment) machines at the same time. These machines must communicate using the tools described in Section 4.1. In cases where only a Windows computer is available, it is necessary to port the robotic code executed in Linux. This code can be embedded in Unity as C# scripts, as suggested by [37], or executed as external modules (e.g., written in Python or C++). However, this can be a complex and time-consuming task. Therefore, researchers and practitioners must discuss the suitability of these approaches [37] based on the needs, experience, and economic/computational resources of the end users. For robots providing cross-platform software development kits (SDK), such as NAO and Pepper robots, relying on user-friendly and platform-independent communication tools based on lightweight and portable libraries can be a suitable and user-friendly alternative to ROS in certain cases. An example in this direction is presented in [96], which proposes a cross-platform system and low-cost VR alternative for industrial robots using the Message Queuing Telemetry Transport (MQTT) protocol.

According to [97], sustainability aspects (e.g., computational performance and power consumption), which is a Society 5.0 and Industry 5.0 [24] main concern, are rarely contemplated by the robotics research community. Additionally, cost and resource consumption can be particularly relevant when developing robotic systems for use outside laboratories and industrial applications [97]. As suggested in [37,48,58], lowering the economic and technical barriers of XR systems is relevant for promoting their adoption in custom contexts and exploring the potential benefits that robots and XR technologies can provide together. Moreover, the lack of reliability and ease-of-use aspects in many cases is used by industry to justify the hesitation and lack of acceptance of XR solutions for use on larger scales [58].

In this review, we observed that the Unity game engine was the most commonly used platform in the reviewed articles. The only exception was [80], which used Unreal Engine 4. One factor that can motivate robotics researchers to use Unity over Unreal engine is that Unity presents a lower learning curve and requires fewer programming skills to build virtual worlds. The very recent release of Unreal Engine 5 promises photorealistic visuals, which can be used by developers and researchers to create immersive experiences. However, usable tools and frameworks that further reduce the effort required to produce XR applications that integrate physical robots remain required in order to expand the popularity of the Unreal engine in the HRI community.

7. Conclusions

This review article has identified commonly used and relevant software frameworks and tools used by the robotics community to integrate physical robotic platforms with virtual elements generated by game engines, specifically, the Unity and Unreal Engines. We have presented and classified these software tools as networking (including robotic middleware and multiplayer networking) and interaction tools. Additionally, we have presented the main contributions of the reviewed articles and classified their application contexts into four main groups: social robotics, teleoperation, end-user programming, and HRC. We observe that using these game engines along with robotic middleware and multiplayer networking frameworks is becoming a trend in the creation of XR applications in HRI for industrial settings. However, its use for social robotics remains in the initial stages.

The world is passing through a paradigm change towards Society 5.0 and Industry 5.0, and XR technologies are often considered keystone elements of these paradigms. However, the articles reviewed in this work suggest that the acceptance and success of these technologies in HRI depends on both the efficiency that the proposed systems are able to achieve and on the human, economic, and sustainability factors of their software and hardware elements.