A Comparative Study of Safety Zone Visualisations for Virtual and Physical Robot Arms Using Augmented Reality

Abstract

The use of robot arms in various industrial settings has changed the way tasks are completed. However, safety concerns for both humans and robots in these collaborative environments remain a critical challenge. Traditional approaches to visualising safety zones, including physical barriers and warning signs, may not always be effective in dynamic environments or where multiple robots and humans are working simultaneously. Mixed reality technologies offer dynamic and intuitive visualisations of safety zones in real time, with the potential to overcome these limitations. In this study, we compare the effectiveness of safety zone visualisations in virtual and real robot arm environments using the Microsoft HoloLens 2. We tested our system with a collaborative pick-and-place application that mimics a real manufacturing scenario in an industrial robot cell. We investigated the impact of safety zone shape, size, and appearance in this application. Visualisations that used virtual cage bars were found to be the most preferred safety zone configuration for a real robot arm. However, the results for this aspect were mixed for a virtual robot arm experiment. These results raise the question of whether or not safety visualisations can initially be tested in a virtual scenario and the results transferred to a real robot arm scenario, which has implications for the testing of trust and safety in human–robot collaboration environments.

1. Introduction

Robotics technology has changed industrial operations, enabling the efficient and accurate completion of tasks. One of the most significant applications of robotics is the use of robot arms for repetitive, complex or hazardous tasks in various industrial settings, including manufacturing, construction, and healthcare. However, ensuring the safety of both humans and robots in these environments is a critical challenge that needs to be addressed. To ensure the safe operation of robot arms in industrial settings, it is essential to provide clear visual representations of safety zones—areas around the robot arm that are free from obstacles and where human workers should not enter to avoid collisions or accidents. Traditional approaches to visualising safety zones include physical barriers, warning signs, and colour-coded floor markings [1]. However, these methods may not always be effective in dynamic environments, where the layout of the workspace may change frequently, or in settings where multiple robots and humans are working simultaneously [2].

Virtual and augmented reality technologies have the potential to overcome these limitations by providing dynamic and intuitive visualisations of safety zones in real time. Virtual reality (VR) technologies can create fully immersive environments, where users can simulate the operation of robot arms without the need for physical equipment [3]. Augmented reality (AR) technologies, on the other hand, can overlay holographic-like images onto the real world, providing a more realistic and contextually relevant representation of safety zones [2,4,5,6,7]. Recent advances in mixed reality devices, such as the Microsoft HoloLens 2, have enabled users to interact with such holographic images overlaid onto the real world. The HoloLens 2 has been used in various applications, including remote assistance [8], training [9], and manufacturing [10] in industrial settings. However, little research has been conducted to evaluate the effectiveness of safety zone visualisations in virtual and real robot arm environments using the HoloLens 2.

In this study, we aim to compare the effectiveness of safety zone visualisations in virtual and real robot arm environments using the Microsoft HoloLens 2. Our focus is on understanding how different safety zone visualisations impact on users’ perceptions of space in relation to a robot arm in both virtual and real environments. By examining the differences between the two environments, we can gain a better understanding of how to design and develop future human–robot interaction systems that incorporate effective safety zone visualisations, and thus, potentially, how this impacts on a user’s feeling of safety.

Previous research work has considered different approaches to safety zone visualisation, including 2D areas [2], safety curtains [4], user-configurable barriers [5], and geometric objects [6,7]. In this paper, we extend the idea of using geometric objects by considering virtual cage bars and investigating different configurations of shape, size and appearance. In addition, unlike other work, we compare the use of safety zone visualisation in relation to both a virtual robot arm and a real robot arm. We use a collaborative pick-and-place application for this work.

The rest of the paper is organised as follows. Section 2 presents related work. Section 3 describes the system, which includes two servers and a HoloLens client, the pick-and-place application, and the virtual and real robot experiments we conducted with different safety zone configurations. Section 4 presents the results of the experiments. Finally, Section 5 presents conclusions.

2. Related Work

A range of AR display technologies have been used in studying the safety and efficiency of robot arm operations. For example, Hietanen et al. [4] used both projection-based AR and Microsoft HoloLens 1 in the context of a human–robot collaboration task involving a diesel engine assembly, utilising depth sensor information for work space monitoring in both cases. The results showed that AR interaction led to a reduction in task completion time by up to 21–24% and reduced robot idle time by up to 57–64%. Moreover, the user survey indicated that the projector–mirror setup was deemed more comfortable and easier to use than the HoloLens 1, which users found to be bulky and uncomfortable to wear for extended periods. Additionally, the limited field of view of the HoloLens 1 was identified as an issue. Lotsaris et al. [2] also used a HoloLens 1 in a system that enabled the display of robot sensor information and safety zones to a user. Feedback from the study revealed that, similar to previous research, the HoloLens 1 was considered uncomfortable to wear for prolonged periods, thereby rendering it unfit for extensive use in a work environment.

Hoang et al. [5] presented a virtual barrier system equipped with an AR interface to ensure safe human–robot collaboration (HRC). The system offers two distinct virtual barriers, namely a person barrier that surrounds and tracks the user and a user-defined virtual barrier for objects or areas. The study evaluated the performance of a pick-and-place task that simulated an industrial manufacturing procedure using the Microsoft HoloLens 2 compared to a standard 2D display interface. They concluded that the proposed configurable virtual barrier system, coupled with AR, enhances HRC safety compared to a conventional 2D display interface. Cogurcu and Maddock [7] considered user-configurable virtual geometric objects, such as cubes and cylinders, that enveloped a real Universal Robots 10 (UR10) robot arm. These objects were either a single shape that adapted dynamically to the robot arm’s movements or individual shapes that encapsulated the robot arm’s components. The virtual safety zone sizes or the objects are calculated based on a combination of ISO standards [11], hardware, and network latencies, which were not taken into account in prior studies.

In the context of VR technologies, several studies have evaluated the use of head-mounted displays in robot arm environments. For example, the work of [3] aimed to evaluate the performance and mental workload of users in virtual versus physical collaborative robot (cobot) operations under different levels of mental load. The results suggested that virtual simulations of HRC have potential advantages for mental well-being and industrial production, particularly for complex or demanding tasks. The study suggests that pupil size variation could be a reliable and flexible index of implicit workload in HRC, which could have implications for robotic automation.

Other work has considered the use of virtual and real environments for testing and optimising HRC. For example, [12] explored the use of virtual environments for testing safety systems in HRC, and [13] investigated whether VR technology used in manufacturing could provide faster design options. While these studies have provided valuable insights into the potential benefits of virtual environments, the overall picture is less clear. Some research suggests that virtual environments are not advanced enough to accurately simulate real-world conditions, and real-world testing is necessary for a comprehensive assessment of systems involving robots [9], while other research argues that virtual environments can provide a suitable approximation of real-world conditions, with some limitations [14]. For example, according to [15], the use of VR technology has been shown to provide users with a similar experience to physical interaction with a robot arm. In their study, the authors found that users were able to perform tasks with a virtual robot arm just as accurately and efficiently as they could with a physical robot arm. As we will show in our paper, there are further complications that must be considered when transferring from virtual to real scenarios.

3. Methods

We use a collaborative pick-and-place application for our experiments, as described in Section 3.2 and Section 3.3. However, before that, Section 3.1 will briefly present the system that we created to support this.

3.1. The System

The system is comprised of three main components as depicted in Figure 1: a digital twin server, a robot operating system (ROS-Melodic) server, and a HoloLens client. The digital twin server and the ROS-Melodic server are physically connected to the same local area network (LAN), while the HoloLens client is connected via Wi-Fi. This configuration was selected to minimise the impact of latency while still allowing for the mobility of the human operator.

The digital twin server is a component of the Confident Safety Integration for Collaborative Robotics (CSI:Cobot) project and is an instance of the digital twin framework (DTF) [16]. The DTF, created in Unity3D, is a modular framework that enables the development and testing of safety-critical digital twins. The digital twin server contains a virtual workspace that consists of 3D representations, connections to robots, APIs, and network services. The HoloLens 2 is linked to the server, allowing it to participate in the current workspace session and communicate with both the virtual environment and the robot. The digital twin server handles all the computations in the system, making the client lightweight and simple, consisting of virtual models and a network component. The client can be updated to perform calculations through Matlab or ROS, and any improvements made to the digital twin server will be instantly visible through the HoloLens.

The ROS server connects the DTF to the UR-10 robot arm. Communication between the ROS server and DTF is persistent and bilateral, allowing for the exchange of information and feedback across multiple user sessions. The server runs on Ubuntu 18.04 with ROS-melodic and is responsible for controlling the Robotiq 2F-85 gripper end effector. The ROS server transfers messages between the real robot arm and the digital twin server, which is then displayed on the HoloLens 2. The HoloLens 2 acts as a client to the digital twin server and provides selective, functional, and intuitive access to the data-rich workspace. A Unity client application, built in Unity3D with the Mixed Reality ToolKit (MRTK) version 2.7.3, runs on the HoloLens and communicates with the digital twin server via a UDP network interface. A bilateral TCP/UDP protocol was developed to enable communication between the digital twin server and HoloLens, allowing for easy integration with new devices and ensuring platform independence.

3.2. Experimental Setup

The collaborative pick-and-place application used in our experiments employs a Universal Robots 10 (UR10) robot arm and a Microsoft HoloLens 2 for control and visualisation (Figure 2). This application mimics a real task that would typically be carried out in an industrial robot cell. In our application, the robot arm moves a block from its initial position on the first table (Area A) to the intermediate table (Area B), and then the user manually moves the block from the intermediate table to its final position on the second table (Area C), as illustrated in Figure 3 for a real robot arm. This setup is used for both experiments, with the real robot arm and the real wooden blocks replaced by a virtual robot arm and virtual wooden blocks for the second experiment, as illustrated in Figure 4.

Using the HoloLens 2, a virtual safety zone is visualised around the robot arm. Four safety zone configurations were used in the experiments (as shown in Figure 5 and Figure 6):

• A single, dynamic virtual safety zone (transparent red cuboid) enveloping the complete robot arm, with the size dependent on both the robot arm’s position and the ISO 15066 standard (Figure 5a and Figure 6a);
• A single, larger dynamic virtual safety zone (transparent red cuboid with virtual cage bars) surrounding the entire robot arm, with a supplementary distance layer for hardware and network latencies (Figure 5b and Figure 6b);
• A sectional virtual safety zone, consisting of a grouping of transparent, red cylinders around the principal components of the robot arm, with the dimensions dictated by the ISO 15066 standard (Figure 5c and Figure 6c);
• A larger sectional virtual safety zone, comprising a collection of transparent, red cylinders (with virtual cage bars) around the main segments of the robot arm, along with an additional distance layer for hardware and network latencies (Figure 5d and Figure 6d).

The dynamic cuboid was chosen to provide a closer fit to the robot arm than using a fixed size cuboid cage that would have to be big enough to accommodate any movement. The latter is the case for a physical fixed cage in an industrial setting. The dynamic cuboid allows the user to get closer to the robot arm. The sectional cylinders were chosen to best mimic the shape of the main pieces of the robot arm. The two smaller configurations used only transparent shapes, whilst the two larger configurations added virtual cage bars to mimic real cage bars.

Figure 5. Four configurations used in real robot arm experiment.

Figure 5. Four configurations used in real robot arm experiment.

Figure 6. Four configurations used in virtual robot arm experiment.

Figure 6. Four configurations used in virtual robot arm experiment.

To calculate the size of the safety zones displayed around the robot arm, we used ISO standards and also considered latencies in our system. The ISO/TS 15066:2016 [11] standard sets the safety requirements for the use of collaborative industrial robots in the work environment. This standard complements the requirements and guidance in ISO 10218-1 and ISO 10218-2 [17], which outline the four different types of cooperation scenarios in human–robot collaboration: safety-rated monitored stop, hand-guiding, speed and separation monitoring, and power and force limiting. In the speed and separation monitoring (SSM) scenario, the robot and operator can work in the same workspace with reduced risk, by maintaining a protective separation distance (PSD) between them. Previous studies [7,18] explained how to calculate PSD by taking into account the definition of ISO 13855 [19]. We use a robot controller latency of 8 ms, and the operator’s speed is calculated to be a maximum of 1600 mm/s, giving a minimum PSD of 21.6 cm. When using HoloLens 2, we add an additional 80 mm safety margin due to the device’s latency in displaying 3D models, which can vary between 14 and 50 ms. To minimise network latency in the client–server architecture (Figure 1), all servers use the same router. The delay between HoloLens and the digital twin server is higher than other delays due to the wireless connection. The lowest latency between HoloLens and the digital twin server was 7 ms, while the highest value was 50 ms. To ensure maximum safety in human–robot collaboration, the worst-case value of 50 ms was selected, resulting in a total network latency of 90 ms, or an additional 14.4 cm of safety margin. These extra safety margins can be set to be on or off using the system interface.

We also had to consider how to align the safety zone with the moving real robot arm. This involves two aspects: initial alignment and continuous alignment as the robot arm moves. Aligning the virtual robot arm with the real robot arm is a challenge when it comes to placing digital twins in the real world. We solved this problem by using a combination of a QR code and manual alignment. After this alignment process, the ROS commands to the real robot arm can also be used to control the virtual arm, thus keeping the real and virtual robot arms in sync and ensuring that the virtual safety zones remain aligned with the real robot arm (since they are aligned with the invisible virtual counterpart). For the virtual robot arm experiment, there is no real robot arm, so the initial position setting is straightforward. The virtual robot arm is given a starting pose and remains visible as it is moved by the ROS commands.

3.3. Process

In total, 28 participants were recruited, with 14 participants for each experiment. For the real robot arm experiment, none of them had prior experience with AR and VR technology. For the virtual robot arm experiment, none had prior experience with AR but 3 participants had some experience with VR.

For the experiment (with either the real or virtual robot arm), each participant repeated the same pick-and-place task for the three wooden blocks four times, once for each configuration, as described above. The order of configurations was varied for each participant to counter any order effects. After completing all four configurations of the experiment, each participant was asked to fill out a questionnaire using a five-point Likert scale, ranging from “strongly disagree” to “strongly agree”. For each question, users were also given the opportunity to explain their preferences in short paragraphs. The questions for both the real robot arm and virtual robot arm experiments are listed in

Table 1. The questions used for the real and the virtual experiments.

Question No	Questions
Configurations
Q1	This configuration made it easier for me to do the task than the other configurations.
Q2	This configuration made me trust the robot arm when I was doing the task.
Q3	This configuration made me feel safe when I was doing the task.
Q4	I would choose this configuration if I had to do a similar task again.
Q5	Please put the configurations in your order of preference. Your preferred best configuration should be rated as 1 and the others 2, 3, and 4. Tick the relevant boxes accordingly.
Use of Microsoft HoloLens 2
Q6a	I found the HoloLens 2 easy to use.
Q6b	I found the HoloLens 2 to be comfortable to use.
Visualisation
Q7a	The red transparency effect without the virtual bars worked well for displaying the safety zones.
Q7b	The red transparency effect with the virtual bars worked well for displaying the safety zones.
Q7c	Red is a good colour to use for the safety zone display.
Q7d	The cuboid safety zones dynamically changed in size during the experiments. I did not find this distracting.
Q7e	Overall, I thought the safety zones were well displayed.
The System
Q8a	I was confident whilst doing the task using the system.
Q8b	Interaction with the robot arm was easy using the system.
Q8c	I think that I would need the help of a technical person to use the system in the future.
Q8d	I would imagine that most people would learn to use this system quickly.
Q8e	When I made a mistake or the robot made a mistake, it was easy and quick to recover and continue with the task.

. Each AR experience (with the real or virtual robot arm) took about 30 min. After finishing their AR experience, a participant had 30 min to complete the questionnaire.

To ensure safety during the real robot arm experiment, three measures were implemented. Firstly, the system is designed to detect when a user’s hand breaches the virtual safety zone surrounding the real robot arm, and immediately stops the robot arm’s movement. The user can restart the movement after removing their hand from the visualisation zone. The same process was implemented for the virtual robot arm. Secondly, for the real robot arm, a researcher was monitoring the experiment and could press a physical stop button for the robot arm if necessary. This was not necessary for the virtual robot arm. Thirdly, the real robot arm was operated at a low speed to ensure that potential collision forces remained below pain thresholds for the hand and arm, as specified in [20]. The virtual robot arm ran at the same speed as the real robot arm, irrespective of the fact that no collision injury could be caused. All participants were compensated for their contribution, and the university’s ethics procedure approved the study.

4. Results and Discussion

The two separate experiments will be presented before further comparative discussion.

4.1. The Real Robot Arm Experiment

Table 2. Questionnaire results for the real robot arm experiment. Note that ‘-’ means 0 to reduce visual clutter.

	Strongly Disagree	Disagree	Neither Disagree Nor Agree	Agree	Strongly Agree
Configuration
Q1–configuration made it easier to do the task
Small cuboid	-	-	-	-	-
Small cylinders	-	-	-	-	-
Large cuboid with cage bars	-	-	-	3	-
Large cylinders with cage bars	-	-	-	3	8
Q2–configuration made me trust the robot arm
Small cuboid	-	-	-	-	-
Small cylinders	-	-	-	1	1
Large cuboid with cage bars	-	-	-	2	3
Large cylinders with cage bars	-	-	-	1	6
Q3–configuration made me feel safe
Small cuboid	-	-	-	-	-
Small cylinders	-	-	-	-	1
Large cuboid with cage bars	-	-	-	2	3
Large cylinders with cage bars	-	-	-	2	6
Q4–would choose this configuration again
Small cuboid	-	-	-	-	-
Small cylinders	-	-	-	1	-
Large cuboid with cage bars	-	-	-	3	1
Large cylinders with cage bars	-	-	-	-	9
HoloLens 2
Q6a–easy to use	-	2	-	10	2
Q6b–comfortable to use	-	1	1	7	5
Visualisation
Q7a–red transparency without virtual bars worked well	1	7	2	3	1
Q7b–red transparency with virtual bars worked well	-	-	-	1	13
Q7c–red is a good colour to use for the safety zone display	-	-	2	3	9
Q7d–dynamic cuboid is not distracting	1	-	-	4	9
Q7e–safety zones were well displayed	-	-	-	7	7
The System
Q8a–user confident with system	-	-	-	3	11
Q8b–interaction was easy	-	-	-	2	12
Q8c–need the help of a technical person	6	6	2	-	-
Q8d–people would learn the system quickly	-	-	-	6	8
Q8e–if mistake made, it was easy and quick to recover	-	-	-	5	9

summarises the results from the questionnaire used for the 14 participants in the real robot arm experiment. The results of questions Q1–Q4 (considering which configurations made it easier to do the task, trust, safety, and future choice of configuration) suggest that the large cylinders with cage bars were the most popular configuration. The cage bars made it easier to observe the boundaries of the cylinders when they were moving. Based on the comments in the questionnaires, the use of large cuboids caused some users to feel unsafe as they moved towards them.

Table 3. Numbers of users who ranked each configuration (question 5) for the real robot arm experiment, where the rating is 1 = best and 4 = worst. Note that ‘-’ means 0 to reduce visual clutter in the data cells.

	Rankings
Configurations	1	2	3	4
Small cuboid	1	1	1	11
Small cylinders	-	5	6	3
Large cuboid with cage bars	3	6	5	-
Large cylinders with cage bars	10	2	2	-

provides the ranking results for question 5, where the larger cylinders with virtual cage bars was the most preferred option. The users found that the use of virtual bars increased the perception of depth and made the safety volume clearer.

Question 7 evaluated the effectiveness of visualising safety zones. Results for questions 7a and 7b indicated that virtual cage bars were preferred. Question 7c showed that using the red colour to indicate safety zones was acceptable. Question 7d suggests that users found the dynamically changing size of cuboids to be non-distracting while the robot arm was in motion, although some users, as noted above, did say that the large cuboids made them feel less safe than other configurations. Nonetheless, overall, the perception of the visualised safety zones was positive, as indicated by question 7e.

Question 6 considered the comfort of using the HoloLens 2 headset and question 8 considered the participants thoughts on the overall system. The HoloLens 2 headset was found to be mostly comfortable by the participants, although two participants found it less comfortable. The results of questions 8a and 8b indicated that users were comfortable with the system and did not require additional assistance despite their lack of AR experience. However, the two participants who were less comfortable wearing the HoloLens 2 were also the only ones who answered ’neither agree nor disagree’ to these questions. Question 8d asked participants to predict how quickly others could learn to use the system, to which the responses were positive. Finally, question 8e focused on the system recovery process in the case of safety violations. A floating button was used for starting the process, and a pop-up menu appeared immediately if the safety zone was breached. Users found these easy to use and intuitive.

4.2. The Virtual Robot Arm Experiment

Table 4. Questionnaire results for the virtual robot arm experiment. Note that ‘-’ means 0 to reduce visual clutter.

	Strongly Disagree	Disagree	Neither Disagree Nor Agree	Agree	Strongly Agree
Configuration
Q1–configuration made it easier to do the task
Small cuboid	-	-	-	1	-
Small cylinders	-	-	-	2	2
Large cuboid with cage bars	-	-	-	2	4
Large cylinders with cage bars	-	-	-	-	3
Q2–configuration made me trust the robot arm
Small cuboid	-	-	-	-	1
Small cylinders	-	-	-	1	2
Large cuboid with cage bars	-	-	-	2	4
Large cylinders with cage bars	-	-	-	2	2
Q3–configuration made me feel safe
Small cuboid	-	-	-	-	1
Small cylinders	-	-	-	-	3
Large cuboid with cage bars	-	-	-	1	6
Large cylinders with cage bars	-	-	-	2	1
Q4–would choose this configuration again
Small cuboid	-	-	-	-	1
Small cylinders	-	-	-	1	2
Large cuboid with cage bars	-	-	-	2	2
Large cylinders with cage bars	-	-	-	1	5
HoloLens 2
Q6a–easy to use	-	1	-	4	9
Q6b–comfortable to use	-	-	2	4	8
Visualisation
Q7a–red transparency without virtual bars worked well	2	5	3	2	2
Q7b–red transparency with virtual bars worked well	1	1	1	3	8
Q7c–red is a good colour to use for the safety zone display	-	-	2	7	5
Q7d–dynamic cuboid is not distracting	3	1	-	7	3
Q7e–safety zones were well displayed	-	-	-	8	6
The System
Q8a–user confident with system	-	-	-	4	10
Q8b–interaction was easy	-	-	1	4	9
Q8c–need the help of a technical person	5	9	-	-	-
Q8d–people would learn the system quickly	-	-	1	3	10
Q8e–if a mistake was made, it was easy and quick to recover	-	-	1	6	7

summarises the results from the questionnaire used for the 14 participants in the virtual robot arm experiment (which were different participants to those in the real robot arm experiment). In contrast to the real robot arm experiment, whilst configurations with cage bars were more popular, the preferred shape was less clear. Again, the edges and corners of the cage bars made it easier to recognise the volume of the safety zones. In relation to shape, some said that the multiple cylinders resulted in a cluttered visual display and that a single cuboid was a clearer and more manageable option. Perhaps having multiple virtual elements, i.e., safety zone, arm and wooden blocks, compounded this feeling in contrast to the physical certainty of a real robot arm and real blocks.

Table 5. Numbers of users who ranked each configuration (question 5) for the virtual robot arm experiment, where the rating is 1 = best and 4 = worst.

	Rankings
Configurations	1	2	3	4
Small cuboid	1	3	4	6
Small cylinders	3	4	6	1
Large cuboid with cage bars	6	4	1	3
Large cylinders with cage bars	4	3	3	4

provides the ranking results for question 5, where the large cuboids with cage bars and large cylinders with cage bars were the most preferred options. The users found that the use of virtual bars increased the perception of depth and made the safety volume clearer, as in the real robot arm experiment.

Question 7 assessed the effectiveness of visualising safety zones. Similar to the real robot arm experiment, the participants preferred the use of virtual cage bars. Question 7c indicated that the use of the colour red to indicate safety zones was acceptable. The dynamically changing size of cuboids while the robot arm was in motion, as shown by question 7d, did not distract the users, similar to the results of the real robot arm experiment. The overall perception of visualised safety zones was positive, as demonstrated by question 7e, again, consistent with the real robot arm experiment.

Questions 6 (the comfort of using the HoloLens 2 headset) and 8 (participants’ thoughts on the overall system) gave similar results to the real robot arm experiment. Question 6 showed that participants generally found the use of the HoloLens 2 headset comfortable and question 8 showed that they were comfortable using the system.

4.3. Comparing the Results of the Real and Virtual Robot Arm Experiments

For questions 1–4 and questions 6–8, we used a Mann–Whitney U test to compare the results of the real and virtual robot arm experiments. This non-parametric test was chosen, as our data set did not meet the normality assumption. The test uses a comparison of the medians of the two sets of results. Following a similar procedure to previous work [21,22,23], we converted the Likert scale scores for each participant into numerical scores for each question for each experiment. SPSS Statistics (v28) was then used to apply the Mann–Whitney U test to these values, giving a comparison between the set of participants for each question across the two experiments. For question 5, we used a comparison of means within participants, with separate results for each experiment.

For questions 1–4, we had to convert each participant’s Likert score into a value to use in the Mann–Whitney U test. To do this, the Likert scale category range was mapped to a numerical value, such that ’Strongly disagree’ was assigned the value of 1 and ‘Strongly agree’ was assigned a value of 5. Likewise, the four configurations (small cuboid, large cuboid with cage bars, small cylinders, large cylinders with cage bar) were mapped to the values 1…4. The score for a participant was then formed by multiplying the numerical value of the Likert scale category by the configuration. For example, if a participant selected the ‘Agree’ response option, a numerical value of 4, and the option is ‘Large cuboid with cage bars’, a numerical value of 3, the score for that question would be calculated as 4 * 3 = 12. A similar process was followed for questions 6–8. However, here, only the Likert category requires a numerical value, as the questions do not contain configuration options.

Figure 7 gives the results for questions 1 to 4 (considering which configurations made it easier to do the task, trust, safety, and future choice of configuration). This shows that the Mann–Whitney U results for questions 1–3 are significant, but the results for question 4 are not significant. Additionally, the result for a combination of questions 1–4, the results are significant. For question 1, considering this significance in conjunction with the Likert scores shown in Table 2 and Table 4, it suggests that participants in the real robot arm experiment preferred the large cylinders with cage bars more. Questions 2 and 3 suggest that this configuration also led to an increased level of trust and safety for participants using the real robot arm. The combined significance for questions 1 to 4 also adds to the suggestion that the participants preferred the large cylinders with cage bars more for the real robot arm.

Question 5 asked the participants to rank the configurations in order of preference. Table 3 suggests that users ranked ‘Large cylinders with cage bars’ highest. Figure 8a gives the results when each participant’s ranking is used to create a set of mean scores with standard errors for each configuration. It is clear that ‘Large cylinders with cage bars’ is the highest rank choice. Interestingly, the second-highest ranked option is ‘Large cuboid with cage bars’, again supporting the observation that adding cage bars helps with the visualisation of safety zones when a real robot arm is used. In contrast, the results for the virtual robot arm experiment are less clear, as illustrated in Figure 8b. Here, the highest ranked option is ‘Large cuboid with cage bars’, but there is little difference from the second- and third-highest options ‘Small cylinders’ and ‘Large cylinders with cage bars’. As we commented above, the increase in the amount of virtual elements in the scene (i.e., robot arm, wooden blocks, and safety zones) could have contributed to this.

For questions 6–8 (the comfort of using the HoloLens 2 headset, visualisation options and participants’ thoughts on the overall system, respectively), only questions 6a and 7d showed a significant difference for the two experiments, with U-values of 50 and 52, respectively, both below the critical value of U, 61, at p < 0.05. Question 6a considered whether or not the HoloLens 2 was easy to use. The results suggest that the participants in the virtual experiment found the HoloLens 2 easier to use. Perhaps this was because all the experimental conditions were virtual, i.e., there was no physical robot arm to consider. Question 7d considered whether the cuboid safety zones dynamically changing in size during the experiments was distracting. The results show that participants in the real robot arm experiment found this dynamic change less distracting than those in the virtual experiment. Perhaps this was because they were more focused on the physical presence of the real robot arm. These are aspects which need more exploration.

4.4. Discussion

Previous studies utilised transparent volumes in AR studies [2,4,5,6,7]. The findings in our study clearly show that the addition of virtual cage bars can increase the user’s perception of the safety volume when a real robot arm is used. Despite the fact that the safety areas with virtual cage bars were larger than those without, participants in the study preferred the larger safety areas with bars. The bars made it easier to track the area. These results indicate that virtual cage bars can play an important role in enhancing the user’s perception of depth and spatial awareness, which could ultimately improve safety in a given environment.

Our experimental findings also suggest that particular aspects of virtual and real robot arm experiments have different impacts on users. Despite creating a uniform testing environment for both virtual and real robot experiments, users’ perceptions differed between the two experiments. The preferred safety zone configuration when a real robot arm was used was much clearer (i.e., large cylinders with virtual cage bars) than in the virtual robot arm experiment, although the small number of participants in each experiment must also be considered. This difference could be attributed to the absence of genuine risk in the virtual robot arm experiment. Using a virtual robot arm is possibly perceived as safer than using a real robot arm. This changes the perception of the user and has potential implications for the transferability of skills between virtual and real environments. A further aspect of the virtual robot arm experiment that could impact on transferability is the sense of touch. In the virtual robot arm experiment, virtual wooden blocks were manipulated by the user. Comments from some users suggested that they had difficulty grabbing the virtual blocks. This was not the case for the real wooden blocks. This could suggest a potential issue for the transferability of precision and efficiency skills in training situations. Overall, transferability remains an open research question [9,14,15].

In both experiments, using the HoloLens 2 did not cause any discomfort or fatigue for participants. In fact, most participants found the HoloLens 2 headset to be comfortable and lightweight and did not experience any discomfort or motion sickness during the experiments. This is in line with other studies [4,5,15] that have found that the HoloLens 2 is comfortable to wear and does not cause significant discomfort or fatigue during extended use. However, it is important to note that our study was conducted with a relatively small sample size and for a short duration of time, so further research with larger sample sizes and longer duration would be needed to validate this.

5. Conclusions

The objective of this study was to assess user preferences and opinions on the size, shape, and appearance of safety zones when collaborating with a robot arm. To accomplish this, we employed a collaborative pick-and-place application in two experiments—one with a real robot arm and one with a virtual robot arm—whilst varying the configuration of the safety zone.

Whilst previous studies have used transparent safety zone visualisation in real robot arm experiments, our results show that users preferred the use of virtual cage bars as part of safety zone visualisation when working with real robot arms. Virtual bars gave better demarcation of safety volumes. If a user did violate the safety zone with their hand(s), the robot arm stopped, and any further movement was under the control of the user. We feel that this measure, together with the clearer demarcation of the safety zone, could lead to more trust and more efficient human–robot collaboration. However, it was less clear which safety zone configuration was preferred in the virtual robot arm experiment.

Our work suggests that the issue of transferability of skills between a virtual scenario and a real scenario is more complicated. Previous work considered transferability in terms of training. Our work suggests that the visualisation itself must also be considered since the users’ perceptions of the environment they are in may also impact on the judgement of the visualisation being employed. The results of the real robot arm experiment clearly showed a preference for large cylinders with virtual cage bars, whereas the results of the virtual robot arm experiment were less clear. Future work should consider how to disentangle such related aspects in transferability.