Benefits and Challenges of Virtual-Reality-Based Industrial Usability Testing and Design Reviews: A Patents Landscape and Literature Review

Abstract

With the introduction of new devices, industries are turning to virtual reality to innovate their product development processes. However, before the technology’s possibilities can be fully harnessed, certain constraints must be overcome. This study identifies the benefits and challenges of virtual-reality-based usability testing and design reviews in industry through a patents and articles review. We searched Derwent Innovation, Scopus, and Web of Science and identified 7 patent filings and 20 articles. We discovered an increase in patent filings since 2016 and strong development in the technology space, offering opportunities to enter an area while it is still young. The most frequently researched field is the automotive industry and the most used device is the HTC VIVE head-mounted display, which is frequently paired with motion capture systems and Unity 3D game engines. Virtual reality benefits design reviews and usability testing by providing the visualization of new angles that stimulate novel insights, increasing team engagement, offering more intuitive interactions for non-CAD specialists, saving redesign cost and time, and increasing participants’ safety. The challenges faced by virtual-reality-based prototypes are a lack of realism due to unnatural tactile and visual interactions, latency and registration issues, communication difficulties between teams, and unpleasant symptoms. While these constraints prevent virtual reality from replacing conventional design reviews and usability testing in the near future, it is already a valuable contribution to the industrial product development process.

1. Introduction

Given the complexity of products that are under development, designers must perform a number of procedures to guarantee that the finished product fits customer demands and is accepted by the market [1]. As a result, technological advancements are continually assisting the improvement of product quality [2].

The strategies employed to ensure the usability of the product include elements of competitiveness, distinctiveness, and good practice [3]. In standard ISO-9241 of the International Organization for Standardization (ISO), usability is defined as “the extent to which a system, product or service can be used by specified users to achieve specific goals with effectiveness, efficiency, and satisfaction in a specified context of use” [4].

The usability level assesses how enjoyable and simple the product is to use based on the customer’s experience. The usability attribute is influenced by the customer’s perception of how the product is used. An example of this would be a satisfactory level of perceived experience when a simple operation, such as turning on a radio system, followed by an attempt to set up a specific radio station, is easily completed as expected by the user [5]. Usability refers to a user interface’s many characteristics, which are related to the attributes of learnability (easily learned), memorability (easily recalled), errors (low error rate when used), and satisfaction (pleasant to use) [6].

Industries employ prototypes for usability testing to anticipate any potential customer issues with a product’s operation. As a result, flaws can be addressed during the development process, reducing the likelihood of customer dissatisfaction when the final version is released to the market [7].

Typically, industries follow a method to test usability in which individuals interact with the system (tasks). A moderator can collect quantitative data during these tests, such as the number of unintentional mistakes, slips, or omissions by a user while performing a task. To collect data for qualitative metrics, moderators assess the user’s behavior through tasks and interviews [8,9]. The usability test with users makes it possible to assess a product’s efficiency, effectiveness, and satisfaction.

Despite the advantages of physical prototypes, their production is time-consuming and costly. Furthermore, once prototypes are produced, any rework deriving from design modifications and updates is challenging [10].

In this context, the industry is adopting virtual reality for verification, simulation, and interaction with virtual prototypes, which are less expensive and take a shorter time to manufacture than physical prototypes [11]. Virtual reality is defined as “a computer-generated digital environment that can be experienced and interacted with as if it were real” [12]. Research indicates that virtual reality facilitates design verifications and the conventional review process with physical prototypes can be replaced by its virtual counterpart, which in turn could lead to cost and time reductions for manufacturers [13]. Thus, in the product development process, the use of virtual reality technologies to enhance engineering design reviews has been an area of interest for researchers since the advent of modern virtual reality [14,15,16].

Indeed, with the emergence of low-cost devices, such as markerless motion capture, an increasing number of organizations have used them for ergonomics evaluations in the manufacturing sector without fully understanding their overall performance and drawbacks [17]. The global virtual reality market is projected to grow from $6.30 billion in 2021 to $84.09 billion in 2028 [18]. Due to low costs and the high quality of virtual reality devices, virtual reality could become ubiquitous for several engineering processes and support everyday engineering tasks [19].

However, virtual reality has its own barriers. Studies indicate that consumers may struggle to adapt to virtual reality devices in a virtual environment [14] and the findings of standard usability testing are influenced by technological constraints, such as a lack of graphic quality and the hardware processing power to support the fluid visual movement. There are still research challenges to be addressed until industries can fully benefit from the technology’s potential [20].

To the best of knowledge, there is no previous work that reviews the patents and research on the benefits and challenges of virtual reality for design review and usability testing in industrial product development. Knowledge on this subject may provide a reference for practitioners and researchers as they continue to develop novel solutions for design reviews and usability testing using virtual prototypes. Furthermore, patents are sources of strategic information for industry, and they are used for technological management and innovation. Therefore, a patent prospection can drive the mapping of a given technology, enabling the collection of crucial data on technological developments, inventors, market trends, and other factors.

Thus, this study aims to identify the benefits and challenges of virtual-reality-based usability testing and design reviews in industry through a patents and articles review.

This document is organized as follows: Section 2 describes the materials and methods utilized, Section 3 presents and analyzes the results, and Section 4 provides our conclusions and suggestions for further research.

2. Materials and Methods

This systematic review followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines, which was designed to “help systematic re-viewers transparently report why the review was done, what the authors did, and what they found” [21]. Additionally, the process provided in [22], which consists of seven steps, was utilized: planning, defining the scope, searching the published research, assessing the evidence base, synthesizing, analyzing, and writing.

An expert in virtual-reality-based usability testing of industrial product development developed the initial search strategy. The candidate strategy was then assessed by two researchers of the use of virtual reality for innovation in the industrial product development process. Finally, an interdisciplinary focus group, consisting of four senior practitioners specialists on product development in the automotive industry and virtual reality, gathered for a brainstorming session. The strategy that resulted from this validation process is described in the sections that follow.

2.1. Planning

The knowledge bases that will be investigated are determined during the planning step [22].

In the case of patents, a search for records was conducted in the Derwent Innovation Index database. Derwent Innovation was selected because it has 39.4 million patent families and 81.1 million patent records, with coverage from 59 international patenting authorities and two journal sources. A database must be evaluated using certain key criteria for a patent search, and commercial tools, such as Derwent Innovations, offer unique resources that improve the database’s capacity to retrieve information. The “Smart Search” tool, for example, makes use of artificial intelligence to improve keyword discovery [23]. Another feature is Derwent World Patents Index (DWPI), the world’s most comprehensive database of enhanced patent information, with expanded patent titles and abstracts, and English abstracts of the original patents. It has a sophisticated classification system and patent family information with non-convention equivalent identification [24]. Section 2.3 provides details on “Smart Search” and the DWPI features.

Regarding articles, the search was conducted in the scientific databases Scopus and Web of Science. These databases were chosen because they are reliable and multi-disciplinary scientific databases of international scope with comprehensive coverage of citation indexing, providing the best data from scientific publications. Scopus now includes 81 million curated documents [25], whereas Web of Science covers more than 82 million entries [26].

2.2. Defining the Scope

Defining the scope relates to properly stated research questions [22].

Three pertinent research questions were selected for this systematic review, namely:

Q1:

How are patents in industrial virtual reality-based usability testing and design review characterized?

Q2:

In terms of application fields, methods, hardware, and software involved, how is current knowledge on the application of virtual reality in usability testing and design review in industry defined?

Q3:

What are the benefits and challenges of using virtual reality for usability testing and design review in industry?

2.3. Literature Search

The literature search step entails investigating the database defined in the planning step with a particular string depending on the research questions posed in defining the scope step [22].

The candidate search phrases were collected from the titles, abstracts, and keywords sections of two previously published articles. The prospective search phrases were then peer-reviewed by five other members of our research team, who have experience of utilizing virtual reality for product design in industry. The following candidate peer-reviewed search terms were provided by the Derwent Innovation database: (“USABILITY EVALUATION” or “USABILITY TESTING” or “USABILITY ASSESSMENT” or “USABILITY ENGINEERING” or “DESIGN REVIEW”) and (“MIXED REALITY” or “VIRTUAL REALITY” or “IMMERS*” OR “IMMERS* PROTOTYP*” or “VIRTUAL PROTOTYP*”) and (INDUSTR* or “PRODUCT DEVELOPMENT” or “PRODUCT DESIGN”).

Next, to identify relevant keywords, we submitted these search phrases into the Derwent Innovation database’s “Smart Search” tool. The “Smart Search” engine examined it to generate significant words about the technology mentioned in the text, and then extended those key terms to include synonyms. Next, using the expanded search criteria, toll scanned all Derwent Innovation patent databases and showed the most relevant patents linked to that technology. More information on the technology behind the “Smart Search” resource may be found at [23].

Similar search phrases were used for article retrieval, with slight modifications to accommodate the search engine specifications of the Scopus and Web of Science knowledge databases.

The search was carried out in January 2022, and the preliminary identification yielded 11 individual patent records and 180 articles.

2.4. Assessing the Evidence Base

The assessment step applied inclusion and exclusion criteria filters to reduce the number of related records identified during the literature search step [22]. By combining the following exclusion criteria, we were able to limit the number of records returned during the previous step:

• E1: Exclude patents filed or articles published before 2016;
• E2: Exclude articles not written in English language;
• E3: Exclude patent applications that are no longer alive;
• E4: Exclude patents and articles not related to the industrial domains, such Medicine, Social Sciences, Physics, and Environmental Science.

According to research, the advent of technologically advanced virtual reality headsets in this year represented a breakthrough for virtual reality applications; therefore, we excluded documents filed or published before 2016 [27,28] and practitioners [29,30].

We applied the E2 criterion exclusively on the articles, since the Derwent Database Enhanced Patent Data includes patent documents that have been translated into English.

After applying exclusion criteria, we screened nine individual patent records and 78 articles.

2.5. Synthesizing and Analyzing

The retrieved documents were then combined with project-related features [22]. The documents were subjected to a single screening, in which a reviewer with expertise in usability testing and virtual reality technology inspected each record in order to locate relevant patents and articles linked to the Q1, Q2, and Q3 research questions. The papers were chosen based on their title, abstract, and author keyword fields, as well as their connection to the project’s purpose:

• Product design applications (rather than, for example, building information modeling (BIM) applications);
• Virtual reality providing users with immersive experiences (because some researchers or database-automated mechanisms correlate the terms “mixed reality”, “augmented reality”, or “virtual environment” with immersion properties);
• Studies of the usability of virtual reality devices and the equipment itself (rather than a usability evaluation of the industrial product being developed).

Finally, after excluding duplicate entries from the Scopus and Web of Science databases, we included 7 patent records and 20 articles for examination.

The patents were analyzed using Derwent analytical and insights tools. The retrieved documents were uploaded to the Mendeley Reference Manager tool, and spreadsheets and visualizations were created in Microsoft Excel.

Figure 1 depicts the flow of the systematic review from searching the published research to synthesizing processes.

3. Results and Discussion

The research questions Q1, Q2, and Q3 were addressed in order to identify the opportunities, benefits, and constraints of using virtual reality for usability testing and design reviews in industry. In the sections that follow, we analyze our findings.

3.1. Patents Landscape

The seven patent records retrieved by the search strategy are shown in

Table 1. List of retrieved patents.

Ref.	Publication Number	Title	Assignee
[31]	CN111414083A	Method and device for availability test of space	CCB Fintech; China Construction Bank
[32]	CN111414084A	Space availability test laboratory and using the method and device thereof
[33]	JP2021068278A	A design-review system and the design-review method	Nippon Steel Texeng
[34]	US20210011593A1	System for rendering applications based on real-time accessibility assessment	Bank Of America
[35]	US20210090343A1	A method and a system for design reviews and trainings	Activa Innovations Software
[36]	US20190227626A1	Neuro-adaptive body sensing for user states framework	HRL Lab LLC
[37]	KR20190088710A	Method for evaluating usability on vehicle infotainment systems	Hanyang University

.

To address the first research question, these patents were analyzed to answer frequent concerns and uncover patterns in assignees, filings per year, the International Patent Classification (IPC), and benefits.

Q1: How are patents in industrial virtual-reality-based usability testing and design review characterized?

In terms of assignees, their identification may help the discovery of industry leaders, the evaluation of possible rivals, and the identification of niche players.

The widely fragmented dispersion of patent applications across many assignees is intriguing: just one assignee, CCB Fintech and China Construction Bank, submitted two patents, whilst the other patents were filed by different assignees.

Rather than a relatively equal-sized, but large, portfolio held by a few companies, which indicates an active competitive market with strong investments by multiple companies, suggesting that the market is difficult to enter, we discovered a large number of assignees, each with a small number of records, demonstrating a developing technology space. This domain may be entered by acquisition or rapid development. There are several companies, each with a small number of patents, indicating an opportunity to enter this area while it is still young, either by licensing existing technology, purchasing one of the players, or developing new technology that is not already patented.

It is also worth noting that three of the six assignees are major financial institutions, while one university filed one of the seven patents.

In terms of patent filings per year, Figure 2 illustrates the yearly filing of patents from 2016 to 2020. We omitted 2021 patents from the analysis since they were still being filed at the time of this study.

We observed an initial period spanning the years 2016 and 2017, with no patents registered. This trend changed in 2018, followed by growth in 2019 and 2020.

These results support the claim in [28] that the year 2016 marked a technological breakthrough in the domain of virtual reality. Before 2016, commercial virtual reality systems required users to connect a headset, controllers, and sensors to an external high-end computer, which was an expensive, bulky, and inconvenient setup. Thus, the current all-in-one virtual reality systems are a significant step forward from only a few years ago [38]. The upward trend slowed in 2020, although one likely explanation is the 18 month patent legal secrecy restriction.

A important trend is that not only is the global virtual reality market projected to grow from $6.30 billion in 2021 to $84.09 billion in 2028 [18], but this rise may be much greater, given that the COVID-19 epidemic boosted the usage of virtual reality even further [39]. As a consequence, this scenario suggests that the exponential increase in patenting filings related to virtual reality and design review and usability testing will continue for a few more years.

The International Patent Classification (IPC) is an approach to determining a standard classification for registered patents, thereby enabling the search for and access to technical information accessible in documents connected to the same subject. The system is a hierarchical patent classification system used in over 100 countries to uniformly classify patent material. It creates a separation into classes and subclasses that are applicable to various technical domains and aids in the standardization of patent classification.

Figure 3 illustrates the IPC classifications that we found in our review.

Section G (Physics), which covers all physics-related material, is designated to all seven patent filings. As the class level progresses and technological information is described in more depth, the most frequent class is G06 (computing calculating or counting), with subclasses G06F (electric digital data processing), G06Q (data processing systems or methods), and G06T (image data processing or generation, in general).

Other patents grouped under G Section are G02B (optical elements, systems, or apparatuses), G09G (arrangements or circuits for control of indicating devices using static means to present variable information), and G10L (speech analysis or synthesis; speech recognition; speech or voice processing; speech or audio coding or decoding).

Section A (Human Necessities) was also used to classify one patent, grouped under the A61 class (medical or veterinary science; hygiene), with subclasses A61B (diagnosis; surgery; identification) and A61N (electrotherapy; magnetotherapy; radiation therapy; ultrasound therapy).

In terms of benefits, we examined the seven patents in relation to the advantages of each invention, as described by its authors, and the novelty of each invention, i.e., the unique innovative feature introduced by the inventor that is not conventional and constitutes an improvement on existing technology.

The JP2021068278A patent [33] proposes to provide a design review system and method that reduce the time required for computer-aided design (CAD) data conversion. The proposed design review system includes a CAD apparatus, into which CAD data are produced or edited, and a data conversion unit that performs the data conversion from CAD data to virtual reality data. The advantage of the invention is that the design review system shortens the time required for data conversion, such as data conversion from CAD data to virtual reality data.

The US20210011593A1 patent [34] proposes a system for producing applications based on real-time accessibility assessments. The system identifies that a user is accessing an application on a device (such as a virtual-reality device), captures the real-time accessibility data, inputs it into a machine learning model, generates an accessibility score, and renders the application based on this score. The advantage of the invention is that the system identifies whether the user is accessing another application on the same user device and utilizes the accessibility score stored in a data repository to render the other application; if the amount of data stored in the data repository increase over a period of time, this improves the efficiency with which they are processed. The method enables the calculation of a real-time accessibility score to comprise the operating configurations of the user device, as well as the status of its hardware components, to establish internet connectivity in an effective manner.

The CN111414084A [32] and CN111414083A patents [31] are related. The former proposes a space usability testing laboratory comprising a test area for displaying a 3D model of the space based on immersive virtual reality, and the experience data of test users are collected during the test process [32]. The latter proposes a method for the usability testing of the space, which involves obtaining a 3D model of the space to be tested, and the test task is performed by a user wearing a virtual-reality-based wearable device [31]. According to the inventors, the advantages are that the user’s understanding of the user experience may improve this experience, space usability testing is more convenient, and its results are more accurate. Furthermore, the manner of implementation could truly display the design appearance, the user experience is real, the test result is accurate, and the test consistency and efficiency are high.

The US20210090343A1 patent [35] proposes a method for providing design reviews using virtual-reality devices, involving the processing of interactions of users in a virtual-reality format and the generation of output on the basis of users’ actions. The advantage of the invention is that the method would make it possible to generate a complete design review cycle in an easy manner, so that time consumption could be reduced effectively. The inventors claim that the technology would open the door to a range of new applications that have not been possible until now and that the invention would revolutionize design reviews with a radically new experience using virtual reality. Multiple users might collaborate remotely and perform design reviews within the virtual reality environments.

The US20190227626A1 patent [36] proposes a system for personalizing a human–machine interface (HMI) device based on the mental and physical state of a user. During the performance of a task in a simulation environment (such as training in virtual reality), the system extracts biometric features from data collected from body sensors and brain entropy features from electroencephalogram signals. Both data are correlated to generate a mental-state model. The mental-state model is deployed in a, HMI device during the performance of a task in an operational environment for the continuous adaptation of the HMI device to its user’s mental and physical states. The advantage of the invention would be that the continuous adaptation of the HMI to the mental and physical states of the user would reduce the workload and enhance decision-making. The need for unnecessary modifications in the interfaces would be eliminated, and the designs would be more user-centered and customized to the real needs of users.

Finally, the KR20190088710A patent [37] proposes a method for assessing the usability of automotive infotainment systems that comprises facilitating interactions between numerous virtual infotainment systems and drivers in a virtual driving environment while investigating the usability of vehicle infotainment systems. The usefulness of a virtual infotainment system is determined by evaluating both the execution time of the operation command and the running condition of a virtual car as a result of the execution of the operation command. The virtual car’s running state contains the virtual vehicle’s speed, its distance from the preceding vehicle, the distance between the lane and the virtual vehicle, and the steering angle during the execution of a drive operating instruction. The novelty of the invention is its evaluation of the cognitive load of a driver by using the vehicle infotainment system and its use of the driving state of a virtual vehicle in accordance with the performance time of the operation command and the performance of the operation command to evaluate usability for each of virtual infotainment system.

3.2. Scientific Mapping

Table 2. List of selected articles.

Ref.	Title	Publication Year
[40]	A novel design engineering review system with searchable content: knowledge engineering via real-time multimodal recording	2017
[41]	An Industry Case Study: Investigating Early Design Decision Making in Virtual Reality	2017
[42]	Automatic derivation of geometric properties of components from 3d polygon models	2017
[43]	Interaction techniques for virtual reality based automotive design reviews	2017
[44]	VR-based operating modes and metaphors for collaborative ergonomic design of industrial workstations	2017
[15]	A virtual reality supported 3D environment for engineering design review	2018
[45]	Chances and Limitations of a Virtual Reality-supported Tool for Decision Making in Industrial Engineering	2018
[46]	Comparison of the Usability of a Car Infotainment System in a Mixed Reality Environment and in a Real Car	2018
[47]	A study of deriving usability evaluation factors on virtual reality contents	2019
[16]	Analyzing the potential of Virtual Reality for engineering design review	2019
[48]	Combining Virtual Reality (VR) Technology with Physical Models—A New Way for Human-Vehicle Interaction Simulation and Usability Evaluation	2019
[49]	Usability Evaluation of VR products in Industry—A Systematic Literature Review	2019
[50]	User centered design of interaction techniques for VR-based automotive design reviews	2019
[51]	User Experience-and Design-Oriented Virtual Product Prototyping System	2019
[52]	Virtually the Same Experience? Learning from User Experience Evaluation of in-Vehicle Systems in VR and in the Field	2019
[53]	Virtual-Reality-based Approach for Cognitive Design-Review and FMEA in the Industrial and Manufacturing Engineering	2019
[54]	A Transdisciplinary digital approach for tractor’s human-centred design	2020
[19]	Optimizing the Design Review Process for Cyber-Physical Systems using Virtual Reality	2020
[20]	Supporting teamwork in industrial virtual reality applications	2020
[55]	Implications of Virtual Reality on Environmental Sustainability in Manufacturing Industry: A Case Study	2021

shows the twenty articles selected by the search strategy.

These articles were reviewed in order to answer the research questions Q2 and Q3. It is worth noting that the attributes addressed by Q2 and Q3 are not abstracted at the same level and are not always mutually exclusive in the studies reviewed; furthermore, some studies did not mention some of these aspects. Our findings are provided in the subsections below.

Q2: In terms of the application fields, methods, hardware, and software involved, how is current knowledge on the application of virtual reality in usability testing and design review in industry defined?

3.2.1. Application Fields and Methods

The application field of an article was defined similarly to [56]’s approach as the industry and/or technical environment targeted by the study. One of the twenty articles selected is a literature review [49] that assessed some of the studies under investigation. As a result, we omitted it from our quantitative analysis of application fields in order to avoid counting the same study twice.

The application fields observed can be divided into three main categories: (1) automotive industry, (2) industrial machinery manufacturing, and (3) laboratory environment or field undetermined. Figure 4 shows the distribution of papers in each category.

The automotive industry was the subject of seven articles (37%) [43,46,48,50,52,54,55]. It was expected that a substantial number of virtual reality studies would be conducted in the vehicle industry. These findings are congruent with those of [56]. The importance of the automobile industry may be related to the fact that virtual technologies have long been utilized in this sector in a range of specialties and applications, such as manufacturing, training, and maintenance, to mention a few [13,57,58]. Departments such as design, engineering, maintenance, and assembly are already using the technology to support practically everything in the automotive sector, from product development to task assistance in machine assembly or maintenance procedures. Furthermore, the automotive industry is one of the most mature manufacturing industries, with cutting-edge technologies being used for the first time on a regular basis [56].

Regarding the evaluated systems, three studies examined the virtual interaction with an automobile multimedia system, utilizing distinct basic functionalities and, as a result, producing diverse usability testing scenarios. Among the functions are a navigation system, air conditioning, a phone, a radio, a driving assistance, and car parking. Other systems undergoing design review or usability testing include a BMW vehicle’s exterior design and Audi [46] and Volvo multimedia stations.

Some usability tests guided interaction operations in which participants were immersed in dynamic virtual environments, such as driving a vehicle in a virtual city [46]. For dynamic testing [46,52], participants drove on public roads in a real-world setting, interacting with the vehicle’s panel controls and performing tasks as assigned by the moderator. Each participant interacted with the digital system while driving around virtual streets in the virtual usability testing environment, while a moderator observed the participant’s behavior and collected metrics.

Two studies did not consider how individuals interacted with a product prototype. The authors of [43] focused on design issues, asking their participants to utilize a variety of assessment techniques to determine the best way to interact with a virtual automobile model. The following exchange possibilities were investigated in the study: voice command, gestures, first-person vision, and physical controllers.

Due to the variety of systems studied, several operations, such as air conditioning, navigation, telephone, and audio controls, were used by various studies in diverse contexts of use, command design, and visual representation. This limits the evaluation of the connection between the results obtained in the research [46,48,52], since the use of standard methods for usability testing would benefit from a thorough examination of the data and conclusions gathered from various investigations.

Five studies (26%) reviewed industrial machinery manufacture. Three of them [15,16,45] were published by the same research group and focused on industrial power units, while the other two looked at hydraulic pump production [41] and specific machines for automation technologies [53], respectively.

Seven (37%) of the studies did not indicate the field of application [19,20,40,42,44,47,51]. It is common for virtual-reality applications developed by an academic research team to be tested in their own laboratory with prototypes and artifacts [59]. Figure 5 is an example of a study conducted in a laboratory environment.

In terms of the methods and metrics used, we identified that the studies focused on the accuracy of experimental results when compared to testing with conventional CAD models and physical prototypes. The articles under consideration spanned a broad range of methods and metrics. The main approaches used by the studies under examination are listed in

Table 3. Methods and metrics used in the studies reviewed.

Ref.	Metrics
[46,48]	Time to Complete the Task
[43,46,52]	Scale Usability Scale (SUS)
[46,48]	Number of Mistakes
[44,54]	Rapid Upper Limb Assessment (RULA)

.

Despite the fact that the reviewed studies utilized a wide variety of methods for collecting qualitative and quantitative data, some of them were referenced in several studies.

Three studies used the Time to Complete the Task metric. The Time to Complete statistic is used to calculate how much time each participant spent on each task.

The qualitative data were often gathered by providing a standardized questionnaire to respondents after they completed the tests. The System Usability Scale questionnaire was often used to measure the perceived user-friendliness of the system with which the participant interacted. The SUS approach is frequently used in industry and is effective for system comparison [46].

Another often-used metric is the Number of Mistakes, in which moderators utilize visual observation to quantify the mistakes made by participants while performing tasks other than those for which they were trained.

The Rapid Upper Limb Assessment score was used in two studies. The RULA is one of the most frequently used measures for assessing employees’ exposure to ergonomic risk while performing manual upper body activities such hand, neck, and limb twisting [46].

The Sense of Presence Inventory (ITC) and User Experience Questionnaire (UEQ) are two methodologies worthy of note. The Sense of Presence Inventory is a metric used to examine how people perceive physical space (defined as “a sense of physical placement in the mediated environment and interaction with and control over parts of the mediated environment”) [52]. The User Experience Questionnaire is used to obtain evaluations from participants on the following aspects: attractiveness (overall impression of the product), perspicuity (ease with which the product can be understood and used), efficiency (ability to use the product efficiently), dependability (the feeling of being in control of interactions), stimulation (excitement and motivation to use the product), and novelty (perception of the innovation and creativity of the product).

Other methods and metrics were only referenced in one study, such as FMEA (failure modes and effects analysis), criticality analysis (CA), and expected final distance [53]; intuitiveness and task weight [45]; readability of information, command display, and function controls [44], as well as several others, such as heat maps, pupil diameter, heart rate (hr), breathing rate (br), activity (vmu), posture, heart beats per minute, breath per minute, etc.

Table A1. Methods, hardware, and software referred to in each article reviewed.

Ref.	Objectives, Methods, Hardware and Software
[41]	Described an industry case study of the use of immersive VR as a general design tool with a focus on the decision-making process; Nintendo Wii remote; three-walled immersive environment (3 projectors + 2 walls (4 m 3 m and 3 m 3 m) and a floor (4 m 3 m); infrared-based optical tracking; stereo glasses; surround sound system; TEAMCENTER LIFECYCLE VISUALIZATION 9.1 Siemens PLM Software
[15]	Proposed a low cost multimodal VR-supported tool for design review; HTC Vive; PC; GeForce GTX 1080 8 GB GPU; CATIA; Unity3D; 3dsMax
[16]	Described the development and evaluation of a VR-based tool to support engineering design review; HTC Vive; PC; GeForce GTX 1080 8 GB GPU; CATIA; Unity3D; 3dsMax
[45]	Discussed a set of application areas for VR in industry and modes of visualization and interaction and described the implementation of a light-weight VR-system for industrial engineering applications; HTC Vive; PC; GeForce GTX 1080 8 GB GPU; CATIA; Unity3D; 3dsMax
[40]	Prototyped and tested a potential knowledge engineering capture and reuse solution, demonstrating real-time user-logging using virtual design environments focused on team-based design reviews; Scale Usability Scale (SUS); full-HD (1920 × 1080) 3D projector; 3.2 m × 1.8 m power wall projection; active shutter glasses; UbiITS framework; microphones; cameras
[47]	Derived the factors for evaluating usability of virtual reality (VR) contents; unspecified
[42]	Presented a set of algorithms to automatically determine the geometrical properties of machine parts based only on their triangulated surfaces/Intel Core i7-3770 CPU 3.4 GHz; Platform for Algorithm Development and rendering (PADrend 1.0); Escript
[19]	Addressed the design review process for CPS by introducing a VR-driven concept, taking CPS characteristics into account, like the use data of (previous) product instances in the field as an additional source of information; workstations, HTC VIVE; 3D Unity; Autodesk Forge; Autodesk Fusion 360; Google Firebase;
[20]	Presented approaches to counteract this issue in a shared VR space for industry purpose; Xbox-Controller; HTC Vive Pro
[51]	Proposed a virtual product prototyping system based on interaction of consumer and producer in terms of user experience and design; HTC Vive; motion capture; pupil tracer; ECG/GSR sensor; Space UI; 3D 360-degree virtual space; Unity 3D; 360 VR images method
[44]	Evaluated two new operating design modes and their collaborative metaphors enabling two actors, a design engineer and an end user, to work jointly in a collaborative virtual environment for workstation design; RULA; large four-wall immersive room, size was 9.60 m long, 3.10 m high and 2.88 m deep; flystick device; a desktop computer with two windows;
[48]	Utilized the currently popular virtual reality technology to solve the contradiction between the increasingly complex technologies applied in the automotive and the gradual shortening design and development cycle of the automotive due to market pressure; Time to Complete the Task; Number of Mistakes, T-test; 1:1 cockpit 2018 Mercedes-Benz E200L; PC; HTC vive; Logitech G29 steering wheel kit; Unity
[54]	Proposed a mixed-reality set-up to support human-centered product and process design, where systems and humans interacting with them are monitored and digitalized to easily evaluate human–machine interactions, with the scope to have feedback for design optimization; Dreyfuss 3D; OWAS/RULA/REBA; human joint angles; Ergonomic ratings (Factory operations); Eye fixation; Pupil diameter (PD); Gaze plot, heat maps; pupil diameter; heart rate (HR); breathing rate (BR); activity (VMU); posture; heart beats per minute; breath per minute; magnitude of resultant vector of mean; acceleration in three directions; stooping angle on sagittal plane; Siemens JACK; VICON tracking; VICON Bonita cameras; 3D-printed rigid bodies with markers; Tobii Pro; Zephyr BioHarness; GoPro; XSensor IX500; New Holland T5.120 tractor model cabin
[55]	Explored the feasibility of developing VR technologies to reduce environmental impact, drawing from a case study in an automotive company; 3D laser scanner; HTC Vive; Unity3D
[49]	Reviewed the Usability Evaluation Methods practiced by Industrial researchers while building VR Products; Systematic literature review
[53]	Discussed the use and the potential of the virtual reality technology in the industrial environment; FMEA (failure modes and effects analysis); criticality analysis (CA); completion time per trial; expected-final distance; HTC Vive; Unity 3D
[43]	Reported insights of their approach aiming at appropriate VR interaction techniques supporting designers, engineers, and management executives optimally in design assessment; Scale Usability Scale (SUS); intuitiveness; task weight; 55” LCD Full HD; MS Kinect; Apple iPad; 3DVIA Virtools, MS Kinect SDK; MS Speech API
[46]	Investigated whether the usability evaluation of a car entertainment system within an MR environment provides the same results as the evaluation of the car entertainment system within a real car; time to complete the task; number of mistakes; System Usability Scale (SUS); readability of information; command display; function controls; driver’s seat, a steering wheel, three pedals and individual control panel with gearshift lever knob and RPB from the center console of an Audi A4; Oculus Rift; Leap Motion; Unity3D
[50]	Reported insights of their user-centered approach aiming at appropriate VR interaction techniques to support designers, engineers, and management executives optimally in design assessment; Scale Usability Scale (SUS); intuitiveness; task weight; 55” LCD with Full HD resolution; Microsoft Kinect Sensor; Apple iPad; game engine 3DVIA Virtools, Microsoft Kinect SDK; Microsoft Speech API
[52]	Investigated how a VR study context influences participants’ user experience responses to an interactive system with an UX evaluation of the same in-vehicle systems; System Usability Scale (SUS); User Experience Questionnaire (UEQ); Sense of Presence Inventory (ITC); Volvo S90, semi-assisted driving system and a parking camera; 9 inch touch-based infotainment system; 12.3 inch digital driver information display, HTC Vive; system for semi-autonomous driving; screen; Dell Precision 5000; LeapMotion; Unity3D

lists the methods and metrics specified in each of the articles reviewed.

Some studies had methodological limitations that may have influenced their findings, specifically the relationship between the usability testing results in physical prototypes and virtual prototypes. In certain cases, participants had prior interactions with the systems being assessed, which influenced the evaluation of the system’s ease of use in the virtual environment [45].

Other methodological limitations that we observed relate to the quantity and profile of the participants in usability testing.

The authors of [60] caution against involving people who have a link to the product under development, such as corporate executives. Employees’ involvement in usability testing may be unconsciously impacted due to their relationship to the business, affecting the dependability of the data acquired. As a consequence, the authors recommend that the external target group, or future customer, participate in usability testing at some point throughout the development process. However, regarding the respondents’ profiles, the usability testing included personnel from Volvo [52], BMW [43], and Audi [46], as well as university students [48].

Additionally, the quantity and profile of participants in usability testing may affect the accuracy of collecting specific metrics, thereby reducing the repeatability of the usability testing findings when repeated with different participants. For the SUS metric, for example, thirty-five individuals are recommended for experiments in order to obtain satisfactory results [61].

Table 4. Number of participants.

Ref.	Number of Participants
[20,48,53,55]	11 to 20
[43,46]	21 to 30
[52]	31 to 40

shows the number of participants in the examined studies.

3.2.2. Hardware

Because virtual reality is a complex technology that combines interactive media, sensors, displays, human–machine interactions, simulations, computer graphics, and artificial intelligence technologies to expand human perception, virtual reality systems frequently require the use of multiple devices [56]. Depending on the application, virtual reality hardware may range from a basic computer to specific display devices, motion capture equipment, and interactive gadgets, such as wearable devices, cameras, head-mounted displays (HMD), and so on.

Table 5. Hardware, equipment, gadgets, and apparatuses utilized in the examined research.

Ref.	Hardware	Category
[15,16,19,20,45,48,51,52,53,55]	HTC Vive	HMD
[46]	Oculus Rift
[43,50]	Microsoft Kinect	Motion Capture
[54]	Vicon
[51]	Motion Capture (model unspecified)
[54]	New Holland T5.120 tractor model cabin	Cockpit
[46]	Driver’s seat, steering wheel, three pedals and the center console of an Audi A4
[48]	1:1 physical model 2018 Mercedes-Benz E200L
[48]	Steering wheel kit Logitech G29
[54]	Zephyr BioHarness to record human physiological data	Sensors
[54]	IX500 pressure mat (to collect seat pressure data (used only for cabin design)
[51]	ECG/GSR (for Real-time analysis of user’s emotion)
[41]	Three-walled room (two walls and a floor)	CAVE-like systems/immersive rooms
[44]	four-wall room
[40]	Power wall projection setup
[41]	Nintendo Wii Remote	Interaction devices
[20]	Xbox controller
[44]	Flystick
[46,52]	Optical tracking Leap Motion
[54]	Tobii Pro	Glasses
[40]	Active shutter glasses (model unspecified)
[41]	Stereo glasses (model unspecified)

lists the hardware, gadgets, and apparatuses referenced in the evaluated publications.

We found a vast variety of technologies, both in terms of the devices and the models employed in each category. Some studies did not specify which model was used. We discovered that the most frequently utilized hardware may be classified into seven basic categories: (1) head-mounted displays (HMD), (2) motion capture systems, (3) cockpits, (4) sensors, (5) automatic virtual environments (CAVEs), (6) interaction devices, and (7) glasses. These categories are not mutually exclusive, and the arrangement is often built by combining various technologies.

Both visualization and tracking technologies are required in a virtual-reality environment. Head-mounted displays and projection-based systems are the most frequently utilized virtual reality visualization technologies in the industrial sector. Head-mounted displays are devices that are affixed to the head of a virtual reality user and generally feature one or two screens as the image source, as well as a collimating lens between the eyes and the display [62]. Projection virtual-reality systems, on the other hand, include single or multiple projector-based powerwalls, as well as surrounding, walk-in installations based on numerous projection screens (e.g., CAVEs).

The authors of [56] noted that the usage of CAVEs and head-mounted displays is unusual in industry, which they attribute to high costs. However, since head-mounted displays were the most widely used devices in the research we considered, we discovered a distinct situation. One likely reason is that, until recently, the use of head-mounted displays was restricted due to their high cost and technological restrictions.

However, there was a technical breakthrough in 2016, with the first public release of technologically mature virtual-reality head-mounted displays, such as the HTC Vive and the Oculus Rift [27,53]; since then, there has been an increase in worldwide research on virtual [28]. Not only academics, but also practitioners agree that the equipment launched in 2016 was a “very big breakthrough” for virtual reality applications [29,30]. Therefore, although the cost of these devices remains relatively expensive and technical limits exist, the price of virtual reality equipment has reduced year on year, technological constraints have decreased and new features have been developed, resulting in the increased use of head-mounted displays.

The HTC VIVE was the most commonly reported model in the studies that employed head-mounted displays. Ten of the studies examined used HTC VIVE, well ahead of the second most frequently mentioned model, the Oculus Rift.

The authors of [15,16,45] stated that they chose HTC Vive because its tracking sensors work reliably and its controllers allow multimodal hand inputs. Furthermore, HTC Vive supports development in the Unity3D game engine, which has become the standard for the development of virtual reality.

The HTC VIVE is a virtual reality headset that consists of a head-mounted display, two wireless handheld controllers, and two lighthouse base stations that emit pulsed infrared lasers. It allows the user to move about and interact with a 3D world using motion-tracked handheld controllers. The VIVE system has two 1080 × 1200 resolution displays, one for each eye. The headset and controls also have 70 infrared sensors, a gyroscope, and an accelerometer. These sensors, together with the two lighthouses, track the operator’s motions with millimetric precision. The operating system is SteamVR, which runs on Microsoft Windows. A USB connection attaches the VIVE system to the computer [63]. Figure 6 is an example of a study in which HTC VIVE head-mounted displays were employed.

Three of the studies examined employed immersive rooms and CAVE-like systems. It is worth noting that all three were published in 2017, shortly after the previously mentioned launch of accessible head-mounted display releases in 2016. Prior to the popularity of head-mounted displays, CAVE equipment was the most frequently utilized technology [18]. However, since CAVEs are expensive, have poor immersion, and are not especially portable, they have been increasingly replaced with head-mounted displays. These claims are supported by the authors of [64], who found that in 2022, 28% of industrial presentations used a virtual reality system, whereas just 4% used a CAVE setting.

Motion capture devices are also frequently mentioned. Because sensing in single postures while handling a virtual object is currently inadequate, motion capture devices are utilized to supplement the detection of virtual world players and to reduce occlusion problems. Microsoft Kinect and Vicon models were utilized, with Kinect being used in two different studies by the same research group.

Cockpits are regularly cited devices in research on automotive applications. During testing, physical structures equipped with steering wheels, benches, and pedals were mixed with virtual reality elements. A cockpit with a head-mounted display device and sensors to detect hand motions was a common combination. In some experiments, vehicles with their original multimedia systems were employed in comparative evaluations of usability testing in real environments. Participants traveled along simulated routes while engaging with the vehicle’s interior controls. The use of physical device, such as seats, steering wheels, pedals, and gear shifters, allows the user to interact with the virtual environment in a similar way to haptic devices, which are capable of increasing the participant’s sense of immersion during the test and, in some ways, supporting the tactile feedback of the virtual-reality experience.

We observed the use of other hardware with specialized functions, such as Nintendo Wii remotes, Xbox controllers, and Leap Motion to improve interactions.

A wide range of other gadgets, including 3D printed rigid bodies with markers [54], GoPro cameras [54], and 3D laser scanners [55], were only mentioned in one study. Table A1 lists the hardware specified in each of the articles reviewed.

3.2.3. Software

Similarly to what we identified on hardware devices, we discovered the use of a broad variety of editors, programs, engines, and frameworks. The software adopted, like the hardware, is not mutually exclusive; rather, we found that there is typically a combination of various complementary solutions. The software referenced in the articles examined is shown in

Table 6. The software and engines employed in the research under consideration.

Ref.	Software
[15,16,19,45,46,48,51,52,53,55]	Unity3D
[43,50]	3DVIA Virtools
[43,50]	Microsoft Kinect SDK
[15,16,45,54]	CATIA; Siemens JACK
[15,16,45]	Autodesk 3DsMax
[43,50]	Microsoft Speech API

.

Game engines are the most often used virtual reality software. The authors of [54] also found that gaming engines have been used in virtual manufacturing in recent years. These game engines may provide useful application programming interfaces for rendering, interaction, and physics, allowing for the creation of high-quality and high-fidelity virtual environments, as well as extensive online documentation [56,65].

Unity 3D, a game production platform that incorporates a game engine, was mentioned by ten studies and was the most widely used product. The authors of [15,16,45] declared that they chose Unity3D because it allows the integration of scripted behavior and offers a simple import workflow for 3D data.

Only one study used the game engine 3DVIA Virtools rather than Unity 3D. The authors explained that 3DVIA Virtools was chosen to develop the virtual-reality application because it can be built on top of a sophisticated BMW shader library and production data import workflow, and it has a flexible graphical scripting language that allows the programming and refining of interaction logic at run-time.

In terms of CAD and modeling tools, three studies produced by the same research group used CATIA [15,16,45], and Siemens JACK was used in the experiment in [54] for product digitization.

A wide variety of software, such as plugins, editors, and libraries, among many others, were cited by only one study. Table A1 lists the software specified in each of the articles reviewed.

Q3: What are the benefits and challenges of using virtual reality for usability testing and design review in industry?

The examined articles highlight different perspectives in which virtual reality might overcome the constraints of conventional design reviews and usability testing processes.

According to the findings, virtual reality has two key advantages: benefits over conventional CAD models on screens and benefits over physical prototypes. In terms of advantages over CAD on screens, the advantages highlighted include the visualization of data from different angles and on a true scale, increased team collaboration and feelings of engagement, and more intuitive and natural interactions for non-CAD specialists. The benefits over physical prototypes include cost and time savings on redesign and increased safety. These advantages are discussed in depth below.

3.2.4. Visualizing Products from Different Viewpoints and on a True Scale Stimulated Novel Insights

The most frequently cited benefit of virtual-reality reviews of CAD on a screen is enhanced 3D visualization and manipulation. The conventional review process is often carried out on a computer with the use of CAD tools, on a flat screen. The visualization of CAD on a screen may not always satisfy all of the criteria for functional and ergonomic validations of complicated 3D models [16]. Virtual reality allows more novel modes of visualization and interaction to enhance engineering design reviews in this situation [16].

Several studies reported that the opportunity to observe a product from different angles and with more detail generated innovative discoveries [55]. The authors of [41] observed that visualizing planes in a virtual reality environment gave participants a better understanding of the spatial relationships between product components, as well as the interaction space around the assembly line, allowing the design team to understand the operating clearances in real size. The authors [42] found that relevant information about transport routes or kinematic properties, which is either not modeled in the CAD data or is lost during conversion processes, played an important role in reviews and provided an observation of machine mechanics (e.g., motor-driven mechanisms). The authors of [41] observed that the possibility of viewing and interacting with the geometry of the product (a pump) and the surroundings (an assembly line) on a true scale benefited the team in understanding important viewability problems during a subassembly engagement. The team received kinesthetic and ergonomic information on operator movement via natural engagement with the Wii remote [41].

On average, we found a consensus that this new visualization facilitated overall mistake detection. The authors of [16] found that, when compared to a standard CAD software approach on a flat computer screen, participants are more likely to spot faults in a 3D model inside an immersive virtual reality environment. In [41], the team uncovered design flaws and possible solutions that could not be detected or verified using conventional computer tools. Virtual-reality-enabled design review enables users to detect significantly more flaws in a 3D model than a CAD-software-based approach on a PC screen [16].

The benefits of combining fully digital CAD models with physical components of hybrid prototypes, such as the cockpit gear utilized in the automotive industry, are also emphasized. With the addition of the physical model, designers can evaluate their designs both visually and tactilely. This adds a further physical dimension, allowing designers to not only “see” their designs, but also “touch” them, providing designers with simulated interaction solutions in the early stages of design [48]. In [19], it is argued that virtual reality’s benefits contribute to the Industry 4.0 and cyber–physical system issue of linking the physical and digital worlds.

3.2.5. Increased Team Collaboration and Feeling of Engagement

The advantages of improved collaboration and engagement were also cited frequently in the research reviewed.

Industrial design reviews and usability testing are complex processes involving a variety of stakeholders, including designers, engineers, and end users. Computer-aided design is utilized as a communication tool in a conventional review process to transmit design ideas and enable a better common understanding among diverging needs and perspectives [15]. Because virtual reality decreases the possibility of some groups being excluded from the review process, it has the potential to foster collaboration among stakeholders [16].

According to some studies, the high focus that virtual-reality-based reviews provide [52] increases a feeling of team engagement. In [41], it is asserted that in the conventional process, design teams cluster around conference tables with laptops, mobile phones, and paper notes while, at best, one person manipulates the design on a giant 2D screen. Maintaining team engagement and attention becomes more challenging when battling with technological device distractions. The virtual environment allowed the design team to move away from the traditional conference room and into a creative area with fewer distractions. During design discussions in the immersive environment, the team observed increased engagement. Thus, the immersive virtual reality environment enhanced team engagement, which resulted in better discussions and fuller participation from team members in decision-making [41].

Figure 7 shows a design review on a screen and a VR-based process.

3.2.6. More Intuitive and Natural Interactions for Non-CAD Specialists

According to many studies, virtual-reality-based design evaluations and usability tests are more friendly and intuitive. While CAD software does not allow the intuitive analysis and manipulation of 3D models by users without a CAD or computer science background [15], interaction in virtual reality environments is generally simple and intuitive [16,19], and 3D engineering data can be visualized in virtual reality. Because of the high level of immersion provided by virtual-reality head-mounted displays, conducting design reviews and interacting with 3D models is regarded as more intuitive and “natural” for non-CAD specialists [16].

The authors of [16] observed that the intuitiveness of interactions in a virtual-reality system enabled a considerably faster entry into the design review.

3.2.7. Cost and Time Savings for Redesign

When compared to the conventional review process of physical prototypes, the most frequently noted advantages of virtual reality are its cost and time savings. According to the research analyzed, industries may employ virtual prototypes to save money [43,46,48], minimize redesign time, and expedite time-to-market.

Prototyping is an essential step in the product development process. However, after building a product model, testing its design and functionality requires time and money. In this context, a virtual prototyping system based on virtual reality technology has the potential to overcome these shortcomings [48].

The authors of [54] also describe the benefits of lowering the time for product design reviews and the amount of design and engineering design changes, as well as reducing time-to-market and optimizing costs.

Immersion in virtual reality may result in cost savings as well as better and/or quicker design review processes. Because physical prototypes and mock-ups may be replaced by their virtual counterparts, virtual reality improves design verifications and the review process, which might contribute to cost savings for manufacturers [13,19].

In [46], it is claimed that the principle of “simultaneous engineering”, in which elements are designed and tested virtually concurrently with vehicle development, is achievable with, and potentially strengthened by, the use of virtual reality [46]. The authors use the automotive industry as an example to demonstrate that a type of control could be created early and parallel to the development of a new car model and tested within a virtual car model worldwide. Therefore, the cost of redesigning a model can be reduced if the type of control can be changed in the course of development. This capability enables creative departments to produce and test novel concepts without disrupting the conventional flow of product development.

Products in development may benefit from rapid design updates, which save significant time and costs. Virtual reality allows therefore a novel, concrete, and resource-saving design evaluation method with significant application potential, since designers only need to produce the models that need to be tested, which greatly reduces time and costs [48].

Instant feedback and design modification could further improve product quality by making it possible to detect issues at an earlier stage [55]. In [54], the possibility of anticipating potential problems and design changes and having precise feedback about human–machine interactions thanks to the virtual simulation, before products are produced, is discussed.

One study [55] focused on the cost savings associated with reduced travel frequencies, since virtual reality allows the replacement of physical review meetings and interaction for immersive technical discussions.

A frequent worry when using virtual-reality-based prototypes is whether the correlation with conventional review and testing methods is maintained, as well as the degree of accuracy for usability testing.

The reviewed studies indicate that virtual-reality prototypes do not compromise correlation, i.e., virtual-reality-based usability testing applies to the same degree and may provide results equal to conventional testing with physical prototypes.

Several studies [48,51,66] reported that virtual reality usability testing has a significant connection with physical testing in terms of the metrics collected. The variance of quantitative data, such as operation errors and the time spent completing tasks, was statistically analyzed, and their correlations were validated and reported with sufficient correlational data between physical and virtual prototype testing outcomes. It was found [52] that there were no significant differences in UX questionnaire data between virtual reality and the physical prototypes in the field, but that there were correlations between rated presence in the virtual reality system and UX ratings, particularly for reported stimulation. The authors of [48] found that the data and experience from a mixed-reality prototype vehicle were equivalent to those from a fully physical prototype vehicle. It was observed [46] that a physical prototype automobile and a mixed-reality prototype have comparable rated usability and that there is no significant difference between the metrics they collect.

Although the authors of [43] do not provide comparable testing methodologies for virtual systems and real-world environments, they also conclude that virtual reality might reduce costs and shorten the time taken for the virtual testing of products in development, based on metrics gathered during supported testing and interviews with participants.

3.2.8. Increased Safety for Participants

For dynamic usability testing, such as driving a car, virtual testing may provide a substantial benefit, since participants are not subjected to any actual risks. Therefore, situations such as a vehicle collision while driving, or a car accident involving passengers, would not occur in the virtual environment [52,66,67].

Despite all of these benefits, virtual reality has certain drawbacks. According to the research, there are various ways in which virtual reality might impose additional constraints on conventional design reviews and usability testing processes, limiting its widespread adoption.

In fact, the technology still needs to evolve [16] and implementing virtual reality technology may be difficult. The challenges we identified are related to a lack of realism as a result of unnatural tactile and visual senses, latency and registration issues, communication difficulties between teams, and motion sickness and other unpleasant symptoms. In the sections that follow, we analyze these problems.

3.2.9. Lack of Realism as a Result of Unnatural Tactile and Visual Senses

The difficulties of non-natural interactions between individuals and virtual prototypes owing to visual and tactile constraints were addressed frequently in the research reviewed. The complicated interactions in virtual reality have proven to be major drawbacks in industrial settings [50].

The authors of [15] reported that haptic feedback and multimodal interactions are still problematic and concluded that there is a lack of visualization and interaction techniques that fully harness virtual reality’s potential.

The authors of [48] found that the movement of participants’ hands could not be well simulated, and that positioning offset occurred frequently, while [41] identified that interacting with the geometry using the Wii Remote game control was too awkward and unnatural to fully investigate participants’ assembly inquiries. The authors of [41] found that the collision detection experience was insufficiently robust to be helpful in their study.

Several users reported issues with virtual object feedback when compared to external device motions, such as steering-wheel twisting. Another issue mentioned was that the sense of reach and the dimensions of virtual items do not correspond to physical interaction features, such as buttons or flat surfaces that mimic multimedia screens [46,52].

Given the significance of natural human connection with a virtual interface, haptic devices such as gloves, suits, and others may strengthen the sensation of immersion. Including haptic technology to allow human interaction modeling of natural human senses and motion would result in a significant improvement in usability testing. In [48], for example, it is suggested that replacing the handle with data gloves could simulate hand movement.

As a result, the effectiveness of incorporating haptic devices to improve usability testing is connected to the purpose of the testing, the complexity of the interaction, and the maturity of the virtual item under consideration. Haptic devices should be avoided for usability testing on products that are still in the early stages of development and do not have accurate hand-and-finger interaction.

The lack of realism is caused not just by touch sense issues, but also by visual sense issues. In [46], it was observed that readability and representation in a virtual prototype system were troublesome, which the authors attributed to the head-mounted display’s low display resolution. Some participants in the tests conducted in [46] complained that they could not view the virtual environment clearly. The study discovered that the image in the virtual environment is not fine enough, and that it is limited by the hardware devices, requiring more dynamic movement behavior and improved graphics resolution from the virtual prototype.

Due to complex interaction systems or surroundings with a substantial amount of visual information, high graphical representativeness is essential. Otherwise, only systems with a limited number of interactions may effectively correlate with conventional usability testing [67].

A major issue is that employing virtual-reality technology without an auxiliary device, such as hand-tracking sensors, may influence user perceptions and, as a result, test findings. The adoption of a physical property, such as a flat wood table emulating the screen of a multimedia system, is one strategy for providing tactile feedback to a user engaged in a virtual environment [66].

Each participant’s perception of a product’s depth, reach, and dimensions is spontaneous and instinctual during usability testing with physical prototypes. However, in virtual reality, the user desires touch with certain physical devices in order to handle objects in the virtual world; thus, visual calibration and positioning algorithms are necessary to modify the environment and virtual objects in order to provide an accurate user experience.

Virtual prototypes pose a challenge to the industry when human interaction that goes beyond visual verification is required. When creating functional virtual prototypes that are designed to offer visual, tactile, and aural feedback, the difficulty is to produce a high-fidelity virtual prototype that has the same features as a physical prototype. Geometric component qualities, such as high-fidelity colors and textures, part structure animations, such as when a vehicle’s door opens or when a refrigerator door handle is pulled, or even functional touchscreen displays require powerful hardware to process data within the virtual reality to ensure a reliable, immersive experience for the immersed individual.

Besides unnatural tactile and visual senses, occlusion issues also need to be addressed. It is critical to assess the participant’s emotional reaction while designing the qualitative metrics of a usability test. The authors of [52] found that virtual surroundings diminish some sensations throughout an activity, such as user happiness or dissatisfaction. The authors discovered that while engaged in the virtual environment, the attention of some participants became predominately focused on the need to accomplish tasks. Due to the inability to watch the participants’ facial behavior, which the head-mounted display was partially hiding, the moderator did not fully observe the participants’ emotions. As a result, physical prototypes are advised for some tests in which participant behavior must be assessed by face observation [46].

3.2.10. Latency and Registration Issues

Another issue that is commonly cited is the time between head movements and viewing the scene.

The authors of [46] identified registration issues between virtual and physical elements of the environment, which they linked to hand and finger optical tracking controller calibration or tracking errors. The authors sought to improve tracking by instructing participants on how to adjust the head-mounted display before beginning the trial; however, the adjustment cannot be controlled from the outside by the experimenter. The resolution of the head-mounted display, according to the authors, might play a role in this scenario.

When interacting with mixed physical-virtual prototypes, several study participants mentioned a dimensional discrepancy between the virtual and physical aspects of the prototypes (for example, the cockpit’s physical wheel and the virtual air conditioning control).

The necessity of calibrating the position and visibility of virtual objects with physical prototypes was identified as a barrier throughout the research. The main difficulty is the amount of time it takes to calibrate the system, given that calibration modifications are dependent on the user’s participation.

3.2.11. Communication Difficulties between Teams

The authors of [52] identified that virtual-reality-based design reviews of complex CAD data often suffer from communication issues between virtual reality users and team members who observe the virtual reality scene from an outside perspective (e.g., a TV screen). As a consequence, spoken descriptions are often insufficient to express a specific detail about a machinery component.

The authors of [52] also discovered that in the virtual environment, participants voiced less affect, and [20] found that the social exclusion of virtual reality users sharing the same physical space as colleagues during a design review session has a negative impact on communication and cooperation among team members.

3.2.12. Motion Sickness and Unpleasant Symptoms

While immersed in virtual-reality environments, several people reported unpleasant symptoms, such as nausea or headache. The source of these symptoms, according to the participants, was the lack of shadows and reflections of objects, as well as a delay in the virtual environment related to body movements and gaze [43,52].

Virtual-reality-based usability testing needs careful consideration of the method carried out. Immersion in a virtual world may be the first such experience in some individuals’ lives, and the strangeness of wearing a head-mounted display and handheld controllers may impair their confidence in interacting with virtual items and interfaces. When a virtual-reality-based usability testing method is designed, it is crucial to evaluate the participants’ health condition.

4. Conclusions

Industry is under pressure to shorten the time taken for new products to enter the market. Our study found that virtual-reality technology is a powerful tool for enhancing the redesign process in industrial product development.

When compared to conventional usability testing and design reviews, virtual-reality technology improves the process by visualizing new angles that stimulate novel insights, increasing team engagement, providing more intuitive interactions for non-CAD specialists, saving redesign costs and time, and increasing participant safety.

Virtual-reality-based prototypes have to address technological challenges, such as a lack of realism owing to unnatural tactile and visual interactions, latency and registration issues, communication difficulties across teams, and unpleasant symptoms.

However, a significant technological breakthrough occurred only a few years ago, with the first public release of technologically mature virtual reality equipment. The devices remain relatively expensive and technical constraints exist, but the price of virtual reality equipment decreased year on year, technological constraints have been reduced, and new features have been developed, resulting in the increased development of VR applications.

There has since been a significant growth in global research on virtual reality. In terms of inventions, we observed a scenario in which patent applications have boomed since 2016, in a technology space that is rapidly evolving, offering opportunities to enter an area while it is still young. Thus, this exponential growth in patent applications for virtual reality, design review, and usability testing should continue for a few more years.

Previous forecasts estimated that the worldwide virtual reality market would expand from $6.30 billion in 2021 to $84.09 billion in 2028, but the COVID-19 outbreak encouraged the use of virtual reality even more. Therefore, this increase might be considerably larger.

As a result, an increasing number of companies are contemplating implementing virtual-reality-based design reviews and usability tests without fully comprehending the overall advantages and restrictions. Our findings on inventions, current application fields, methods, and hardware and software used—as well as the benefits and challenges of combining virtual reality with conventional design reviews on flat screens or physical prototypes in usability testing—may serve as a reference for decision-makers and researchers as they continue to develop novel solutions for the industrial product development process.