Perspectives on Assistive Systems for Manual Assembly Tasks in Industry

Abstract

Small lot sizes in modern manufacturing present new challenges for people doing manual assembly tasks. Assistive systems, including context-aware instruction systems and collaborative robots, can support people to manage the increased flexibility, while also reducing the number of errors. Although there has been much research in this area, these solutions are not yet widespread in companies. This paper aims to give a better understanding of the strengths and limitations of the different technologies with respect to their practical implementation in companies, both to give insight into which technologies can be used in practice and to suggest directions for future research. The paper gives an overview of the state of the art and then describes new technological solutions designed for companies to illustrate the current status and future needs. The information provided demonstrates that, although a lot of technologies are currently being investigated and discussed, many of them are not yet at a level that they can be implemented in practice.

1. Introduction

Modern manufacturing bases on the concept of small lot sizes, so that in theory almost every item produced may be different. This increased variety in products means significant changes for people doing manual assembly tasks and increases the complexity of assembly tasks. New mechanisms are needed to support the required flexibility [1], such as collaborative robots and instructive assistance systems. Collaborative robots can interact with people directly in the same workspace to assist physically in performing tasks [2]. Instructive assistance systems are systems that provide instructions to assist people as they do a task, e.g., using tablets or augmented reality to remind them of what to do next. Since the steps required vary, instructive systems can generally be considered assistive. Assistive systems such as these can potentially provide a variety of different opportunities for manufacturing, such as training employees with less experience, quality assurance, reducing the memory load, and integrating people with impairments into the workplace [1].

This paper is based on an industrial workshop. As part of the workshop, some solutions developed from the point of view of their application in real industrial manufacturing environments were presented by people both from industry and universities. The goal was to give participants information about the options available, trade-offs between them and some preconditions for introducing these systems in companies.

In this paper, we briefly introduce collaborative robots and instructive assistive systems to produce small lot sizes with manual assembly, and describe our perspective on the current needs in this area based on projects done in collaboration with companies. As described by the Technology Readiness Levels (TRL) from NASA [3], there is a large difference between a technology that has been tested (TRL 4), to a prototype utilizing it that has been demonstrated in a relevant environment (TRL 6) and an actual proven system (TRL 9). This paper helps to understand the readiness of these technologies for practical implementation and also suggests future directions for research.

This version extends a conference paper [4] by presenting more information about the background and by summarizing our future perspectives on assistive systems for manual assembly tasks in industry. It presents our opinion based on the presentations made during the industrial workshop at the PETRA (PErvasive Technologies Related to Assistive Environments) Conference.

1.1. Collaborative Robots

As described by Korn et al. “robots have long left the cage of industrial settings: they work together with humans—collaboratively” [5] and thus can assist in performing tasks. While deploying collaborative robots provides flexibility that supports the increased variability of products, it also entails changes to the design of production lines for this interaction. When splitting the work, ideally robots can assist with physically demanding or monotonous tasks, whereas decisions (e.g., solving problems due to unpredictable errors) and small product changes or deviations in a product’s production workflow are handled by humans. Johannsmeier et al. proposed a framework for task allocation in collaborative industrial assembly processes to combine the human cognitive skills with the skills of collaborative robots [6]. For task planning with human–robot collaboration (HRC) tasks, the design of the workplace or the production line must also be considered. Tsarouchi et al. presented a HRC task planning method that also takes into account the design of the workplace [7]. Here, the humans and robots are modeled as active resources, while equipment (e.g., worktables) are represented as passive resources.

Collaborative robots may potentially offer more advantages than just assisting human workers in rapidly changing work processes caused by the increased variability of products. They may also help to reduce costs, for example, by removing the need for separate robot assembly cells and combining the workspaces for robots and humans and also by reducing production time for semi-automated processes, as shown by the authors in [7,8]. Stöhr et al. instead focused on an application with elderly people and those with disabilities [9]. The authors introduced a proof-of-concept prototype that employs people with disabilities in a factory, including special emphasis to ergonomics, personalization and adaptability with the help of a flexible and suitable combination of various assistive systems.

From a technical viewpoint, there are challenges in terms of holistic scheduling of human and/or robot tasks and seamless interaction between workers and robots [10]. For example, most approaches focus on non-generic modeling of specific machine tasks (e.g., PLCopen). As a result, they may not be able to handle the upcoming variability in assembly processes expressed by different workpieces and different assembly orders or new modern assistive systems for workers (e.g., collaborative robots). While there are existing approaches that consider the variability of products [11,12], these do not consider the human–machine interaction. Therefore, recent research focuses on implementing domain specific languages for combined modeling of human and robot tasks [13,14], for example by combining the methods called Methods Time Measurement (MTM) and Robot Time and Motion (RTM) [15,16].

From a human perspective, when introducing collaborative robots, there are different sources of insecurity that can affect the acceptance of these systems by workers, which include [17]:

• complexity of the tool, i.e., having to deal with malfunctions;
• fear of losing their job;
• fear of damaging the robot, also associated with their cost; and
• fear of injury caused by the robot.

Due to the nature of these aspects, they may be able to be addressed in part by educating users, for example by teaching about how to use them to reduce their perceived complexity, explaining how they function to reduce concerns about injury.

1.2. Instructive Assistance Systems

Instructive systems explain individual steps during the assembly process, and so may be used to assist people manufacturing small lot sizes for which the steps often vary, either with or without HRC. As mentioned in the Introduction, assistive systems generally can help support high-quality results and can also be used with people with less experience. There are many software-based possibilities, including solutions based on tablets, projections, Mixed Reality (MR) and Virtual Reality (VR) [18]. In comparison to handbooks, software-based solutions allow for hands-free use, while providing the content and contextual tools people need to complete their jobs. Current trends and approaches related to assistance systems in industrial manufacturing contexts are summed up in [19]. In the following paragraphs, we describe the newer MR and VR assistive systems briefly.

MR interfaces, such as smart glasses, can deliver key information for assembly to workers in a fast, reliable and interactive way. MR merges real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. In comparison to well-established Augmented Reality (AR) applications, MR not only overlays but anchors virtual objects to the real world. The instructions are shown directly at the item being assembled. Following this idea, workers wear a Head-Mounted Display (HMD) that allows them to view 3D models and animated step-by-step instructions directly in the user’s field-of-view, which can help to carry out assembly (and also maintenance) tasks. In comparison to well-established handbooks, this solution guarantees completely hands-free usage, while providing the content and contextual tools they need to complete their jobs. A concise overview of the technical features, characteristics and broad range of applications of MR-based assembly systems was summarized by Wang et al. [20].

Whereas Mixed and Augmented Reality are still caught in the “Trough of Disillusionment”, VR has already reached the “Plateau of Productivity” in 2016 [21]. VR technologies promise great benefit for industrial applications, covering simulations of complex assembly and maintenance tasks, prototyping and design reviews. VR is used to train operators of various kinds of equipment, where initial training in a virtual environment can reduce the expense, risks and problems of monitoring and control associated with training in a real-life situation [22].

2. Future Perspectives

With the advances in hardware and software capabilities, more and more opportunities for (industrial) assistive systems arise. As the innovation picks up pace, companies are facing new challenges and have to overcome obstacles in order to make applications truly usable [1]. Based on the literature and information gathered through collaborative projects with companies, we see the need for new technological options to support companies. This includes both cutting-edge technological solutions and solutions in the areas of instructive assistive systems and collaborative robots. In the following, we describe the directions we are taking, and which were presented at the industrial workshop.

2.1. Part 1: Human–Robot Collaboration (HRC)

In the field of HRC, with regard to upcoming trends in modern manufacturing, which are based on the concept of small lot sizes and increased variety in products, we see need in the following areas:

• Educating workers for industrial HRC;
• New, more intuitive methods of controlling robots, so workers do not have to think about coordinates or the required movements of certain axes of the robot; and
• New modeling methods for assembly workflows that allow flexibility regarding whether the human or the robot performs the next task, to allow for example a seamless step-by-step integration of collaborative robots into a production line.

2.1.1. Educating Workers for Robot Collaboration

Educating users may help overcome the sources of insecurity with respect to robots mentioned previously, and also help to gain the full benefits of these robots and reduce the chance of injuries (for which the companies are liable).

Experience at BRP-Rotax has shown success with training courses with the following contents:

• Familiarization and basic operation, e.g., turning robots off, recovering from errors;
• Learning how to teach the robot and about specific tool types; and
• Safety aspects, e.g., testing that the robot actually stops.

This training encompasses both an introduction for participants without any previous knowledge and continuing education. The contents of the training courses are being tailored to the different job profiles, as people who will work with the robots need different skills than those who maintain them, and need to know, for example how to integrate new sensors. Thus, additional modules may be needed for some user groups, e.g., covering programming skills or comprehensive information safety standards. As such, training is an iterative process that takes place over an extended period of time.

Initially, people may find it challenging getting a robot to move to the desired point in three-dimensional space. For many such aspects, smaller, less expensive robots can be used in training session such as igus robolink Q or UR10.

2.1.2. New Ways of Controlling Robots

One of the most important aspects in the field of collaborative robotics is the human–machine interface. Control aspects are particularly important, as even with social robots having behavioral control affects both the perceived ease of use and the acceptance of novice users [23]. As mentioned above, our industrial partner BRP-Rotax mentioned that some of their workers find it challenging to move an industrial robot to a desired point in three-dimensional space. Thus, we are investigating methods to make it easier to control robots. Recent research focuses on making robots smarter with the help of smartphones [24]. In addition, more intuitive ways of controlling robots, for example by gestures tracked by various sensor inputs (e.g., cameras, microphones, acceleration or gyroscope sensors, smartphones, brain–computer interfaces, etc.), has been examined [25,26,27,28].

However, in terms of popularity, acceptance and usability, we decided to use and evaluate conventional smartphones to control an industrial or collaborative robot. We developed a gesture control of a robot with the help of a smartphone on the basis of the integrated acceleration and gyroscope sensors. This is based on work done with the collaborative robot KUKA LBR iiwa. As a result, the control of a robot and the interaction between robots and human beings can be simplified. Consequently, the people do not have to think in three-dimensional coordinates or worry about which axis of the robot needs be moved to reach the desired position. They can simply control an industrial robot by performing predefined gestures using their own smartphone.

2.1.3. New Modeling Methods for Workflows

There are different ways of modeling and optimizing specific workflows, but none of these methods consider HRC. Therefore, a new modeling approach is needed, which includes the interaction between a human and a robot. Additionally, it must be possible to model work steps which can be performed by a human being, a robot or together. Based on these needs, a new method called HRTM (Human Robot Time and Motion) has been developed, which combines and extends the existing methods MTM and RTM. The HRTM method describes every possible workflow with a series of basic motions. These basic movements are designed and abstracted to such an extent that either a human being or a robot is able to perform them [29].

As a result, a flexible design of the work processes is possible. Thus, it can be decided at runtime, whether the robot or the human worker should perform the next step. Many of our industrial partners would like to use collaborative robots, but are uncertain how such robots can be employed and integrated in the production lines in collaboration with humans. Furthermore, many employees do not yet have the necessary skills. Because of the flexibility of the HRTM method, it is possible to adjust the workflows so that either the robot is more or less involved, based on the human worker’s experience in the field of collaborative robotics. Through this, the employees could gradually become familiar with the new collaborative robots. In addition, companies can optimize their assembly workflows iteratively for HRC as they gain experience with them.

2.2. Part 2: Instructive Assistance Systems

As described above, in the area of instructive systems for workers, there are many options available. We see both MR and VR as core technologies for future assistive systems for manual assembly tasks. However, we also see the need for more understanding of the opportunities, limitations and trade-offs of different solutions to support those developing them. In the following, we present:

• Important trends in mixed and virtual reality support (MR/VR) for industry;
• Opportunities and limitations of the different MR-based assistance systems;
• Tracking approaches for MR and their applicability for industrial applications; and
• Considerations when selecting and designing instructive assistance systems for manual assembly tasks.

2.2.1. Trends in Mixed and Virtual Reality Support (MR/VR) for Assembly Tasks

MR and VR technologies are still in their early stages, but as the pace of innovations accelerates, systems allow for more novel modes of assistance to support complex assembly tasks. Based on the collaboration with our industry partners Bosch Rexroth and iFE Doors-Knorr-Bremse Group we see three trends that, from our point of view, will strongly influence the future of assistive systems for manual assembly tasks (see Figure 1):

• VR as tool for the assembly and disassembly verification;
• MR as tool to support step-by-step instructions and remote assistance for hands-free work; and
• Game engines as standard tools for MR and VR development.

VR, for example, is a valuable tool for the assembly and disassembly verification of complex 3D engineering models. To date, Bosch Rexroth conducts assembly verification studies on the PC screen supported by standard CAD tools (for example, Creo View). They criticize that ergonomic studies, such as the accessibility of certain components during the design of an assembly scenario, cannot be realistically recreated in a 3D scene on a 2D screen. They feel VR, on the other hand, renders all characteristics that are relevant to the particular context precisely and realistically in an immersive 3D environment. On the software side, the role of modern game engines (such as Unity3D or Unreal Engine) as de facto standard tools for MR and VR development will become even more important, especially for the industrial sector. Game engines also enable the integration of playfully motivating elements into the 3D environment. The potential of gamification mechanisms in the industrial domain was described by Korn et al. [30].

To learn more about promising MR-related use cases, we conducted expert interviews with service and maintenance staff from iFE Doors-Knorr-Bremse Group. These interviews provided insights into the employees’ needs and visions in regards to assistive systems and revealed that step-by-step instructions and remote support (also known as “Maintenance 4.0”) are promising MR use cases in industrial context. They see a big potential for MR-supported systems that guide the employee through the assembly or maintenance order by providing work instructions projected directly into the field of view. In addition, the use of smart glasses allows the hands to remain free for the assembly work. They also think a remote support feature would be useful for when problems occur: The camera image of the device could be transmitted to a geographically distant expert, who can expand the field of view with digital annotations. Commercial examples for MR-supported remote assistance are provided by Vuforia Chalk and Microsoft Remote Assist.

2.2.2. Evaluation of Different MR-Based Assistance Systems

As current research shows, assistance is manifold, reaching from well-established paper instructions, to in-situ projection, sensor-based systems and MR-based tutors with smart glass support. Recent work has also focused on identifying advantages and weaknesses of each approach in direct comparison. AR- and MR-supported assembly assistance is still struggling with issues concerning usability and acceptance. The authors of [18,31] compared AR-HMD instructions, tablet-based instructions, and baseline paper instructions to in-situ projected instructions using an abstract assembly task. The results indicate that assembling parts is significantly faster using in-situ projection and locating positions is significantly slower using HMDs. Furthermore, participants make fewer errors and have less perceived cognitive load using in-situ instructions compared to the HMD-supported approach.

Our experiences in projects with our industry partners coincide with the results just mentioned: First tests with Hololens from Microsoft already show limitations of the HMD’s hardware specifications. Due to its comparatively small field-of-view, holograms will appear no closer than a meter away from the user. The “sweet spot” of hologram viewing is an image that appears to be about two meters away. Thus, a user standing in front of an assembly table will have to move some steps back to properly see the augmentation. For this reason, a fixed tablet screen or an in-situ projection of assembly steps at the workplace is still the better solution until next-generation HMDs are available.

2.2.3. Tracking Approaches for MR and Their Applicability for Industrial Applications

Tracking is the core technology for MR-applications as it registers position and orientation of the camera with respect to the environment in real-time. A stable tracking algorithm creates an optical illusion that makes virtual objects appear as if they real ones, merging with the real environment. Tracking approaches for MR include image-based tracking, “Simultaneous Location And Mapping” (SLAM) methods and object- or shape-based solutions.

In the case of image-based tracking, the engine detects and tracks features that are naturally found in the image itself. Once the image target is detected, virtual data can be placed upon the image. Although this approach works, it requires some preparation of the scene and good lighting conditions. For an assembly work situation, this would mean affixing image markers on the working table or even on component and machine parts. In a rough working environment, these markers are likely to disappear or to break. For industrial applications, tracking has to work without scene preparation, and components have to be recognized in changing illuminations and surrounding.

SLAM tracking, by comparison, makes it possible to instantly track unknown (and unprepared) environments and to overlay digital interactive augmentations, where the primary mode of sensing is via a camera. The disadvantage here is that the user has to create the tracking map and the augmentations do not appear at a fixed, predefined location (such as a component part or machine).

Thus, for implementation in companies, we see the biggest potential for object- or shape-based tracking algorithms, where real-world objects recognized and tracked by their shape (and their edges) using 3D models or preexisting CAD models from manufacturing. This tackles typical AR problems, such as bad light conditions, dynamic scenes, or low-textured objects. By this means, step-by-step instructions can be overlaid on top of components in complex assembly task. Commercial examples for shape-based tracking solutions are for instance provided by Vuforia and VisionLib. We believe this approach will have a big impact on the future development of assistive systems for manual assembly tasks.

2.2.4. User-Centered Considerations for Instructive Assistance Systems

Although there are many possibilities, the best choice bases on the specific situation: the workers, the assembly task at hand and the manufacturing environment. Due to the extent of the technical challenges involved, the user-centered considerations may receive less attention, but are critical to the success of these systems. The following comments are based on practical experience through projects with companies, including BRP-Rotax, manufacturers of motors for recreational vehicles.

In addition to the technical aspects described above, there are also user-centered considerations relevant to the choice of device. HMDs may cause additional strain, both due to the weight and the need to switch between the display and the item being manufactured. Smartphones must be attached to the arm in some manner, and so may get in the way when manufacturing larger items. On the other hand, a tablet must be mounted in some way, which can be a particular challenge with automated guided vehicles (AGV) and ceiling cables. All of the systems bring with them a new problem of charging. Evaluating prototypes in the real environment for a few days can support the final decision. The choice of device must be made relatively early, as it also has an effect on the programming: whereas the head-mounted and smartphone versions are linked to a single person, if the tablet is connected to the AGV, the tablet is linked to a specific item being assembled.

The information required to do the task also varies based on the specific situation. Instructive systems are valuable with small lot sizes, as it can be difficult for people to remember the steps if there are many different items or this item has not be assembled for a while. People who are new to assembling may need all steps of the assembly to be displayed, but they do not have time to look at all instructions. Thus, audio systems may be more suitable for use at the workplace, while systems explaining all steps can be provided to help train people in advance. People with more experience may overlook the salient details if too much information is provided, and thus instructions can be limited, e.g. to recent errors. Regardless of the steps chosen for display, the system must be integrated with the production system and ideally switch instructions automatically. Since the cognitive load is often quite high in manufacturing jobs, it is important not to increase this through unclear displays or unnecessary distraction. The reliability is also a key factor, as people who come to depend on these systems may be worse off if they are suddenly not available.

For these reasons, using user-centered methods is particularly important when selecting or developing instructive systems: to get the needed information early, to get feedback before the development is complete and also to get long term experiences before rolling it out to a larger number of users (e.g., [32]).

3. Conclusions

This article presents current technologies and future perspectives to support manual assembly tasks for small lot sizes. Increasing complexity and individualization of products requires new mechanisms to assist workers and provide guidance during an increasingly complex assembly process. We describe current trends and needs based on our experiences with company partners in various industry sectors.

At present, assistive systems based on tablets and projection are at a level where they can be used in industrial settings, while MR and VR are still in an early stage of development. While some companies are experimenting with collaborative robots, there are still acceptance problems on the part of workers that impede the more widespread introduction of these solutions. The training and new methods described here may help overcome some of these problems.

We think that HRC and VR- and MR-supported assistive systems will significantly influence the future of manual assembly tasks for complex and diversified products. In addition, the role of modern game engines will become even more important, especially for the industrial sector.