Common Educational Teleoperation Platform for Robotics Utilizing Digital Twins

Abstract

The erratic modern world introduces challenges to all sectors of societies and potentially introduces additional inequality. One possibility to decrease the educational inequality is to provide remote access to facilities that enable learning and training. A similar approach of remote resource usage can be utilized in resource-poor situations where the required equipment is available at other premises. The concept of Industry 5.0 (i5.0) focuses on a human-centric approach, enabling technologies to concentrate on human–machine interaction and emphasizing the importance of societal values. This paper introduces a novel robotics teleoperation platform supported by the i5.0. The platform reduces inequality and allows usage and learning of robotics remotely independently of time and location. The platform is based on digital twins with bi-directional data transmission between the physical and digital counterparts. The proposed system allows teleoperation, remote programming, and near real-time monitoring of controlled robots, robot time scheduling, and social interaction between users. The system design and implementation are described in detail, followed by experimental results.

1. Introduction

Current challenges brought by ongoing conflicts, worldwide threats, and unequal economical conditions [1,2] are introducing major challenges to all sectors of societies. As a concrete example, pandemic restrictions have set limitations for student group sessions at university laboratories, forcing educational institutes to rethink practical arrangements of laboratory exercises [3,4,5]. Climate change requires an increased appreciation of environmental awareness and preferring green choices [1]. An example of a greener choice is the reduction of traveling to only situations truly necessary and favoring alternative approaches whenever possible. In addition to reduced traveling, changes in consumption habits are required; purchasing of new products should not be preferred, instead leasing or renting of existing equipment should be. Efficient utilization of existing equipment located in remote locations preserves the resources required for manufacturing new units. Equality can be promoted by offering people in developing countries the possibility of training remotely utilizing modern equipment [6,7]. As Bryndin [8] notes, the aim of Society 5.0 is to create equal possibilities for all individuals and to create an environment for realizing equal possibilities. A possibility to participate from distant locations would provide a solution to overcome pandemic restrictions, enable exercising without traveling, and increase the utilization rate of existing equipment, supporting a human-centric approach of utilizing i5.0 technologies. In this paper, a common teleoperation platform for robotics utilizing digital twins(DT) is presented as a solution for the distant utilization of robotics.

Moniruzzaman et al. [9] defined robotic teleoperation as a robot that is controlled or managed by a human operator over a data connection. Bi-directional communication between the robot and human operator allows the human operator to provide orders and suggestions to the tasks while the robot executes the tasks based on the input from the human operator. In the scope of this proposal, bi-directional communication enables a user to control robots at the university laboratory from any location, therefore saving time and environmental resources consumed on traveling. Teleoperation platform services are also available for students abroad and studying at other universities. There are previous studies on using teleoperations in universities. Marin et al. [10] proposed a telerobotic system, allowing students and scientists to control robots remotely. In addition to this work, the solution presented here provides time resource management to enable the scheduling of teleoperated equipment and Cybersecurity considerations. A trivial calendar application is provided as a front-end for time resource management, allowing students to reserve time resources for a specific robot to exercise robotic skills. In this proposal, the scope of resource management is limited to students reserving time slots to study robotics. The scope can be broadened to include equipment sharing and rental between companies and educational institutes.

Sharing time resources of available equipment requires user authentication and authorization to create a link between time resources, equipment, and user. The platform presented includes methods for user registration, authentication, and authorization. Daily tasks for user registration, strong password creation, and password recovery have been automated to save administrator time.

Social communication is an important feature for students participating in education from distant locations [11,12]. Communication is needed by students to request help for the usage of the platform and to exchange experiences with fellow students, lecturers, and IT support. On the platform presented, social communication is offered in the form of a live chat, a discussion forum, and an online video meeting. Digital twin (DT) was chosen as a key approach for implementing teleoperation. DT is a digital presentation of a physical world object, serving as a bridge between the physical and cyber world [13]. DTs enable monitoring and operation of physical equipment, production lines, factories, and smart cities independent of location. In this context, DTs enable students to perform practical exercises by teleoperating robots physically located in a university laboratory. The platform includes DTs of mobile, industrial, collaborative, and scara robots for teleoperation. Bi-directional open communication standards are utilized to link digital and physical twins. The digital presentation features a dashboard for controlling and monitoring physical twins. To provide a near real-time view for the teleoperation, a video stream of a physical environment is embedded in the digital presentation.

The platform proposed in this paper is accessible and cybersecure (CS). CS has been considered a key enabler for a sustainable platform performing data transfer and providing services over the internet. CS is also a building block of physical safety when Cyber-Physical Systems (CPS) such as teleoperated robots are considered [14,15]. Weak CS can enable an attacker to gain control of physical equipment and cause physical damage to equipment and the surrounding environment [14]. CS of the proposed platform is implemented by the following methods: secure authentication, authorization, firewall rules, and data encryption. The best efforts by the authors have been made to protect data connections and services of the proposed platform against cyber attacks. Addressing CS, however, is continuous work, requiring periodic reviewing and actions such as vulnerability scans and updates of vulnerable services. Previous studies on utilizing teleoperation for robotic laboratory exercises lack consideration of CS. Instead of writing a single paragraph about CS, in this paper, CS aspects are considered in each of the following chapters.

While many approaches of educational robotics platforms have been reported previously, implementations presented have been fragmented with restrictions or absence of at least one of the following features: social communication, resource sharing, content management, teleoperation, or support for mobile devices. The proposed approach contains the aforementioned services and is based on open-source software. Figure 1 presents the platform requirements and services. Research questions to answer are: Is teleoperation platform utilizing DT’s suitable for robotics education and training? Is common teleoperation platform implemented with open-source software cybersecure?

The contributions of this work are as follows:

• Implementation of robot teleoperation platform utilizing DT’s
• Method for time-resource management of teleoperated equipment
• Cybersecurity road-map for teleoperation platform
• Teleoperation platform based on open-source software

The remainder of the paper is organized as follows: Section 2 provides methods and approaches, including functional and non-functional requirement specifications of the platform. Section 3 presents a literature review describing existing solutions and previous research on the topic. In Section 4, implementation and validation of platform is presented and further discussed in Section 5. Section 6 concludes this paper.

2. Methods and Approach

Methods utilized during the development of the proposed platform are presented here. Requirement specification and requirement validation presented in Section 2.1 is considered a fundamental guideline during the implementation phase described in Section 4. The development phase begins with platform architecture specification and validation. Development is followed by implementation, including iterative programming, testing, and functionality validation tasks. To validate the functionality of teleoperation and digital twin data transfer, one high-level task was defined for each robot. Tasks were chosen based on the capabilities of the robot type in a question and are explained in Section 4.6. After implementation conforms to defined requirement specification, the platform was considered ready for evaluation in real applications. Figure 2 presents the methodology of this work to plan, develop, implement, and validate the proposed platform. Methodology presentation is based on work of Garcia et al. [16].

2.1. Requirement Specification

Requirement specification for the platform was defined based on needs for training basic skills in robotics and on requirements found during a related research study described in Section 3. Group defining requirements specification consisted of authors of this paper having pedagogical, software engineering, ICT, and robotic knowledge. Requirements were divided into two main groups: Functional (F) and Non-Functional (N). Functional requirements describe requirements directly related to the functionality of the platform such as teleoperation and monitoring of robots. Functional requirements were further divided into Must and Could, depending on the urgency to implement specific requirements on the platform. Some of the requirements listed are considered future work and are beyond scope of this publication, and the level for those requirements is defined as Could. Those requirements are considered in more detail in Section 5.

Non-Functional requirements define quality requirements, such as how many users should be supported or the availability of the platform. Non-Functional requirements were divided into sub-categories of Usability, Security, and Performance depending on the nature of the requirement. Requirements specified in

Table 1. Functional requirements.

ID	Name	Description	Level
F1	Digital twins	The system can be used as a digital twin allowing bi-directional data transmission from simulation models to physical robots and vice versa.	Must
F2	Teleoperation and programming	A registered user can teleoperate and program available robots physically located in laboratories. Robot movements can be controlled directly through an user interface or using robot programs.	Must
F3	Monitor	The laboratory can be remotely monitored through the system. This is implemented using cameras that are streaming to the system.	Must
F4	Registration	A person can register to the system through a dedicated web-page using a valid email address. Person can create a password after receiving confirmation email.	Must
F5	Role definition	Users can have different roles: user, publisher, and admin. Roles can be modified through the system by admins.	Must
F6	Scheduling	Users can make reservations on available robots for teleoperation and programming through the booking calendar, including time-slot management.	Must
F7	Discussion	Users can discuss or request help about different topics on discussion forum or chat. Admins can delete any thread.	Could
F8	Download/upload	Digital material can be stored on, accessed in, and downloaded from the system by users. Material can be documents such as images, videos, audio, and text documents.	Could
F9	Examination	Exams at different scales can be performed in the system. Different scales mean that a student can execute the course only partially. Evaluation pass/fail.	Could
F10	Certificate	The system provides a participation certificate based on the examination. The certificate includes the scope on the course taken. The certificate is sent by e-mail.	Could
F11	XR support	XR-techonology can be used to visualise simulations, teaching, and remote usage.	Could
F12	Virtual deployment	Robots can be virtually deployed using the system.	Could
F13	Group formation	User groups can be formed by all user types and groups can collaborate on same tasks.	Could
F14	Group lecturing	Lecturer can arrange meetings with groups. Meetings can be used for lecturing, mentoring, steering, etc.	Could
F15	Progress tracking	The system tracks users on which parts of course they have completed/studied.	Could

and

Table 2. Non-functional requirements.

ID	Description	Category
N1	The systems clearly focuses on robotics.	Usability
N2	Suitable for beginners and professionals, independent of the entry level. Programming experience or prior knowledge of robotics is not required.	Usability
N3	CS is taken into account. Remote users cannot access the system further than predefined robots. Additionally, robot functionality is restricted for safe operations.	Security
N4	Safety taken into consideration in remote usage.	Reliability
N5	Comparison of live and captured lectures. Feedback from students.	Usability
N6	Support for platform usage.	Usability
N7	Support for different languages (lectures, user interface).	Usability
N8	Laboratory exercises can be done independently from the previous progress. Any exercise can be chosen.	Usability
N9	20 simultaneous users.	Performance
N10	Possibility to transfer material from other platforms.	Usability
N11	Lectures can be online, offline, and live (F2F).	Usability
N12	Material will be organised in sections so that certain sections form ensembles.	Usability
N13	The system is available 24/7.	Reliability
N14	Email address used for registration is validated.	Usability
N15	Delay in stream when observing a physical robot should be less than 250 ms.	Performance
N16	Exercises can be in form of game. Either 3D featuring extended reality or 2D desktop mode.	Usability

are referred in Section 4 to indicate which requirement specific implementation task corresponds to.

3. Background and Related Research

The related research review aims to find software components required to build a common teleoperation platform for robotics utilizing digital twins. The chosen components meet the demands of the requirement specification in Section 2.1. An overview of recent research on DTs and teleoperation is performed with a focus on robotics and education. The modern data transmission protocols are reviewed, as DTs require a bi-directional data flow between the physical and digital twin. The latest research on real-time video transmission protocols for efficient teleoperation is reviewed to meet the requirement N15. In addition, authentication and authorization of an online system are essential for CS providing privacy and security to users requirements F4, F5, and N3.

3.1. Digital Twin

The concept of Digital Twin (DT) was presented in 2002 by Michael Grieves as a conceptual idea for Product Lifecycle Management [17]. Grieves presented virtual and real systems having a link throughout all phases of the product lifecycle from design to disposal. Since then the DT definition has been evolving, and misinterceptions of the “twin” metaphor have existed [18]. In this paper, definitions presented by Cimino and Kritzinger are considered as guideline definitions for DT. Cimino defines DT as a virtual copy of a physical system able to interact in a bi-directional way [19]. According to Kritzinger, bi-directional automated data flow distinguishes DT from the digital model and digital shadow [20]. DT has been utilized in the analysis, monitoring, maintenance, engineering, and testing [21]. DTs can be used to represent a single system such as a CNC machine, industrial robot, welding equipment, autonomous vehicle, or larger entities such as oil refinery or chemical plant [21]. DT enables a user to interact with low-level functions available on the physical part of the twin. A physical twin is an actual system put together to perform a certain function. Digital twin as a digital presentation of physical twin can present physical parameters such as temperatures, linear positions, and vibration frequencies in digital format. DT enables user interaction with the physical twin, providing operator teleoperation capabilities [22]. Teleoperation enables operators to control robots over large physical distances [22,23]. In the context of this paper, DTs enable control of physical robot cells, providing a method for teleoperation. Digital Twin consortium provides glossary [24] of DT-related terms. The following terms will be used in our paper later on and are explained here:

• Digital twin is a virtual presentation of real-world entities and processes, synchronized at specified frequency and fidelity
• Physical twin is a set of real-world entities and processes that correspond to a digital twin
• Digital twin platform is a set of integrated services, applications, and other digital twin subsystems that are designed to be used to implement digital twin systems
• Digital twin system is a system-of-systems that implements digital twin
• Cyber-Physical system is a system consisting of physical and digital systems integrated via networking.

DT concept is now in its third generation where AI and deep learning algorithms utilize online data [25,26]. First generation DTs were virtual presentations based on scripting languages and second generation of DTs introduced in 2012 were more simulation-oriented [25]. In addition to presenting individual systems, DTs are capable of combining individual DTs to present complete factories, supply chains, airports, and smart cities [21,25]. International Organization for Standardization (ISO) has defined standards for the DT framework in 2021 [27], and standardization is an indication of DT as a mature concept. Deep-learning can enable DTs to self-optimize and thus advance autonomously [25,26].

A cyber attack against a DT may enable an attacker to gain access to physical twin [14]. Therefore, unauthorized modification or destruction of data can lead to unexpected behavior of physical twin, leading to catastrophic damages [14,28]. According to Holmes et al. [28], the amount of CS attacks against DTs are increasing. Integrity and confidentiality of data transfer between physical and digital twins are identified as the main CS challenge for DTs [28], as CS attacks disrupting the integrity or confidentiality of data might endanger the safety of the physical twin environment.

Data Flow between Physical and Digital Twin

DT definition requires automated bi-directional data flow between physical and digital parts [20]. Synchronized data transfer from individual sensors to digital presentation and from digital presentation to individual actuators is a key enabler of DT concept [19]. This enables users to monitor and control physical twin by a user interface of a digital twin. Furthermore, the data flow of individual DTs can be combined to represent complete factories or complete supply chains [25]. Connecting individual DTs to present larger entities, a common platform is required. A common platform utilizing open connection standards enables flexible combining of DTs. Currently, OPC Unified Architecture (OPC UA) is a widely adopted open standard cross-platform solution for data exchange [29]. OPC UA client-server model has lately been accompanied by publisher–subscriber solutions developed for IoT such as Message Queue Telemetry Transport (MQTT).

MQTT, originally developed by Andy Standford-Clark and Arlen Nipper [30], is considered one of the most popular messaging protocols for IoT- and IIoT-devices [31]. MQTT has been widely adopted in logistics, automotive, manufacturing, and smart home applications. MQTT protocol has three participants: publisher, subscriber, and broker. A publisher is a device publishing certain topics such as temperature and humidity. A subscriber is a device subscribing to certain topics to receive information. Both are connected to a broker to deliver published topics to subscribers [31].

MQTT was not originally designed with CS in mind. Current MQTT-versions 3.1.1 and 5 offer authentication as a measure against CS attacks. To encrypt MQTT messaging Transport Layer Security (TLS) is needed and is supported by some MQTT-implementations [31]. Authentication and encryption provide strong protection against CS attacks. MQTT meets requirements F1, F2, and N3 of digital twins, teleoperation, and CS.

OPC Unified Architecture is a cross-platform open standard IEC62541 for data exchange. The standard is actively developed by independent committee OPC Foundation [32]. The history of OPC (OLE For Process Control) began when Microsoft introduced the BackOffice suite of server products in 1996 [33]. OPC UA standard defined in 2006 is widely recognized in industrial automation and has been chosen as Industry 4.0 reference standard [29,34]. In 2018, part 14 was added to define publisher/subscriber communication model in addition to client/server model [35]. MQTT, AMQP, and UADP messaging protocols are currently supported to offer publisher/subscriber communications to OPC UA.

OPC UA standard is secure-by-design, providing confidentiality and integrity by signing and encrypting messages [14]. Basic security provided by username and password authentication can be extended with encryption to enable a high level of CS.

3.2. Teleoperation

Teleoperation of robots has been studied for decades since the beginning of robotic systems in the 1950s [36]. MIT started to study teleoperation in the mid-1960s and the mid-1990s when prof. Sheridan reported progress in the field of teleoperation in several countries and application areas including space, undersea, mines, toxic waste cleanup, telediagnosis, and telesurgery [37,38]. Since then, teleoperation applications have been researched for healthcare [23,39,40], industrial [41], education [5,10], underwater [42], nuclear [43], and energy applications [44]. González et al. [41] proposed a teleoperation system to control industrial robots. The system proposed allows the operator to perform finishing, sanding, deburring, and grinding operations, which are hard to do with industrial robots. By using teleoperation it is possible to preserve the physical integrity of human work and to allow people with motor disabilities to perform grinding and sanding processes. A teleoperated robotic system for healthcare performing ultrasound scanning was presented by Duan et al. [39]. Tests were conducted and proven that using teleoperated robots to take ultrasound was safe and effective [39]. Caiza et al. [44] used the lightweight protocol MQTT for the teleoperation of robots. An operator can be in a different location and control the robot movements and actions remotely. In the application proposed by Caiza et al., an operator can control a mobile manipulator to perform inspections on oil and gas equipment [44].

Real-Time Video

Low latency of real-time video transmission is critical in teleoperation applications [45]. Delay in video transmission has a negative impact on user experience and may cause the user to misguide teleoperated equipment. Web Real-Time communication project (WebRTC) is an open-source project offering near real-time web-based communication [46]. Most modern web browsers support WebRTC, and Round Trip Times of 80 to 100 milliseconds have been measured on mobile platforms [47]. WebRTC utilizes User Datagram Protocol (UDP) for video streaming. Since UDP does not support congestion control, a custom congestion control algorithm is required to control the video stream.

3.3. Authentication and Authorization

To meet requirements F4, F5, and F6, tools to authorize platform users and define user permissions are needed. Most modern CMSs’ provide described functionality and also tools for website content management [48]. CMS provides schemes and front-end for user registration, strong password creation, password authentication, and password recovery. Currently, three major open-source CMS solutions are WordPress, Joomla, and Drupal [49]. A wide variety of third-party plug-ins are available for mentioned three major CMS. Plug-ins can add features such as calendars, clocks, and photo galleries to CMS [48].

To allow registered users to log in to all services on the platform with a single username and password, a single sign-on (SSO) method is required. Single sign-on allows registered users to log in once and access all services on the platform. Open Authorization 2.0 (OAuth 2.0) is a token-based open standard for authentication and authorization [50]. OAuth 2.0 is a widely utilized standard in Internet communications utilized by Google, Amazon, and Paypal [51]. Traditional server–client style authentication is based on sharing credentials between resource owner and client. Sharing credentials with clients can lead to password leaks, data breaches, and unwanted access to protected resources [50]. OAuth 2.0 differs from traditional server–client style authentication by passing access tokens instead of credentials from the authentication server to the client. OAuth 2.0 enables users to utilize all services on the platform by creating single credentials on Drupal CMS. OAuth 2.0 is directly related to requirement N3.

4. Implementation and Validation

4.1. System Architecture

The system architecture of the platform was defined based on requirement specification and software component selections described in Section 3. Internal and external connections of architecture are described in detail in Figure 3.

The main part of platform system architecture is cloud server running services required to enable the platform for digital twins. Jitsi-Meet, Node-RED, and Mosquitto-MQTT services are directly related to the teleoperation of robots. DTs on the platform are digital entities validating and passing data between teleoperation user interface and physical systems. DT implementation of the platform is described in detail in Section 4.5. NodeJS runtime is required to run Node-RED, DTs, and Rocket.Chat servers on the system. Apache is required to host Drupal CMS and to act as a secure reverse proxy for Rocket.Chat and Jitsi-Meet services. In addition to content management, Drupal and OAuth-plugin provide registration, authentication, and authorization of users. Fullcalendar-module is required to enable functionality for time-resource management. SSH–server provides administrator access to the cloud server.

Robo3D Lab Server is located at Centria robotics lab. The purpose of this server is to act as a link between the cloud server and robots located at Robo3D Lab. Robotic systems available on the platform are physically connected to the Robo3D Lab server by LAN and WLAN connections. Additionally, web cameras utilized for monitoring teleoperated robots are connected to this server by USB connection. Robo3D Lab server is running OPC UA server and client to connect the industrial robot to cloud server, MQTT-app to connect mobile robot, and Jitsi-Meet client to stream video to Node-RED user interface. Additionally, the Kandao 360-camera streamer is connected to the Robo3D Lab server to stream a 360 view of the laboratory for a user to control and locate the mobile robot.

UiT Server is located at the UiT manufacturing laboratory in Narvik, providing a link between the cloud server and equipment located in Norway. Industrial and Scara robots and the conveyor system are connected physically to UiT Server utilizing LAN connections. Similar to the Robo3D Lab, server web cameras for monitoring CPS are connected to the server by USB connections. The server is running OPC UA Server and OPC UA/MQTT-bridge to connect robots to the cloud server.

4.2. Cloud Server

To set up our platform, DNS address was registered for the project at Centria’s DNS server, redirecting all requests to the static IP address provided by the cloud service provider. Ubuntu 20.04 with the latest updates was installed to provide a server OS. Apache, PHP, and MySQL server stack were installed on top of Ubuntu and configured to host Drupal CMS, Rocket.Chat, Jitsi-Meet, and Node-RED. A firewall was enabled on both the Ubuntu server and cloud service, allowing access to only ports required for platform functionality explained in Figure 3. CA certificate request was approved by GEANT Vereniging, and after approval, Apache was set up to only accept encrypted HTTPS connections. These steps correspond to requirements N13 and N3, requiring a 24/7 system availability, and CS was taken into account by encryption described in Table 1 and Table 2.

4.2.1. Content Management System

Drupal provides automation for user registration, secure password creation, password recovery, and permission management tasks. Drupal also features calendar and modern authorization server plugins and was therefore chosen as CMS for the platform. Drupal CMS was installed to provide a front-end of the platform and a secure mechanism for user registrations, password recovery, and authentication. The front end of the educational platform was created with content management tools provided by Drupal. The front-end consists of menus, web pages, articles, and document files. Providing platform users information about utilizing platform and robotics study materials, Drupal add-on “Open authentication 2.0”-server described in Section 3.3 provides trivial and secure login to platform services Rocket.Chat, Node-RED, and Jitsi-Meet. Drupal add-on module Fullcalendar-module provides a method for teleoperation time resource management. Drupal CMS and add-on modules described providing solutions for requirements F4, F5, F6, and N3, N6, N7, N12.

4.2.2. Time Resource Management

FullCalendar-module was installed as a time resource manager. The back-end of FullCalendar is written in PHP and the user interface in Javascript. Since FullCalender-module does not offer logic for user-specific time resource management, the module source code was modified to include user id information for events and to monitor current time and reservation time, including user id and monitoring of current time enabled implementation of logic needed for time resource management. FullCalendar-module is directly related to the scheduling requirement described in requirement F6.

4.2.3. Real-Time Video Server

Jitsi-Meet utilizes Google Congestion Control (GCC), which is the most widely utilized congestion control algorithm for WebRTC [47]. There is a variety of open-source video conferencing applications based on WebRTC, such as BigBlueButton, Jitsi-Meet, and OpenVidu [46].

Jitsi-Meet was chosen for the proposed platform since it has been proven to perform well compared to BigBlueButton and OpenVidu [46]. Jitsi-Meet is a collection of open-source projects providing voice, messaging, desktop sharing and video conferencing applications in real-time browser-to-browser fashion [47,52]. CS is built-in, as Jitsi-Meet accepts only encrypted communication. User authentication is also possible and authentication was set up for this implementation.

Jitsi-Meet server provides common features of videoconferencing for the platform. The most important feature is web-based real-time video streaming. Video meet provides near real-time monitoring of teleoperated robots and enables a way for students and teachers to communicate. Jitsi-Meet was installed on a cloud server utilizing a ready-made installer. Jitsi-Meet requires SSL encryption and encryption was set up with the CA certificate described earlier. Jitsi-Meet CS was further improved by allowing only OAuth 2.0 authenticated users to start and join meetings on the platform. Jitsi-Meet relates to requirements F2, F3, and N3, N8, N9, N11, and N15.

4.2.4. Teleoperation User Interface

Node-RED was chosen as a development tool for the platform teleoperation user interface. Node-RED is a flow-based programming tool developed by Nick O’Leary and Dave Conway-Jones at IBM [53]. Node-RED provides visualization of MQTT-topics by web-based flow-editor [53]. Flow-editor provides a library of useful functions and templates, integration of custom Javascript code blocks, and connections between code blocks. Node-RED requires NodeJS runtime and runs as a service on a cloud server or local workstation. Node-RED is well documented and the developer community is active. Node-RED offers encryption of data and authentication of users by username and password or OAuth2 token as CS measures.

Node-RED server setup allows connections by default on port 1880. Node-RED is an important part of the platform connecting with DTs, providing an interface to control and monitor CPS. After setting up, Node-RED flows and dashboards to control robots at Ylivieska were created. Node-RED provides native support for SSL encryption and encryption was set up by utilizing the CA certificate described earlier. Node-RED relates to requirements F1, F2, F3, N2, N3, N8.

4.2.5. Cloud Data Transfer

Mosquitto was chosen as MQTT-broker for the platform, as it provides authentication and SSL-encryption with CA certificate [54]. Mosquitto MQTT-broker was set up to route messages between digital and physical twins. Port 1883 was specified for non-encrypted data transfer and port 8883 for encrypted. Authentication and encryption of data was enabled and connections without authentication were disabled. Encrypted connections were secured with CA-certificate provided by GEANT Vereniging. Simple topic structure was defined for publishing and subscribing messages. nnnTopic/from was defined as topic for receiving messages from equipment and nnnTopic/to was defined as topic for sending messages to equipment. For example, fanucTopic/to and fanucTopic/to were defined as topics for Fanuc LR-mate industrial robot to publish and subscribe messages.

A combination of OPC UA and MQTT standards provide industry-standard data exchange between digital and physical twins. These two protocols were chosen for data exchange on the platform proposed. These protocols are widely supported by IoT and IIoT devices and are capable of cross-communication. Open62541 was chosen as the OPC UA implementation for this project since it has been proven CS and to use the least computational resources [14,29]. Open62541 corresponds to requirements F1 and N3.

4.2.6. Social Communication

Rocket.Chat was set up to provide a way of social communication for platform users, administrators, and developers. Users do not always require real-time communication and instead chat type communication is sometimes more convenient. Rocket.Chat is set up to require a valid Drupal user account, and to allow anonymous messaging. Anonymous messaging lowers the bar to raise questions related to exercises and usage of the platform. Rocket.Chat supports OAuth2.0 and a valid Drupal user account is verified from CMS. To enable encrypted secure connections for Rocket.Chat it was set up with Apache ReverseProxy, as described in Rocket.Chat manual [55]. Rocket.Chat relates to requirements N2, N6, N3, and F14.

4.2.7. Vulnerability Scans

After installation, vulnerability scans on the platform were performed with Nessus and OWASP ZAP software. The first scan revealed several CS risks on the platform. The server was not running the latest versions of Apache and PHP. After updating Apache, PHP, and disabling weak SSH–algorithms, only fine-tuning of Apache parameters to hide server version information was required. As the latest scan indicated no CS issues, the platform was considered currently up-to-date and cybersecure. However, since new security issues appear daily, vulnerability assessment is scheduled to run monthly. The report is e-mailed to the site manager and measures to correct CS issues will be taken. Periodic vulnerability scans are directly related to the requirement N3.

4.3. UiT Manufacturing Laboratory

At the manufacturing laboratory in Narvik, Norway, there is a reconfigurable manufacturing system (RMS) that consists of multiple platforms. The platforms can be easily moved to any location in the environment. Three of the platforms in the RMS have been connected to the cloud system; two robot platforms and one conveyor platform.

4.3.1. Industrial Robot

The Nachi robot is connected to a CFD-0000 controller and can be used with other PLC systems, but it does not allow for controlling from an external computer. Therefore, a FD High Speed (FD-HS) system is used, which is connected between the CPU board and servo board of the controller and allows for control by an external computer. The FD-HS system can only be used for joint control, and therefore, a small NUC computer is added with ROS and MoveIt for inverse-forward kinematics calculation. In addition, the NUC computer is also used to connect the robot with the OPC UA server for remote operation.

4.3.2. Scara Robot

The original controller of the Scara robot had become obsolete and has been replaced with a LinuxCNC controller, and a second control computer (NUC computer) is added to connect the robot with the OPC UA server. The second control computer also runs ROS and uses MoveIt for the inverse-forward kinematics calculation, the same as the Nachi robot.

4.3.3. Conveyor

The conveyor includes the motor controller and five distance sensors. Figure 4 shows the control interface of the conveyor platform. Users can control the conveyor velocity and direction while reading the sensor values in this interface.

A Raspberry Pi operates as the control computer to control the motor of the conveyor, collect data from the sensors, and synchronizes the motor and sensor data with the OPC UA server.

4.3.4. UiT Server

For the local system in Narvik, the OPC UA server is used to gather/upload data from the local platforms and distribute the data from the cloud to each of the robot arms and the conveyor. To communicate with the robot arms there are two sets of variables: requested joint values and current joint values. If the requested values in the OPC UA server are changed, the robot will automatically start to move towards the new requested values. The current value is updated while the robot is moving. An illustration of the connection can be found in Figure 3.

The cloud server uses MQTT for communication. Therefore, a communication program synchronizes the data between the local server in Narvik and the cloud server MQTT.

4.4. Centria Robo3D Lab

At Ylivieska Robo3D Lab, two articulated arm robots and one mobile robot were chosen as hardware for implementation. The industrial arm robot is Fanuc LR-Mate intended for educational purposes. Fanuc LR-Mate is an industry-standard robot, providing 6 rotary joints for positioning and a gripper for attaching and detaching work pieces. The collaborative arm robot is Universal Robot UR3. UR3 is a collaborative robot providing 6 rotary joints, an adaptive gripper, and functions enabling human–robot collaboration. In addition to robots, a server PC was required. Robo3D Lab server is required to Host OPC UA client/server for Fanuc robot, stream real-time video broadcast for a collaborative and industrial robot, and host custom Python script to control a mobile robot. Configuration and connections of Robo3D Lab server are described in Figure 3.

4.4.1. Industrial Robot

Fanuc industrial robot controller provides OPC UA server for communication, MQTT is not currently supported. Therefore, custom OPC UA to MQTT-bridge was implemented. After enabling the OPC UA server option, the OPC UA server with factory default structure was running on a robot controller. Default structure included current position registers and alarm registers. A requested position register was added to receive commanded joint positions from the platform user interface. OPC UA client and server are modified implementations of Open62541. OPC UA Client is a connector between the robot OPC UA server and Robo3D Lab OPC UA server. The purpose of the OPC UA server is to act as gateway routing messages received from implemented OPC UA client to the cloud server MQTT-broker.

Fanuc robot OPC UA server does not support authentication or encryption of OPC UA data connections. To maintain CS confidentiality and integrity of data transfers between robot and platform, an isolated subnet connection consisting of only Fanuc LR-Mate robot and Open62541 implementation was set up. Server running Open62541 implementation utilizes two network adapters: one for connecting to an isolated network, including the Fanuc robot, and one for connecting to the internet router. Connections of setup are described in Figure 3.

Physical safety considerations for this robot are fully considered by the manufacturer, as this robot cell is CE-approved. The robot is installed inside a fully enclosed structure accessible only through a sliding door. The sliding door is secured by a safety switch stopping robot motion if opened.

4.4.2. Collaborative Robot

Universal Robot UR3 controller does not provide OPC UA or MQTT communications by default. Instead, the controller provides the possibility to install third-party applications, including OPC UA clients and MQTT connectors. In this case, MQTT-connector provided by 4Each software solutions was installed on the robot controller. 4Each MQTT Connector enables publishing and subscribing messages to MQTT-broker.

MQTT-connector connects to the cloud server transmits robot’s current position and receives the requested position from the user interface. The program was also modified to allow commanding the robot to Tool Centre Point (TCP) to requested XYZ-coordinate positions.

The user interface created with Node-RED consists of six gauge-type indicators for current joint positions and sliders for each joint to set the wanted position. Above the gauge and slider elements is an embedded live video of a collaborative robot station. The user interface is presented in Figure 5.

A risk assessment was conducted for this robot to evaluate possible safety risks regarding teleoperation. UR3 robot differs from Fanuc industrial robot described previously since it was not delivered as a complete cell but as a robot arm and controller unit. The final installation of the robot, gripper, and controller was conducted by Centria, and no CE approval for a complete cell was provided. Risks during teleoperation of the robot are that people near the physical robot are not aware of teleoperation. UR3 is a collaborative robot, designed to share working space with humans so the risk for physical harm is small. Enclosure manufacture of transparent polycarbonate sheet could provide added protection if required.

4.4.3. Mobile Robot

Omron LD-250 was chosen as a mobile robot for the platform. LD-250 provides Wifi-connection and proprietary command-line interface “ARC link” to control robot movements and to request the status of the mobile robot online. The mobile robot was connected to the ASUS router described earlier, and a Python-based MQTT application was written to send commands to the robot and request status information from the robot controller. MQTT-app is running on Robo3D Lab server and connects to MQTT-broker with paho-mqtt client. The control interface for the mobile robot was implemented on top of the floorplan image of Robo3D Lab. On the higher layer on top of the floorplan, clickable circle objects were created. A user can send pre-defined locations to the mobile robot controller by simply clicking circle objects on the floorplan. Real-time video of mobile robot movement is provided by Kandao 360 camera installed stationary in Robo3D Lab. Video is streamed from Kandao 360 camera to the Robo3D Lab server utilizing RTMP protocol. Figure 6 presents mobile robot user interface.

Physical safety considerations for this robot are fully considered by the manufacturer as LD-250 mobile robot is CE-approved. Mobile robot has sensors to detect and avoid obstacles in a planned path.

4.5. Digital Twinning

As already mentioned, DTs are core components of the platform presented. Each physical system connected to the platform has a digital twin counterpart and automated data flow between physical and digital twins. DTs on the platform are implemented to present physical parameters of CPS, such as current joint and coordinate positions. In addition to presenting the current status of the physical system, DTs validate and transfer user requested positions to physical twins. Physical limitations of connected systems are defined in DTs and a programmed logic is implemented to prevent a user from moving a robot to non-permitted or out-of-reach positions. DTs are written in Javascript language and run on top of NodeJS runtime on the cloud server. DTs communication protocol to the physical world is MQTT. However, the platform supports several communication protocols to integrate a CPS. Integrating a physical system by MQTT, OPC UA, and proprietary “ARC link”-protocols have been described in this paper. Support for the aforementioned protocols is enabled by local servers at Uit and Centria laboratories. Furthermore, by writing a custom MQTT-app as described in Section 4.4.3, it is possible to connect virtually any system with an Ethernet interface to the platform.

4.6. Functionality Validation

To validate the functionality of the proposed platform, a high-level task was defined for each robot station. High-level tasks defined are specific to the robot type. For example for industrial robot’s pick and place task is typical, and therefore, a simple pick and place task was defined for both industrial robots. For a collaborative robot, a task utilizing feedback of gripping force and distance was defined. A material transport task was decided suitable for the mobile robot. Tasks were planned by creating a flowchart defining the functions required for the task. A flowchart for material transport task is presented as an example in Section 4.6.3. If a defined task could be performed utilizing digital presentation, validation was considered successful, otherwise validation was considered failed. Individual validation for each robot station is described in detail in the following subsections. The goal for validation is to verify that the implementation meets functional requirements F1-F6 and non-functional requirements N1, N2, N3, N4, N9, and N15.

4.6.1. Industrial Robots

For both industrial robots, a simple pick and place task was defined. The task includes moving a cylindrical workpiece from one position to another. Conducting this task requires positioning the robot to approach, pick, place, and retract positions. Additionally, control for gripper’s close and open functions is required. The correct coordinates for pick and place positions were provided prior to programming. Initial validations pointed out that functionality for the gripper state feedback was missing and the robot did not wait for the gripper to open or close before initiating the next move. Feedback functionality for the gripper was implemented and validation was successful. Figure 7 describes the cycle of the pick and place task defined for industrial robots.

4.6.2. Collaborative Robot

For a collaborative robot, a simple assembly task was defined. The task is to press a plunger to a predefined depth of 45mm inside a solenoid housing. Programming of this task requires the utilization of linear positioning, joint positioning, adaptive gripper force, and position commands. During the validation, it was noted that the DT of the robot station did not have all functionalities required to perform the defined task. Missing functionalities of setting the gripping force and position were added to DT. Additionally, a function to switch between programming of either linear or joint move was added to pass the validation.

4.6.3. Mobile Robot

For the mobile robot, a material transport task was defined. The task requires the planned path from the park location to the loading station and from there to the unloading station. The mobile robot acts on “ready to load” and “ready to unload” messages received from two robot stations. The initial validation failed because the functionality to send and receive messages from and to robot stations was not implemented on the platform. These functions were added to the MQTT-app controlling the mobile robot and validation was successfully carried out. The mobile robot MQTT-app loading logic is described in Figure 8.

4.6.4. Scara Robot and Conveyor

For Scara and conveyor, a combined task was created to validate functionality. The objective of the task is to use the conveyor and Scara robot to interact with an object. In the task, the conveyor moves an object over to the Scara robot and the Scara robot interacts with the object. In the task, the Node-red interface is used to control both the Scara robot and conveyor and it is possible to switch between these two interfaces. The validation test showed that it is possible to make different machines interact with each other and it is easy to switch between the Node-red control interfaces.

5. Discussion

This paper utilized DTs to control equipment in two laboratories. Controlled pieces of equipment were industrial, mobile, and collaborative robot stations and a conveyor system. The overall description of the platform is presented in Figure 9. To validate the functionality, the prototype of the proposed platform was created and presented. Platform development presented in this paper opens discussions on how a common platform for teleoperation can be utilized for sharing equipment resources, creating a roadmap for cybersecure CPS, and a teleoperation platform created with open-source software. Key findings are listed in the subsections below.

5.1. MQTT Is an Efficient Communication Protocol for DT

MQTT was chosen as the main data exchange protocol of the proposed platform because MQTT is a modern protocol based on publish/subscribe communication. MQTT allows any node to publish or subscribe to messages, providing a flexible messaging platform. OPC UA, on the other hand, is tied to the server/client communication model. MQTT communication is trivial to set up because no creation of variable structures is needed. Creating variable structures on the OPC UA communication model would be a time-consuming phase during the implementation. Utilizing the MQTT protocol does not require defining variable structures and is not tied to the client/server model.

5.2. Latency of Video Stream Is Critical in Teleoperation

The latency of live video stream is a critical aspect in teleoperation applications. During implementation, it was observed that latency of over 500 ms is unacceptable and has a negative impact on user experience. Choice of open source video conferencing applications based on WebRTC provides ready mechanisms for controlling video resolution to maintain latency usable for teleoperation. WebRTC has been proven to provide latency in the sub 200 ms range which was found adequate for the teleoperation of robots. Modern web browsers support WebRTC and can act as clients for near real-time video streaming.

5.3. Cybersecurity Is a Key-Enabler

CS of data transfer is a fundamental part of the online DT platform. CS aspects require consideration in the planning, development, and implementation phases of the platform. During planning it is important to select only components that are cybersecure, as one component including a vulnerability can compromise cybersecurity of the complete platform. Corruption of data can lead to unwanted movement of CPS causing damage to teleoperated CPS or the surrounding physical environment. A leak of user credentials can allow a hacker to gain control of the teleoperation interface causing intentional damage to CPS. Securing all data transfers with authentication and encryption is required to maintain data integrity. Periodic vulnerability scans have been scheduled by Centria to maintain CS since new vulnerabilities are found on a daily basis. Vulnerability scans have indicated only a few minor CS issues. The presented platform is based on open-source software and is cybersecure.

5.4. Privacy and Safety Concerns

During the development and implementation of the platform, concerns related to privacy and safety of humans physically near teleoperated CPS arose. It was noted that: (1) web cameras used to monitor CPS feature built-in microphones and can be used for eavesdropping, (2) web camera live video enables the possibility for spying, and (3) teleoperated CPS can cause physical danger to humans nearby. There are many possible solutions to overcome the mentioned privacy and safety issues. Placing the teleoperated CPS in a restricted area solves all the identified issues. In practice, especially with mobile robots, this can be challenging since a physically large restricted area for operation is required. Another option to guarantee total privacy is to offer teleoperation only outside office hours. Physical disabling of microphone hardware disables the possibility of eavesdropping but does not solve video spying or physical safety concerns.

5.5. Based on Open-Source

The platform presented was built mostly on open-source software. For many components such as CMS, video conferencing, and OPC UA communications, there are many options to choose from. Setting up open-source software for the platform presented was considered trivial. Support forums exist, and help for developing and setting up software is available. However, not all software components presented in this paper are open source: the collaborative robot MQTT-connector provided by 4Each software solutions is commercial software.

5.6. Is Teleoperation Platform Utilizing DT’s Suitable for Robotics Education and Training?

The proposed platform was piloted and evaluated by a group of 20 students participating on a course of digital twins. Each student registered to the platform by providing an email address. Teleoperation, resource management, chat, and online video capabilities were tested individually and as an educational group. Students discussed on the platform, reserved time-slots for exercises, and utilized teleoperation. Teleoperation was piloted by conducting high-level tasks described in Section 4.6 as online exercises. Based on the feedback, the platform provides required functionalities and is suitable for robotics online education and training. According to the feedback, a user experience could be improved by re-designing user interfaces and adding a variety of exercises and study materials to the platform.

6. Conclusions

This paper proposed a digital twin-based teleoperation platform to control various robotics systems remotely. Full system specification and implementation details were given. Ecological and social values, the aspects of i5.0, were included and considered during the system design. A prototype containing robot cells located geographically in two countries was implemented and reported. The proposed system allows effective resource sharing in situations where suitable devices are not or cannot be possessed but can be rented, borrowed, or otherwise used remotely. The presented resource sharing allows a single user to reserve one robot at a time for the teleoperation. In addition, the system allows multiple simultaneous users controlling different robots at same time. The proposed system is highly flexible; any cyber-physical system can be included in the platform if it supports the defined open interfaces. Because of these interfaces, the platform itself is also highly affordable.

The first steps in building a common platform for teleoperation are presented in this paper. The next possible further steps to develop the proposed platform further are discussed. Utilizing eXtended Reality-technologies(XR) as a user interface for monitoring and programming robots. To provide a platform-independent way to experience XR, a solution supporting mobile devices should be chosen. Utilizing XR would meet requirement F11 and prepare the platform for requirements F12 of virtual deployment and N16 of gamification.

Improving support for mobile devices would increase the accessibility of the platform. Currently, accessing some functions of control interfaces may require scrolling and zooming of view, affecting the user experience negatively. To improve user experience, teleoperation and CMS layouts should be re-designed and re-implemented. Adding functionality for online examination and providing certificates would enable utilizing the platform for certified training and education, meeting requirements F9 and F10.