Building an Accessible and Flexible Multi-User Robotic Simulation Framework with Unity-MATLAB Bridge

Abstract

Multi-user collaborative robotic simulation has great potential for applications in industry and education. Unity is a powerful software for simulation and online multi-user experience, which can be enhanced with third-party mathematical analysis and multiplayer servers. Unity can become a much more capable and user-friendly robotic simulation package through integration with other software. These include MATLAB for computations and the Photon Unity Engine (PUN) for online multi-user capabilities. This study developed a flexible robotic simulation framework that can be adapted to different scenarios for industrial and educational applications. Several simulation scenarios were developed to identify the most efficient data communication methods between MATLAB and Unity. TCP/IP, Shared Memory, Firebase, and MQTT, were selected to assess their performance and interaction with data in Unity and MATLAB. Next, an independent PUN application was created. Both components were integrated into the simulator for evaluation and performance optimization. The performance of this simulation framework was assessed through two case studies. The results demonstrated that the integrated framework is viable, efficient, and flexible for robotic simulation and digital twins. Future research will expand the framework by adding diverse functionalities to provide users with a better interface, enhancing its performance, and integrating additional software packages.

1. Introduction

Multi-user collaborative robotic simulation has great potential for industrial and educational applications in Industry 4.0. Current software for robotic simulation do not possess these capabilities, which has had led developers to look for accessible alternatives. Custom robot simulations allow developers to tailor the software to their required strengths and for their target userbase [1]. Game development engines have emerged as popular options for custom robotic simulators because of their photorealistic graphics and physics engines [1]. These are desirable due to their cross-platform development, user-friendly interfaces, availability on most common operating systems, and low cost. Game engines have been used in previous studies for standalone robotic simulation in car manufacturing [2], digital twins [3], and medical robotics [4]. However, they lack the capabilities of standard simulators, such as specialized computations and advanced data analytics. Developers have mitigated these weaknesses by connecting the game engines with external applications [1]. Game engines feature open development engines, making them available at low costs and with fewer restrictions than some pre-built systems.

Unity has emerged as a promising game development engine for standalone simulation [5] and as a visualization tool for external simulators [1]. Unity’s user-friendly interface and usage of C# make it accessible to new developers. These features allow new developers to rapidly prototype and test their simulation scenarios. Unity features the capability to add external software packages, such as MLAgents, Robotics, and URDFImporter, which expand its capabilities for computation and simulation. Beyond these features, Unity appeals to new developers because of its excellent developer and community support for learning and troubleshooting. Unity is made to be an open development platform which gives developers freedom to create custom packages and solutions for problems that may arise. The source code has not been made available by the Unity company, but, in its current state, it supports several ways for developers to integrate their custom solutions for their problems. This type of environment is promising to developers due to the freedom it offers when customizing simulation environments.

Unity can independently execute simple simulation scenarios, such as a KUKA manipulator controller [5] and Turtlebot maze navigation [6]. It can also serve as a visualization and synthetic data generation tool for external software. For instance, researchers have integrated Unity with MATLAB to control a mobile robot via Shared Memory [7] and with ROS2 for mobile robot control through TCP/IP [8]. Additionally, different researchers connected Unity to external devices such as a robot arm and custom VR controllers using MQTT [9,10]. Each application leverages Unity’s strengths, including its realistic physics engine, the capability to create accurate models, function as a digital twin [7], and its ability to be a hub for multiple input devices [9,10].

These show Unity’s potential as a simulator for industrial and educational applications. For industrial applications, it can be implemented as a design and testing tool before robots are implemented in the real world [11]. Digital twins are useful after systems have been developed digitally to monitor the devices remotely and verify that they maintain normal operation [12]. These elements have been used in areas such as construction for collision detection [13], manufacturing to monitor milling machines [14], and industrial design for machinery [11]. In education, simulation reduces the need for physical hardware to effectively teach robotics. This includes individual hardware kits for students [15] and large machinery, which would be difficult for students to access [16]. Virtual scenarios can be used to study the design process of robots and how their components work at a deeper level [16].

Multi-user collaborative robotic simulation has potential for applications in industry and education. Online multi-user connectivity is useful for simulation because it enables simultaneous participation of multiple users in a shared virtual space. Simulators can use multi-user connectivity through peer-to-peer (P2P) connections or centralized client–server architectures. P2P connections allow players to connect directly, reducing latency and server costs, but can lead to security issues and increased loads on networks [17]. In contrast, server-based connections host the applications on remote dedicated servers, enhancing security and consistency, though resulting in potentially higher latency and infrastructure costs [17]. Server-based connections can be created with external services, which allows developers to focus on software development. Researchers have used online multiplayer platforms for educational purposes by turning simulators into multi-user spaces used to instruct students about manufacturing [18,19], and to create collaborative spaces to learn modern technologies [20].

Current researchers have shown that a multi-user robotic simulation environment based on Unity is feasible. These have focused on building systems that connect several users in aerial drone simulations [21], human–robot interactions in VR [22], and psychology treatments for mental health [23]. Researchers argued that existing simulators often do not reflect real operating conditions for robots, but by using Unity they could amend this. They showed that Unity’s abilities to create environments that closely resemble real-world environments would be indispensable for future simulations [23,24]. They also argued that new methods for multi-user simulation are needed, ones that can scale for large number of users and possess realistic physics capabilities [25]. As shown in the previous studies, Unity features these capabilities, making it ideal for this type of use case.

As shown in the literature by O. A. Aguirre et al. [21], Novitzky et al. [24], and Andaluz et al. [7], specialized simulators in Unity are valuable tools for the target tasks. This study uses a different approach by creating an open flexible robotic simulation framework to simplify the process of robotic simulation. Demonstrations for how to use the application, tools to streamline the addition of multiplayer and messaging applications, and sample robotics assets are also provided. This study presents an in-depth review of the readily accessible software that was used to build the simulation framework and how these can be adapted for a variety of users and systems.

The goal of this study is to build a framework for accessible robotics simulation with Unity. This framework will incorporate support from external software and multi-user robotic simulation. Multiple connection methods were compared to establish the benefits and drawbacks of each one when linking Unity with an external software. The chosen software must interface with diverse communication methods, robust data analysis capabilities, and be accessible to a wide range of users. Assessment of Unity as a viable, efficient, flexible, and accessible alternative to simulate robots at a low cost to the user is a relevant element of this study.

The rest of this paper is structured as follows. Section 2 presents the methodology for the developed framework. Section 3 shows results and discussion. The last section presents the conclusions along with a brief discussion of the ongoing and future research.

2. Materials and Methods

The proposed framework is intended as a testing environment for developers to create collaborative multi-user physics-based robot simulations. Development was split into three stages: development in Unity, software integration using communication methods, and multi-user implementation with Photon Unity Network (PUN). PUN features an extensive list of features ideal for researchers and reliable servers. Additionally, PUN does not create any of the additional strain on the end user devices that a P2P implementation would require. Without loss of generality to other robot models, the Meca500 robot arm and Reeman Mobile Robot were selected as the target hardware for this collaborative framework development. MATLAB was chosen as the software for data analysis because it features user-friendly robotic and communication toolboxes, and it is available in many universities.

2.1. Unity and MATLAB Simulation Development

First, independent Unity and MATLAB robot simulators were created as foundational elements for the framework. The Unity simulators create scenarios to control the joints of the Meca500 robot arm [26,27] and to move the Reeman mobile robot [28] through different environments. The MATLAB simulators solve Inverse Kinematics (IK) for the Meca500 arm and manage the mobile robot navigation for the Reeman robot. The Unified Robotics Description Format (URDF) and 3D Computer-Aided Design (CAD) files are the basis for the robot models. The simulators utilize the Unity Robotics packages, MATLAB robotics library, and MATLAB navigation to interface these models with the virtual environments.

Unity Robot Arm Simulation: The Meca500 Unity simulator was developed in three stages: robot model import, robot control, and user interface. The robot was loaded into the scene with the URDF Importer package, which extracts the link and joint parameters from the URDF file. This process creates the robot skeleton and attaches the robot graphical elements to it. Figure 1 shows the imported robot model in Unity with the joint dimensions and end effector orientation displayed over it. The Denavit–Hartenberg (D-H) parameters in

Table 1. Denavit–Hartenberg parameters table.

Joint	θ (deg)	α (deg)	R (mm)	D (mm)
J1	θ1	0	0	90
J2	θ2	−90	0	45
J3	θ3	0	135	0
J4	θ4	−90	38	0
J5	θ5	90	0	120
J6	θ6	−90	0	70

were used to create this model. This table consists of four parameters which describe the position and rotation of the robot joints:

• θ: the angle between the current joint’s X plane and the next joint’s X plane along the Z axis;
• α: the angle between the current joint’s Z plane and the next joint’s Z plane along the Z axis;
• R: the distance between the current joint and the next joint along the X axis;
• D: the distance between the current joint and the next joint along the Z axis.

The individual joints are controlled with a script that receives a target position, compares it with the current position, and moves the joint accordingly. To add more realism to the simulator, the robot joints were given speed controls and range limits. Finally, the user interface was implemented in the form of sliders and buttons (Figure 2). Each slider was used to create a target position for each joint, and the buttons were programmed to stop all robot motion or to reset all joint angles to zero.

Unity Mobile Robot Simulation: The Reeman Unity simulator was developed in three steps: robot modeling, robot movement, and object detection with sensors. The model was created with an imported 3D CAD file of the mobile robot. This model was given physics-based properties and was assigned a Unity character controller for movement control. Figure 3a shows the digital model of the Reeman robot. The movement was designed to closely match the real robot. As such, the simulated robot only moves in the XY plane and turns to face the direction of movement. The controls for the robot are the four arrow keys in the keyboard for directional control, with other keys used to manage the speed. The LIDAR and Ultrasonic sensors are simulated using arrays of raycast objects, which are rays that operate as motion sensors. To accurately simulate the sensor and its circular shape, the rays in the arrays rotated by a given amount, which are obtained using Equations (1) and (2).

$$\mathrm{r}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{M}\mathrm{o}\mathrm{d}=\mathrm{A}\mathrm{n}\mathrm{g}\mathrm{l}\mathrm{e}\mathrm{A}\mathrm{x}\mathrm{i}\mathrm{s}\left(\left({\displaystyle \frac{\mathrm{r}\mathrm{a}\mathrm{y}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}}{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{O}\mathrm{f}\mathrm{R}\mathrm{a}\mathrm{y}\mathrm{s}-1}}\right)\ast \mathrm{a}\mathrm{n}\mathrm{g}\mathrm{l}\mathrm{e}-\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{l}\mathrm{e},\left(0,1,0\right)\right)$$

(1)

$$\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{r}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{r}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{M}\mathrm{o}\mathrm{d}\ast (\mathrm{0,0},1)$$

(2)

Equation (1) returns rotationMod, a quaternion to represent how much the current raycast object needs to be rotated. It uses AngleAxis, a prebuilt function in Unity, to generate a rotation quaternion for a given raycast. In this function, rayindex is the current index of the raycast object array, numOfRays is the total number of raycast objects in the sensor array, and the angle is the LIDAR range. Equation (2) computes the desired orientation vector for the current raycast object by multiplying the current rotation vectors, the rotationMod obtained in Equation (1), and the vector (0,0,1). Orientation shows how the current ray should rotate to its respective orientation based on its index. This process is repeated for all the rays in the array. Figure 3 shows the process of instantiating LIDAR with an angle range of 135 degrees and a ray count of 240. Figure 3a represents the initial state for all the raycasts in the array before they are transformed. Figure 3b shows the LIDAR with the 240 raycasts transformed following Equation (2).

MATLAB Data Sampling: The MATLAB simulator was developed to provide a reliable system for constant data exchange and analysis with the Unity simulators. Figure 4 shows the process the simulators used to sample for new data. The first stage is a continuous loop surveying the communication method for new data and verifying the correctness of the data. The data analysis for the respective simulator is then initiated once the data are received and verified. The final stage sends the processed data to Unity and restarts the data-sampling loop.

MATLAB Computation for the Robot Arm: The Meca500 simulation first receives the position of user-controlled object and calculates its Inverse Kinematics with the built-in MATLAB function. Then, it uses Equations (3)–(7) to determine the speed for the robot, and finally transmit the data to Unity.

$$\mathsf{\xi }=\mathrm{J}\left(\mathrm{q}\right)\ast \mathrm{q}\to \left[\begin{array}{c}{\mathrm{v}}_{\mathrm{n}}^0\\ {\mathsf{\omega }}_{\mathrm{n}}^0\end{array}\right]=\left[\begin{array}{c}{\mathrm{J}}_{\mathrm{v}}\left(\mathrm{q}\right)\\ {\mathrm{J}}_{\mathsf{\omega }}\left(\mathrm{q}\right)\end{array}\right]\left[\begin{array}{c}{\mathrm{q}}_1\\ \dots \\ {\mathrm{q}}_{\mathrm{n}}\end{array}\right]$$

(3)

$${\mathrm{J}}_{\mathrm{v}\mathrm{i}}={\mathrm{z}}_{\mathrm{i}-1}\times \left({\mathrm{o}}_{\mathrm{n}}-{\mathrm{o}}_{\mathrm{i}-1}\right)$$

(4)

$${\mathrm{J}}_{\mathsf{\omega }\mathrm{i}}={\mathrm{z}}_{\mathrm{i}-1}$$

(5)

$$\left[\begin{array}{c}{\mathrm{J}}_{\mathrm{v}}\left(\mathrm{q}\right)\\ {\mathrm{J}}_{\mathsf{\omega }}\left(\mathrm{q}\right)\end{array}\right]=\left[\begin{array}{ccc}{\mathrm{z}}_0\times \left({\mathrm{o}}_6-{\mathrm{o}}_0\right)& {\mathrm{z}}_1\times \left({\mathrm{o}}_6-{\mathrm{o}}_1\right)& {\mathrm{z}}_2\times \left({\mathrm{o}}_6-{\mathrm{o}}_2\right)\\ {\mathrm{z}}_0& {\mathrm{z}}_1& {\mathrm{z}}_2\end{array}\begin{array}{ccc}{\mathrm{z}}_3\times \left({\mathrm{o}}_6-{\mathrm{o}}_3\right)& {\mathrm{z}}_4\times \left({\mathrm{o}}_6-{\mathrm{o}}_4\right)& {\mathrm{z}}_5\times \left({\mathrm{o}}_6-{\mathrm{o}}_5\right)\\ {\mathrm{z}}_3& {\mathrm{z}}_4& {\mathrm{z}}_5\end{array}\right]$$

(6)

$$\mathrm{V}=\sqrt{{\mathrm{x}}_{\mathrm{i}}^2+{\mathrm{y}}_{\mathrm{i}}^2+{\mathrm{z}}_{\mathrm{i}}^2}$$

(7)

Equation (3) calculates a 6 × 1 matrix which contains the XYZ components of the linear (${\mathrm{v}}_{\mathrm{n}}^0$) and angular (${\mathsf{\omega }}_{\mathrm{n}}^0$) velocities of the end effector. This is accomplished by multiplying the 6 × 6 Jacobian matrix (J(q)) of the robot with the 6 × 1 velocity matrix of the joint (q). The Jacobian matrix returns the XYZ components of the linear and angular effects that the movement of each joint has on the subsequent joints. This matrix has two pieces: the top three elements use Equation (4) to return the linear components, and the bottom three elements use Equation (5) to return the angular components. In Equations (4) and (5), the Z and O elements, respectively, represent the third and fourth columns of the transformation matrix between the current joint and the robot’s origin. Equation (6) shows what the final Jacobian Matrix would look like before the transformation matrixes are plugged in. Equation (7) uses the XYZ components of the angular and linear velocity vectors to determine the velocity magnitude of each joint. This magnitude returns one singular speed, which determines the joint’s actual speed.

Figure 5 shows how MATLAB analyzes the data from Unity. In this process, MATLAB receives the Unity Robot joint angles and user object XYZ coordinates. This process initially receives the robot joint angles and user object XYZ coordinates from Unity. Afterwards, it uses these data to compute the inverse kinematics and its Jacobian matrix. Then, it determines the velocity magnitude for each joint from the Jacobian matrix. Finally, it streams the velocities and new joint positions to Unity.

MATLAB Computation for the Mobile Robot: The MATLAB simulator for the Reeman robot has a mapping mode and a navigation mode. The former uses sensor data to build a map which is used in the latter mode with a navigation algorithm.

• The map is built by generating several LIDAR sensor readings, putting them into individual LIDAR scan objects, and using these objects to build a 2D binary occupancy map.
• The path between the robot’s start and end positions is generated using the Hybrid A-star algorithm. This algorithm generates the path between the two positions and adds an extra perimeter to the map objects to avoid collisions. The path is stored as an array of positions in MATLAB, which is sent to Unity upon request.

2.2. Communication Methods and Complex Simulation

The second stage of framework development involved determining which method would be the most suitable method to exchange data between MATLAB and Unity. Based on the literature review, TCP/IP, Shared Memory (SHM), Firebase, and MQTT were selected to be evaluated in the test scenarios. TCP/IP, SHM, and MQTT were chosen based on their usage in similar applications to this one. Firebase, on the other hand, was selected after reviewing its usage in a smart home environment [29] and an application for streaming music and playing games together [30].

• TCP/IP uses a client/server mode and a static IP address to connect Unity and MATLAB so they can share messages [31]. MATLAB connects to the IP address using built-in commands from the Instrument Control Toolbox. Unity connects to the IP address with the System.Net.Sockets library. To establish the TCP/IP communication, one simulator must be considered as the server and the other as the client. In this study, Unity and MATLAB were both tested as servers and clients.
• SHM assigns applications to the same memory space in the RAM. This allows applications to make their data available to other applications in the same space [7]. SHM was implemented by saving the MATLAB application as a DLL (Dynamic-Link Library) file. This file was then imported into the Unity project and used in the C# scripts with function calls.
• Google Firebase is a real-time NoSQL database which allows users to read and write data using a web interface [29,30]. MATLAB and Unity connect to this database using the REST API, which allows the applications to access the web addresses without a specialized library. Data sharing works by having one application send data with the write or put commands. The database stores these until newer data overwrite it. The other application uses a read command to retrieve the latest version of the data stored in the database.
• MQTT is a publish–subscribe messaging protocol that connects applications through web-based applications [10]. NodeRed is used as the MQTT host application; Unity connects to it using the M2MQTT custom package, and MATLAB connects with the built-in commands from the Instrument Control Toolbox. To share data, both applications first connect to the MQTT host using the broker address, and then they interface with a MQTT topic to read or write data. MQTT topics are filters to which clients subscribe to so they can share data with other subscribed clients. If the application writes, then the current data are copied and sent to the server where these can be held. And if it reads, then data are requested and obtained from the server. Two MQTT flows were built, one where MATLAB sends data to Unity, and the other where Unity sends data to MATLAB.

These communication protocols were evaluated in two stages. The first stage quantifies each method’s interaction latency, reliability, and flexibility. The second stage compares the best-performing methods using two tests to determine their feasibility for simulation. Test A created a simulation scenario that sent and received messages to control the Meca500 robot (Figure 6). Test B created a mapping and navigation scenario for the Reeman Robot (Figure 7). Each test was designed to operate with the following parameters:

• Test A: Meca500 simulation.

  ○

  A player controller sphere was added to the Unity scene, the location of which is continuously streamed to the MATLAB application (Figure 6a). MATLAB uses these continuous pose data to repeatedly compute the Inverse Kinematics and speed for the robot (Figure 6b). Back in Unity, the user can press a button at any moment to retrieve the newest joint position and apply it to the robot (Figure 6b). The messages in this simulator contain fifteen data points. Six for the robot joint positions, six for the joint velocities, and three for the points object’s location.

• Test B: Reeman Mobile robot simulation.

  ○

  Mapping mode: The user maneuvers the robot through the scene, scanning it with the LIDAR sensors whose data are streamed continuously to MATLAB (Figure 7a). Once the user has collected sufficient sensor data, they press a button in Unity which signals the MATLAB application to generate the map. Figure 7b shows a plot of the received LIDAR scans by MATLAB before they are used to create the map. Each message in this simulator contained 726 data points. A total of 720 points came from the LIDAR, which streamed the object distance, X rotation angle, and Y rotation angle from each of the 240 raycast object in the array. The remaining six points were from the XYZ cartesian coordinates of the robot, and three rotation angles for the robot orientation.

  ○

  Navigation Mode: In Unity, a pointer object was added to the scene to indicate the robot’s desired end position (Figure 7a). The user moves this object to the desired location and presses a button to send the object’s and the robot’s data to MATLAB. These data are processed and used to create a path between the two points. The path is then sent to Unity as an array of robot location and orientation coordinates (Figure 7c). Finally, the robot follows the received path until it reaches the end position. The result of the map and data generation process can be observed in Figure 7c. This figure shows the map generated from the LIDAR scans in Figure 7b with a path generated between the robot’s start position and end position from Figure 7a. The messages in this simulator contain three arrays of varying size. Two of these arrays represent the XY coordinates of each step in the path and the third array contains the orientation of the robot at each step of the path.

2.3. Photon Unity Network

The multiplayer client for the Unity players was created using the Photon Unity Network (PUN) servers. PUN works in Unity by having each instance connect to the same server using its unique API key. When the applications connect to the server, they join a shared virtual space and are given a random user ID. This allows users to see and interact with each other through network objects, whose data are streamed to all the connected instances of the application. These objects are categorized as server-owned or user-owned. These categorizations determine which users can modify each network object. Figure 8 shows the differences on how the PUN implementations interact with the MQTT simulator (Figure 8a) and with the Firebase simulator (Figure 8b).

Figure 8a shows that each computer only connects to the PUN remote server and handles the remaining processes internally. The Unity-MATLAB communication process in each computer is isolated from the other systems connected to the PUN server. Figure 8b shows each computer connecting to the PUN and Firebase servers simultaneously. The servers are hosted on separate web addresses; so, they do not interact while the application runs. Firebase allows systems to share data with each other through its server.

The PUN implementations for the Meca500 and Reeman simulators focused on converting their scene objects into network objects and implementing methods so all users could interact with the objects. In Test A, the pointer sphere became scene-owned, the robot became user-owned, and the sliders became player-owned. In Test B, the pointer became scene-owned, and the robot became player-owned. The robots were set up as player-owned objects to ensure they could only receive instructions from one user. An ownership takeover system was implemented to ensure that non-owner users could still access the robot. This system allows non-owner users to press a button to request ownership of the robot, and robot ownership is transferred to them if it is not in motion.

The ownership switching model includes methods to deal with user’s losing connection to the server. This method stores the network object positions globally in case a user disconnects the simulation. If the current owner of an object disconnects, then its ownership will be assigned to the user that connected next. This system will keep the application functional in the event that a user has network issues.

3. Results and Discussion

In this section, evaluation results are presented for the framework components as outlined in the Methods section. Section 3.1 discusses the results of Tests A and B. Section 3.2 focuses on the performance of the communication methods with the Meca500 and Reeman simulators. Section 3.3 discusses how the PUN was added to the Unity-MATLAB simulators. Section 3.4 discusses the integration of all the components.

The tests from this section were performed using professional-grade and consumer-grade computers to determine the usability of these for diverse users.

• The specification of the professional-grade computers is AMD Ryzen 9 5950X 16-core processor, 64 GB of RAM, and a NVIDIA RTX 3090 with 24 GB of VRAM.
• The specification of the consumer-grade computers is AMD Ryzen7 3800X 8-core processor, 32 GB of RAM, and an AMD Radeon RX 5600 XT with 6 GB of VRAM.

3.1. Unity and MATLAB Simulation

The goal of our Unity simulators is to create a digital twin of real robots. The simulators were created by analyzing the real robots’ function and replicating them in the Unity digital world. Once the process was completed, the digital models were tested by synchronizing the Meca500 and Reeman robots with their simulated counterparts and evaluating the accuracy of their motion. This process was replicated multiple times to test for repeatability and accuracy over time.

The Meca500 model was tested by giving it specific joint configurations and comparing them with recordings of the real robot performing the same motion. After several iterations, this test showed that the joint positions and motion of the simulated robot were accurate compared to the real robot. Figure 9 shows the result of two tests, one comparing how the arms extend to their full reach (Figure 9a) and one comparing how the arms performed a complex motion (Figure 9b). These results highlight the importance of accurate robot models, such as URDF, and accurate kinematic models to increase the effectiveness of simulation.

The Reeman mobile robot model (Test B) was tested in two steps: (1) a test for the motion, and (2) a test for the LIDAR. Robot motion was tested by moving the digital model in straight lines, curves, and sharp turns, and then comparing the differences in trajectories with recordings of the real robot. This test uncovered an issue where the simulated robot’s turn point was located at the front wheels (red line in Figure 10a), which is offset from the one in the real robot. This was corrected by inserting a game object in the correct spot (blue line in Figure 10a) and setting it as the turn point. Figure 10b shows a comparison test of one robot with the original turn point (red robot) and another one with the new turn point (blue robot). Both robots started at the same origin and rotated 45 degrees to the left. The results in Figure 10 show that the adjustment made the model’s turn more realistic. This adjustment was tested with repeated motions to verify its accuracy and usefulness for simulation.

The simulated LIDAR was tested by comparing its object detection with the one shown by the web interface offered by Reeman robot vendor. This test compared the two sensors standing still at several locations and verified that the digital one matched the specifications of the real one. The performance of the LIDAR was found to closely resemble what was shown in the web interface. The tests for the MATLAB simulators focused on optimizing the performance of the applications. These showed that the data-sampling loop and analysis processes were performing more slowly than expected. These slowdowns were addressed by adjusting the applications data usage.

3.2. Data Communication Methods

This section focuses on analyzing the implementations for each method and discussing their performance in the two stages of tests. The first test analyzed the basic integration, reliability, and flexibility of the four methods. The second test implemented the best-performing methods into the robot simulators and analyzed how they handle large datasets, fast update frequency, and performance over time.

The first test showed that the four communication methods had straightforward integrations with Unity and MATLAB. Some of the methods had issues in reliability and flexibility. TCP/IP had two reliability issues: lack of bidirectional communication and messages only being sent in one string. The first issue was fixed by creating two server–client pairs, one with Unity as the server and MATLAB as the client, and the other with MATLAB as the server and Unity as the client. The second issue was addressed by adding text markers to the string and using these to detect the necessary information. MQTT also showed reliability issues where the software would not receive data because their timings were not synchronized. This was fixed by adding a JSON file storage node to hold the received data until the other software called it. SHM lacks flexibility because the DLL requires the user to recompile and reimport it after each modification.

The results of the first test showed that all the four methods could efficiently exchange data between the two applications. However, SHM and TCP/IP presented critical flaws during testing. The DLL in SHM cannot be adjusted rapidly, which impacts the developer’s ability to rapidly prototype their simulator. For TCP/IP, the additional computational power and time required to parse the message strings made it an unviable option. On the other hand, Firebase and MQTT performed well in all facets of testing after their issues were addressed. Both methods store data in a JSON format, which enables efficient data parsing, and both allow for efficient modification to their applications. Therefore, MQTT and Firebase were selected for the second round of testing.

The follow-up test results on Firebase and MQTT showed that the Unity applications with continuous data streaming would become unstable in the consumer-grade computer. Figure 11 shows that the Reeman application with Firebase would randomly disconnect from the server. This occurred because Unity could not properly stream 726 data points per message with its variable streaming frequency of 600 Hz to 800 Hz. This caused the user to stream an average of 139.28 Megabytes (MB) worth of messages every minute and approximately 6 GB of data every hour to Firebase (

Table 2. Data usage analytics for streaming at 600 Hz.

Statistics per Minute	Data Usage
Maximum	163.61 MB
Average	139.28 MB
Minimum	0 MB
Data sent int time interval	5.936 GB

). This excessive data usage was deemed the primary cause of the instability in the simulator (Figure 11). This was resolved by modulating the streaming frequency and message size which created the excessive volume of data. Furthermore, Firebase allocates a limited amount of free storage to the user per month which, if exceeded, requires the user to pay a fee for each GB over the exceeded allocation.

Message size was modulated by reducing the LIDAR ray count from 240 to 135. This reduced the streamed data points per message to 415 and kept one ray per degree of the LIDAR. The feasible streaming frequency was determined by testing and comparing the following frequencies: 200 Hz, 100 Hz, 50 Hz, 20 Hz, and 10 Hz. The frequencies were tested using a two-hour trial that analyzed data usage, stability, and update rate for the slower speeds. A programming command was used in the Unity C# scripts to make sure that the data transmission remained at the given frequencies. The command creates a function that is updated at a set rate that is independent from the rest of the application. This makes sure that the application never updates at a faster or slower rate than it should be. This solution was implemented with all the frequencies analyzed during this process.

Due to the Firebase data usage constraints, it was defined that a frequency will be considered acceptable if it streams less than 1 GB of data per hour. Furthermore, to verify their usefulness, frequencies should maintain stability throughout the two-hour trial. This final parameter is crucial because an instability due to hardware issues can create further synchronization complications. This issue occurred during the test shown in Figure 11, where the device performing the simulation had to be restarted. The results of each frequency, which can be observed in Figure 12, are discussed in the section below.

• 200 Hz: The test results showed that applications with 200 Hz streaming became unstable in the second hour of testing. Despite data streaming remaining stable (Figure 12), the application showed error messages and experienced slowdowns. The application also saw exceedingly high data usage, with 5.7 GB of data streamed during the one hour of stability.
• 100 Hz: The results in Figure 12 show the application streaming data at 100 Hz briefly disconnected from the network. This can be attributed to the application briefly becoming unstable due to stutters caused by message parsing.

  Table 3. Statistics for all the analyzed frequencies.

  Data Usage	10 Hz	20 Hz	50 Hz	100 Hz	200 Hz
  Maximum per minute (MB)	6.31	12.75	27.07	63.8	108.63
  Average per minute (MB)	4.7922	10.0590	24.2209	49.8041	97.0666
  Minimum per minute (MB)	5.52	7.49	21.36	0	47.26
  Data sent in 1 h (GB)	0.288	0.604	1.458	3.019	5.791
  Data sent in 2 h (GB)	0.575	1.207	2.906	5.976	11.648

  shows that the application streamed an average of 3 GB of data per hour, which was deemed higher than acceptable for the application.
• 50 Hz: The results showed 50 Hz streaming completed the two-hour trial with no stability issues. These also showed that it used 1.5 GB of data per hour (Table 3), which is considered outside of the acceptable parameters.
• 20 Hz: The test results showed that streaming at 20 Hz had consistent data streams (Figure 12) at a low data usage, with a maximum of 12 MB per minute and an average of 10 MB per minute (Table 3). The results also showed that 20 Hz provides sufficient updates per minute with the mapping application to create an acceptable map.
• 10 Hz: The results show that 10 Hz has the best data usage at 5 MB streamed per minute (Figure 12). The results also show that streaming at 10 Hz on real-time applications leads to insufficient information for effective data processing.

Due to their instability with high volumes of data, it was determined that 200 Hz and 100 Hz streaming would not be viable. Additionally, neither applications could process messages at that speed, leading to lost data. Streaming below 100 Hz showed improved stability and acceptable data usage. Furthermore, 50 Hz had high data usage for the Reeman simulator but it is useful for simulators with smaller datasets. The Meca500 simulator performs well with this dataset because its dataset contains fifteen points. In addition, 20 Hz was deemed suitable for the application due to its acceptable data usage, lack of instability after multiple trials, and real-time responsiveness. Additionally, 10 Hz was slow for real-time applications but has uses for applications where real-time communication is not necessary. These results indicate that 50 to 10 Hz is the ideal range to connect Unity and MATLAB. The streaming frequency for each application will change based on the complexity of the application. The chosen frequency should minimize data usage while maintaining normal operation of the application. If a frequency above 100 Hz were used, it is recommended to use MQTT or some other local method with a high-end computer that can handle the fast computations.

3.3. Photon Unity Engine

This section will discuss the details of the PUN implementation, the improvements made to them, and the user experience for the network clients. The initial PUN implementations for the simulators were completed by following the procedure outlined in the literature [32] and tested using two network clients. The PUN server utilized can handle up to 20 concurrent users and 60 GB of traffic per month. The results showed that network objects in both simulators would lose synchronization for non-owner users and the object movement would be delayed for them. This delay was mitigated using the latency compensation computation in Equation (8).

$$\mathrm{p}\mathrm{o}\mathrm{s}=\mathrm{v}\mathrm{e}\mathrm{l}\ast \left|\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}-\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{v}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\right|$$

(8)

Equation (8) calculates which is the rate at which the object’s velocity is sped up or slowed down for the affected users. This equation multiplies the current velocity (vel) of the object with the absolute value of the time difference. This difference is obtained by subtracting the current time (currTime) and when the instruction was sent (prevTime).

The test in the Meca500 simulator showed that the positional sliders from non-owner users would transmit their unique values to the robot. This would cause the model to perform an erroneous motion. The fix for this was to disable the sliders of non-owner users and to add the request ownership protocol, which re-enables the sliders of the user. The Reeman simulator showed that if a user took over ownership of the robot, the position would reset to the one the new owner last recorded instead of continuing the movement from the new position. This was fixed by implementing a system that would override the last stored position from that user with the last stored position from the previous owner.

The stability of the PUN simulator was tested using 5 of the 20 possible concurrent users during a 2 h period. During testing, the server performance remained stable with no slowdowns occurring while the number of users increased. The game experience remained the same and the monitoring graph showed that all the users remained connected. The evaluation showed that all users had a 100 ms delay when interacting with non-owned network objects. This delay was expected to occur and did not hinder user experience during testing. This test generated minimal traffic on the PUN server because all users utilized the same object. The PUN server did not show major increase in data utilization while all five users were connected to the server. This showed that there was no time limit for these applications but one could occur if a large number of network objects were to be utilized.

The connection stability test showed that the prevention methods worked as intended. When a user disconnected from the server, ownership of game objects was transferred without causing issues to the application. Users that disconnected from the server would reconnect as inexperienced users, causing them to lose their status in the application. Future iterations of this study will focus on restoring users to their status before disconnecting and mitigating delay/desynchronization.

3.4. Performance Test

A core aspect of the proposed framework for this research is verifying that it can be accessible to users with diverse computer configurations. This was tested with the two computers configurations, as mentioned in Section 3. The load from simulators was analyzed while the applications were active. The test was evaluated under two diverse criteria, one focusing on the standalone communication methods, and the other on the simulator applications.

The tests MQTT and Firebase showed that they had negligible resource usage on both computers. MQTT through Node-Red was shown to only use 0.1–0.4% of the CPU and maximum 65 MB/s of RAM (Figure 13). Firebase did not use any hardware components, as shown in Figure 13; the only impact it had on the computer was using up to 1.1 MBPS of networking. This is attributed to Firebase being a web server with no local services needed.

The results of the performance test in

Table 4. How many resources each Unity and MATLAB application used on both computers.

		Unity Editor-Meca500	Unity Editor-Reeman	MATLAB-Meca500	MATLAB-Reeman
Consumer-Grade Computer	Min CPU	30.1%	24.9%	1.5%	27.9
	Max CPU	35.7%	27.1%	3.8%	37.3
	Min RAM	753.3 MB	539.7 MB	1.63 GB	1.711 GB
	Max RAM	763.1 MB	744.5 MB	1.66 GB	1.751 GB
	Min GPU	43.9%	45%	N/A	N/A
	Max GPU	56.8%	52%	N/A	N/A
Professional-Grade computer	Min CPU	27%	22.5%	1.4%	36.9%
	Max CPU	30.1%	25.6%	2.8%	43.1%
	Min RAM	1.314 GB	1.316 GB	1.752 GB	1.814 GB
	Max RAM	1.314 GB	1.316 GB	1.761 GB	1.837 GB
	Min GPU	10.6%	7.1%	N/A	N/A
	Max GPU	32%	24.2%	N/A	N/A

show the impact the Unity and MATLAB applications had on the computers. The Unity editor used to run the applications generated a moderate-to-high load on the CPU, GPU, and RAM on both systems. The CPU usage is higher than ideal but still acceptable for current computers. RAM usage scaled up in each system based on their available resources, as greater usage allows the application to run smoother. The GPU usage had high variance based on the visual complexity of the application. The Meca500 application shows a much higher usage because it had to render the six robot joints, which was a more complex task than that of the mobile robot from the Reeman simulator. All of these metrics stayed in what is deemed acceptable for the application in both computers considering their specifications.

For systems with lower computational power, the Unity editor simulation environment can be run by using the Unity development build instead. This is a lighter version of the application that runs preset scenarios. To modify the built application, users must go back to the Unity editor, make the desired changes, and rebuild it. This ensures that the application remains lightweight. This application uses 4–5% of CPU, 145 to 167 MB of RAM, and 5% of the GPU (Figure 14). This low usage in contrast with the unity editor is due to the simulator running without the overhead from the Unity editor.

The two MATLAB applications show the impact that varying computational complexities can have on the computers. These tests were run using a myriad of datasets to analyze how the application performed under different loads. As seen in Table 4, the Meca500 application did not create a heavy load due to its small dataset used and the more manageable calculation process. In contrast to this, the Reeman application had high resource utilization because of the frequent sampling of the large dataset generated by the LIDAR and the calculations required to process the data. This high usage was handled well by both computers, but it raised concerns about the viability of simulation with Unity and MATLAB simultaneously for less powerful computers. This can impact users such as students or schools who may not have access to higher-capacity systems but still want to execute large simulation models. This issue was solved by using Firebase to offload the computations to a more powerful remote computer. To accomplish this, the user must load the Unity and MATLAB applications on separate computers and then connect them to the Firebase server. The test results (Figure 15) show that this solution is viable for multi-device environments, such as classrooms, where one computer is dedicated to computations and the rest performs the simulations.

The results in this section show that the applications in the proposed framework can be modified to fit the needs of the user’s computer specifications. This allows users with varying degrees of computer configurations to effectively access the simulators and interact with the digital robots.

3.5. Integration Test Results

The next section focuses on the PUN integration with the robot simulators using MQTT and Firebase. The simulators were tested on how PUN interacted with the communication methods. The results in

Table 5. Benefits and disadvantages of integrating Firebase and MQTT with PUN.

Protocol	Benefits	Drawbacks
MQTT	Local communication. Lightweight process. Faster communication. No added latency from network. Users can perform analysis independently.	Requires all connected PCs to run MATLAB. Additional strain on all computers.
Firebase	Only one computer needs MATLAB. Users with weak hardware can join simulation. Synchronized data for all users. Little additional programing.	Data streams cause additional strain on network. Users must take turns sending data. Limited Firebase data usage. Possible latency in communication.
MQTT and Firebase	Offers flexibility to the user. Supports diverse range of users. No interference between protocols.	Increased development complexity.

show the benefits from Firebase and MQTT, and how they can be used together to serve a diverse set of users.

Table 5 shows that MQTT provides a better option for users who wish to perform all analysis locally, and Firebase works best for users who wish to share their data or need to offload computations. As a local process, MQTT requires all instances of the PUN simulators to run MATLAB and perform their own data analysis. This is useful for developers who wish to handle all computations locally and keep data private. As a web-based server, Firebase allows the PUN simulators to operate with only one instance of performing the data analysis in MATLAB. This can be utilized in classroom or industrial demos where only the instructors need to perform the computations and the remaining users can just visualize the scenario. This comparison highlights two important drawbacks for Firebase, the limited data usage and users having to take turns to use the service. These issues will be explored in the future by implementing an alternative web-based communication method. Future work will also focus on interfacing additional software with the framework. This will focus on adding ROS2 and Gazebo support to the Unity framework so their simulation tools can be used in the multi-user environment. Incorporating these and improving remote computations will improve the accessibility of the framework.

4. Conclusions

A multi-user robotic simulation framework connecting Unity to external applications is developed and evaluated in this research. The two simulation scenarios show that Unity has the capability to manage complex simulation scenarios. The addition of external software through messaging applications shows how the strengths of third-party software can be leveraged to improve the framework. Testing of the messaging methods showed that supporting MQTT and Firebase allowed the simulator to operate in a larger number of devices. The multi-user servers were shown to create a remote collaborative digital environment that can be leveraged by several applications. These factors created a flexible framework that can be adapted to varying simulation complexities on several types of computer configurations.

The results here demonstrate that Unity can be used in a robotic simulation framework for complex kinematics. Future efforts will aim to create a robotics simulation hub where users can expand their understanding of robotics and have immersive collaborative experiences. The framework will be expanded to support existing robotic simulators such as ROS2 and other programming languages such as Python. For the extension of this work, it will be necessary to first define the requirements for their desired simulation. The established parameters should be used to create the Unity and data analysis applications. These should be tested to determine their computational needs and if the current computer can handle them. If the system is shown to be insufficient, users can reduce the complexity of simulation or offload parts of the process to a different system. The multiplayer service may be changed to a solution which will provide smoother connections and better options to handle users’ desynchronization. Firebase may be replaced with a different web server with increased data streaming limits. Finally, Unity will be compared with existing simulators such as Gazebo and RoboDK on how they handle visual robot representation and physics simulation.