Virtual Teleoperation System for Mobile Manipulator Robots Focused on Object Transport and Manipulation

Abstract

This work describes the development of a tool for the teleoperation of robots. The tool is developed in a virtual environment using the Unity graphics engine. For the development of the application, a kinematic model and a dynamic model of a mobile manipulator are used. The mobile manipulator robot consists of an omnidirectional platform and an anthropomorphic robotic arm with 4 degrees of freedom (4DOF). The model is essential to emulate the movements of the robot and to facilitate the immersion in the virtual environment. In addition, the control algorithms are established and developed in MATLAB 2020 software, which improves the acquisition of knowledge to teleoperate robots and execute tasks of manipulation and transport of objects. This methodology offers a cheaper and safer alternative to real physical systems, as it reduces both the costs and risks associated with using a real robot for training.

1. Introduction

Service robotics focuses on the development of robotic systems capable of performing a wide range of beneficial tasks in non-industrial environments, including industrial automation applications, with a special focus on interacting with humans to improve their quality of life [1]. With the most recent technological advances, attention has turned not only to meeting human needs, but also to improving comfort and human interaction in various areas of daily life, from the home to hospitals and public spaces [2,3]. Service robotics comprises two main groups, professional and personal use according to the IFR [4]. Among these groups are a number of approaches such as: (i) Transport and logistics:Service robots for logistics fulfil areas of transport, handling, packaging, sorting and delivery of goods in offices, airports, post offices, among others [5]; (ii) Cleaning: where robotic systems are developed for the cleaning of large or crowded areas, most of the commercially available home service robots have been developed with respect to requirements posed exclusively by a specific domestic task such as floor cleaning [2]; (iii) Medicine: providing assistance in surgeries, rehabilitation and patient monitoring, as well as patient transport and orientation tasks, which represents up to 46% of hospital expenses. In addition, virtual AI-based psychotherapeutic devices are used in the field of mental health and for the treatment of diseases that require prolonged observation of the patient that can be managed through online monitoring [6,7]; (iv) Hospitality: focused on entertainment and improving the quality of the human experience; several hotels are even using robots to communicate with guests or perform hotel operations [8]; finally, (v) Agriculture: robotic systems are being developed for outdoor cultivation, improving productivity and reducing farmers’ working hours [9]. Resulting in the need to develop more complex robotic systems.

Due to the complexity of the applications and environments in which service robots must be deployed, the need has arisen to create systems with diverse morphologies and the combination of multiple robots to address a variety of tasks [10,11]. However, the acquisition of highly complex systems entails considerable investment, either for their implementation or study. This is further complicated when considering the need to train individuals in the handling of these robots, in addition to requiring the presence of trained supervisors to ensure their correct use and learning by the users. Currently, the most recent technological advances have opened new alternatives in the field of service robotics, facilitating the learning process through innovative approaches, such as virtual environments, programming with LEGO kits and pedagogical systems, with a STEM educational approach focused on service and educational robotics [12], such educational systems allow me to give a hint in the operation of robotics and programming, but their acquisition cost is quite considerable, in addition to its operation in the presence of people or other objects not considered, can become damaged or cause damage. Another disadvantage is that these educational systems cannot handle large weight loads or perform precision tasks, which generate great limitations [13,14].

To facilitate the learning process in situations that seek to replicate reality, various technological tools are available, with simulators being one of the most prominent in virtual robotics education [15,16]. Within this broad set of tools, virtual environments stand out, allowing the creation of immersive environments that simulate real situations [17,18]. These environments benefit from the use of devices such as helmets, gloves, sensors and controls, which are now widely available at the simulation level [19]. This technology has made it possible to develop virtual reality experiences within these environments, which, in addition to reducing costs compared to the use of real equipment and associated resources, provides a level of fail-safety in the learning process that cannot be achieved in a real physical environment [17].

In this context, the present work proposes the development of a teaching-learning system based on virtual environments in the Unity 3D graphics engine using an omnidirectional robot with an anthropomorphic arm controlled by means of MATLAB 2020 software of MathWorks through a communication channel. In order to facilitate the understanding of the various applications of service robotics and the complexity of robot morphologies. This is achieved by considering the mathematical modeling of the robot in question and a haptic device that allows to train users to perform autonomous and teleoperated tasks in an accessible way in an immersive virtual environment which presents different challenges for user understanding [20].

Finally, this article is organized in sections to address these aspects in a structured way. Section 2 presents the methodology needed to carry out this process, with a focus on educational robotics. Section 3 focuses on the mathematical basis required for the modeling and control of the robot. Section 4 is devoted to the development of the virtual environment. Section 5 presents the results obtained, and Section 6 contains the conclusions derived from this study.

2. Methodology and Process Conceptualisation

The challenge in service robotics lies in operating in unstructured environments, which are highly unpredictable and difficult to manage. Creating a completely unstructured system is complicated by the nature of these environments. Therefore, it is essential to have a thorough understanding of the real world so that the assumed foundations are sufficiently flexible and adaptable to the variability of the environment. Although the current development of robotic systems has advanced, it is still far from achieving effective operation in completely unstructured environments with fully autonomous control, therefore, a semi-autonomous control is established through the use of teleoperation of robotic systems, in this context, it is proposed the following work methodology to address this issue.

As can be seen in the scheme above (Figure 1), It starts from service robotics present in the real world and contextualize it using knowledge gained through experience and theoretical background. This allows us to develop a mental model that describes the operation of an omnidirectional robot and provides an understanding of its behavior. Subsequently, we build a formal model based on physical and geometrical principles, resulting in the mathematical model of the mobile manipulator robot contemplated by the mobile platform and the robotic arm, therefore, the model of each of these robots is considered. In addition, depending on the control objective, the independent model of each robot can be used, or a unified model can be established to control the end-effector of the whole robot. In response to the demands of the environment, a robotic process is created that combines the two previously defined systems for the purpose of evaluating the behavior, limitations and constraints of the robotic system.

Simultaneously, it creates an environment that adapts to real-world demands and leverages contextualized human knowledge. This environment is designed to host multiple tasks or objectives that require system intervention. Subsequently, we carry out the virtualization of this defined environment through the use of CAD software and graphics engines, with the purpose of familiarizing the user and facilitating clarity in the learning process. A schematic of the conceptualisation process involved is presented in the Figure 2.

Therefore, the virtual system designed in the Unity graphic engine includes a mobile manipulator robot, previously modelled in CAD software, with an environment composed of two predefined scenes, based on external resources of the real robot and for the simulation scenarios. The CAD software used is Solid Works 2019.Through a formal mathematical model, which incorporates the kinematics and dynamics of the robotic system, the animations of the robot’s movements are generated by means of scripts integrated in the graphic engine. In addition, control algorithms are developed in MATLAB, which allow the execution of the necessary actions to accomplish the tasks desired by the user. These actions are commanded through the Novint Falcon haptic device, which is connected to the control software. The communication between the control software and the graphics engine is done through a Dynamic-Link Libraries (DLL). Where, MATLAB software sends the control actions to Unity and receives the information from the robot. On the other hand, control actions can also be transmitted to a real robot, using a TCP/IP wireless communication protocol, which allows virtual tasks to be replicated in a physical environment, making it possible to evaluate the controller to remotely control the robot experimentally and also design an autonomous controller for the robot to perform a defined task.

In this way, a virtual environment is created with predefined goals to be achieved by the robotic system driven by a human operator. It can also be used to evaluate autonomous control algorithms for the execution of defined tasks.

3. Modeling and Control

A mobile manipulator robot refers to robots that consist of a robotic arm (also known as a manipulator) mounted on a mobile platform or vehicle. These robots enable tasks to be carried out that require the capability to navigate and manipulate. These systems are characterized by high redundancy.

3.1. Kinematic Model

The kinematics determines the characteristics of motion in the plane or in space of a robot, the kinematics of robots are described here.

3.1.1. Omnidirectional Mobile Robot

The position of the mobile robot is defined in the fixed reference space defined by $\eta \left(t\right)={\left[{\eta }_x\left(t\right),{\eta }_y\left(t\right),\psi \left(t\right)\right]}^T$. The kinematic configuration of the omnidirectional robot can be observed in Figure 3.

Now, it is necessary to obtain the model of the mobile robot as a function of its manoeuvrability velocities. The differential kinematics of the omnidirectional robot is represented by:

$$\begin{array}{c}{\dot{\eta }}_x=u_{l}cos\left(\psi \right)-u_{m}sin\left(\psi \right)\hfill \\ {\dot{\eta }}_y=u_{l}sin\left(\psi \right)+u_{m}cos\left(\psi \right)\hfill \\ {\dot{\eta }}_{\psi }=\omega \hfill \end{array}$$

(1)

written in matrix form (1) results:

$$\dot{\eta }\left(t\right)={\mathbf{J}}_{\mathbf{P}}\left(\psi \right)\mathbf{u}\left(t\right)$$

(2)

with $\dot{\eta }\left(t\right)={\left[{\dot{\eta }}_x,{\dot{\eta }}_y,\dot{\psi }\right]}^T$ which represents the velocity vector of the point of interest with respect to the inertial reference system $\left\{R\right\}$; ${\mathbf{J}}_{\mathbf{P}}\left(\psi \right)\in {\Re }^{3\times 3}$ is the Jacobian matrix that represents the motion characteristics of the omnidirectional robot; and $\mathbf{u}\left(t\right)\in {\Re }^3$ is the manoeuvrability vector of the omnidirectional robot with respect to $\left\{R_P\right\}$ [21].

3.1.2. Robotic Arm

The position of the robotic arm in the fixed workspace $\left\{R_{\mathrm{B}}\right\}$ is defined in terms of its independent articulations, so that it $\mathbf{h}\left(t\right)=f\left({\mathbf{q}}_{\mathbf{a}}\right)$, where the vector of independent coordinates is ${\mathbf{q}}_{\mathbf{a}}\left(t\right)=\left[\begin{array}{cccc}{q}_{a1}& q_{a2}& \dots & q_{a{n}_a}\end{array}\right]$. The configuration of the robotic arm can be observed in Figure 4.

Now, taking the fixed reference system $\left\{R_{\mathrm{B}}\right\}$; and the number of independent arm joints is four, i.e., a 4DOF anthropomorphic robotic arm, (see Figure 4). The position and orientation of the robotic arm is determined:

$$\mathbf{h}\left(t\right)=\left\{\begin{array}{c}{h}_r\left(t\right)=l_2{S}_{q2}{C}_{q1}+l_3{S}_{q2q3}{C}_{q1}+l_4{S}_{q2q3q4}{C}_{q1}\hfill \\ h_s\left(t\right)=l_2{S}_{q2}{S}_{q1}+l_3{S}_{q2q3}{S}_{q1}+l_4{S}_{q2q3q4}{S}_{q1}\hfill \\ h_t\left(t\right)=l_1+l_2{C}_{q2}+l_3{C}_{q2q3}+l_4{C}_{q2q3q4}\hfill \end{array}\right.$$

(3)

where, the expressions in (3) represent: $S_a=sin\left(a\right)$; $C_a=cos\left(a\right)$; $S_{ab..n}=sin(a+b+\dots +n)$; and $C_{ab..n}=cos(a+b+\dots +n)$.

Now, the model of the arm as a function of its velocities is obtained through the partial derivative of $f\left({\mathbf{q}}_{\mathbf{a}}\right)$ in relation to ${\mathbf{q}}_{\mathbf{a}}$, where is obtained the differential kinematics of the robot described in matrix form as:

$$\dot{\mathbf{h}}\left(t\right)={\mathbf{J}}_{\mathbf{a}}\left({\mathbf{q}}_{\mathbf{a}}\left(t\right)\right)\dot{{\mathbf{q}}_{\mathbf{a}}}\left(t\right)$$

(4)

where, $\dot{{\mathbf{q}}_{\mathbf{a}}}$ is the joint velocity vector of the robotic arm joints; $\dot{\mathbf{h}}\left(t\right)={\left[{\dot{h}}_x\left(t\right),{\dot{h}}_y\left(t\right),{\dot{h}}_z\left(t\right)\right]}^T$ is the vector of arm end-effector velocities in workspace; and ${\mathbf{J}}_{\mathbf{a}}\left({\mathbf{q}}_{\mathbf{a}}\right)\in {\Re }^{3\times 4}$ is the Jacobian of the robotic arm that transforms the manoeuvring velocities of the arm to the velocities of the end-effector.

3.1.3. Omnidirectional Mobile Manipulator

The mobile manipulator robot considered in this work consists of a robotic arm located on a mobile robot with omnidirectional traction. The configuration of the mobile manipulator robot is known when the position and orientation of all its points with respect to an inertial reference system is known $R\left(X,Y,Z\right)$. The kinematic configuration of the robot can be seen in Figure 5.

The direct kinematics of the mobile manipulator robot defines the position and location of the end-effector. $\xi \left(t\right)$ as a function of the configuration of the robotic arm and the mobile robot with omnidirectional traction $\xi \left(t\right)=f\left({\mathbf{q}}_{\mathbf{p}},{\mathbf{q}}_{\mathbf{a}}\right)$;

$$\xi \left(t\right)=\eta \left(t\right)+R_z\left(\psi \right)\left[\begin{array}{c}a\\ b\\ 0\end{array}\right]+{\mathbf{h}}_{\mathbf{a}}\left({\mathbf{q}}_{\mathbf{a}},\psi \right)$$

(5)

where, $\eta \left(t\right)={\left[{\eta }_x\left(t\right),{\eta }_y\left(t\right),{\eta }_P\left(t\right)\right]}^T$ represents the position of the omnirectional robot relative to the inertial reference frame $R\left(X,Y,Z\right)$; ${\eta }_x\left(t\right)$ y ${\eta }_y\left(t\right)$ are the positions on the axes $X,Y$, while ${\eta }_P$ represents the height of the mobile robot with respect to Z; a y b represent the distance of the robotic arm location from the moving reference system located at the center of gravity of the omnidirectional robot $\left\{R_P\right\}$; $R_z\left(\psi \right)$ represents the rotation matrix with respect to the Z-axis of the reference system $R\left(X,Y,Z\right)$; and $\mathbf{h}\left(\mathbf{q},\psi \right)$ considers the position of the robotic arm with respect to the position and orientation of the omnidirectional robot, i.e.,

$${\mathbf{h}}_{\mathbf{a}}\left({\mathbf{q}}_{\mathbf{a}},\psi \right)=\left[\begin{array}{c}{C}_{q1\psi }\left(l_2{S}_{q2}+l_3{S}_{q2q3}+l_4{S}_{q2q3q4}\right)\\ S_{q1\psi }\left(l_2{S}_{q}2+l_3{S}_{q2q3}+l_4{S}_{q2q3q4}\right)\\ {\eta }_P+l_1+l_2{C}_{q2}+l_3{C}_{q2q3}+l_4{C}_{q2q3q4}\end{array}\right]$$

(6)

Considering Equations (6) and the position of the mobile robot in (5), the direct kinematics of the mobile manipulator robot can be represented as:

$$\xi \left(t\right)=\left\{\begin{array}{c}{\xi }_x\left(t\right)={\eta }_x+a{C}_{\psi }-b{S}_{\psi }+C_{q1\psi }\left(l_2{S}_{q2}+l_3{S}_{q2q3}+l_4{S}_{q2q3q4}\right)\hfill \\ {\xi }_y\left(t\right)={\eta }_y+a{S}_{\psi }+b{C}_{\psi }+S_{q1\psi }\left(l_2{S}_{q2}+l_3{S}_{q2q3}+l_4{S}_{q2q3q4}\right)\hfill \\ {\xi }_z\left(t\right)={\eta }_P+l_1+l_2{C}_{q2}+l_3{C}_{q2q3}+l_4{C}_{q2q3q4}\hfill \end{array}\right.$$

(7)

Finally, the differential kinematic model of the mobile manipulator robot establishes the derivative of the end-effector location as a function of the derivative of the configuration of the robotic arm and the omnidirectional robot, i.e., $\dot{\xi }\left(t\right)=\frac{d}{dt}\xi \left(t\right)$. As a result, it follows that:

$$\dot{\xi }\left(t\right)=\left\{\begin{array}{c}{\dot{\xi }}_x=u_l{C}_{\psi }-u_m{S}_{\psi }-a{S}_{\psi }-b{C}_{\psi }-\left(l_2{S}_{q1\psi }{C}_{q2}+l_3{S}_{q1\psi }{C}_{q2q3}+l_4{S}_{q1\psi }{C}_{q2q3q4}\right){\dot{q}}_1+\dots \hfill \\ \left(l_2{C}_{q1\psi }{S}_{q2}+l_3{C}_{q1\psi }{S}_{q2q3}+l_4{C}_{q1}{S}_{q2q3q4}\right){\dot{q}}_2+\left(l_4{C}_{q1\psi }{S}_{q2q3q4}\right){\dot{q}}_4+\dots \hfill \\ \left(l_3{C}_{q1\psi }{S}_{q2q3}+l_4{C}_{q1\psi }{S}_{q2q3q4}\right){\dot{q}}_3\hfill \\ {\dot{\xi }}_y=u_l{S}_{\psi }+u_m{C}_{\psi }+a{C}_{\psi }-b{S}_{\psi }-\left(l_2{C}_{q1\psi }{C}_{q2}+l_3{C}_{q1\psi }{C}_{q2q3}+l_4{C}_{q1\psi }{C}_{q2q3q4}\right){\dot{q}}_1+\dots \hfill \\ \left(l_2{S}_{q1\psi }{S}_{q2}+l_3{S}_{q1\psi }{S}_{q2q3}+l_4{S}_{q1}{S}_{q2q3q4}\right){\dot{q}}_2++\left(l_4{S}_{q1\psi }{S}_{q2q3q4}\right){\dot{q}}_4+\dots \hfill \\ \left(l_3{S}_{q1\psi }{S}_{q2q3}+l_4{S}_{q1\psi }{S}_{q2q3q4}\right){\dot{q}}_3\hfill \\ {\dot{\xi }}_z=\left(l_2{C}_{q2}+l_3{C}_{q2q3}+l_{\dot{4}}{C}_{2q3q4}\right){\dot{q}}_2+\left(l_3{C}_{q2q3}+l_4{C}_{q2q3q4}\right){\dot{q}}_3+\left(l_4{C}_{q2q3q4}\right){\dot{q}}_4\hfill \end{array}\right.$$

(8)

The differential kinematic model (8) of the robot is written in matrix form:

$$\dot{\xi }\left(t\right)=\mathbf{J}\left(\mathbf{q}\right)\mathbf{v}\left(t\right)$$

(9)

where, $\mathbf{J}\left(\mathbf{q}\right)\in {\Re }^{3\times 7}$ represents the Jacobin matrix of the mobile manipulator robot, which defines a linear mapping between the joint velocity vector defined by $\mathbf{v}\left(t\right)=\left[u_l,u_m,\omega ,{\dot{q}}_1,{\dot{q}}_2,\right.$$\left.{\dot{q}}_3,{\dot{q}}_4\right]\in R^7$ and the end-effector velocity vector $\dot{\xi }\left(t\right)=\left[{\dot{\xi }}_x,{\dot{\xi }}_y,{\dot{\xi }}_z\right]\in {\Re }^3$. The Jacobian matrix $\mathbf{J}\left(\mathbf{q}\right)\in {\Re }^{3\times 7}$ is a non-square matrix where the number of rows is less than the number of columns; therefore, the mobile manipulator robot is a redundant system.

3.2. Control

The teleoperation scheme allows to command any robot over long distances, i.e., it allows to operate the robot to perform a specific task by the human operator (Figure 6). The defined task usually consists of several operation processes, and these commonly include three stages: (i) approach; (ii) manipulation; and (iii) return process. When the mission is initiated, the human operator must move the mobile robot manipulator close to the object, the robotic arm should be considered to be within its working space. Once the mobile robot reaches the proximity of the target object, the human operator switches the locomotion mode to manipulation mode (control of the robotic arm). When the manipulation mission is completed, the human operator switches the manipulation mode back to locomotion mode and the mobile manipulator robot shall return to a safe area. To accomplish this task, the teleoperation scheme proposed for this work is presented in Figure 7.

3.2.1. Local Site

The proposed scheme allows the generation of reference commands for each robot depending on the task selected by the human operator (h). The reference vectors generated by the human operator are ${\dot{\eta }}_{\mathbf{ref}}^h\left(t\right)$, ${\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(t\right)$ y ${\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)$; such that, each reference is dependent on the haptic device, i.e., $\left({\dot{\eta }}_{\mathbf{ref}}^h\left(t\right),{\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(t\right),{\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)\right)=f\left({\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)\right)$, where ${\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)$ is the direct reference that the haptic device generates at each instant (t) when excited by the human operator (h), and is defined by:

$${\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)=\left[\begin{array}{c}{\dot{p}}_x^h\left(t\right)\\ {\dot{p}}_y^h\left(t\right)\\ {\dot{p}}_z^h\left(t\right)\end{array}\right]$$

(10)

3.2.2. Locomotion

The human operator can select: (i) to control only the omnidirectional mobile robot, (ii) to control the total mobile manipulator robot. For both locomotion cases the commands generated by the human operator through the haptic device shall correspond to velocity commands.

(i) Omnidirectional Robot (Reference Generation). This mode allows only the omnidirectional robot to be controlled. For this, the human operator (h) commands the point of interest of the omnidirectional robot by the haptic device. Therefore, the velocity reference for the robot is obtained through the differential kinematics of the mobile robot (2), such that:

$${\dot{\eta }}_{\mathbf{ref}}^h={\mathbf{J}}_{\mathbf{P}}\left(\psi \right){\mathbf{u}}_{\mathbf{ref}}^h\left(t\right)$$

(11)

Remark 1. 

In this case the reference ${\dot{p}}_z^h$ is not translated to linear velocity but to rotational velocity of the robot since the omnidirectional robot only moves in the $XY$-plane and rotates around the ${\dot{p}}_z^h$-axis of the fixed reference system.

(ii) Mobile Manipulator Robot (Reference Generation). Now, in this mode the operator commands the entire mobile manipulator robot, i.e., he controls the end-effector by the haptic device. Considering the human operator’s reference vector ${\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)\in R^3$, the reference generator considers the mapping of these velocities generated by the human operator as follows:

$${\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)=\mathbf{R}\left(\psi ,q_1\right){\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)$$

(12)

where, ${\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)\in {\Re }^3$ is the reference velocity of the end-effector of the mobile manipulator robot; and $\mathbf{R}\left(\psi ,q_1\right):{\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)\to {\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)$ is a matrix of that maps the velocities generated by (h) at the reference velocities of the operational point of the robot.

3.2.3. Manipulation

(i) Robotic Arm (Reference Generator). In this mode the human operator (h) controls only the robotic arm; for which, the commands generated by (h) through the haptic device ${\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)$ will correspond to position commands of the end-effector of the robotic arm. ${\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(t\right)\in {\Re }^3$. The same expression of (12) is considered, however, taking into account that for this the reference velocities generated by (h) are limited according to the manipulability of the robotic arm, so that:

$$\begin{array}{cc}{\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(t\right)={\mathbf{s}}_{\mathbf{at}}{\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)& {\mathbf{s}}_{\mathbf{at}}=\left\{\begin{array}{cc}{\mathbf{0}}_{3\times 3}& w\le \lambda \\ {\mathbf{I}}_{3\times 3}& w>\lambda \end{array}\right.\end{array}$$

(13)

where $w=\sqrt{det\left({\mathbf{J}}_{\mathbf{a}}{{\mathbf{J}}_{\mathbf{a}}}^T\right)}$, is the manipulability index [22]; and $\lambda \in {\Re }^{+}$ is a real value we define the minimum manipulability for the robotic arm such that $\lambda >0$.

3.2.4. Remote Site

The remote station considers the implementation of control algorithms that consider the signals generated by the human operator (h), through the haptic device. ${\dot{\mathbf{p}}}_{\mathbf{ref}}^h\left(t\right)$, and convert them into signals for the mobile manipulator robot, according to the mode of operation selected by the operator (h). Therefore, each controller generates the vector of manoeuvrability velocities of the robot ${\mathbf{v}}_{\mathbf{ref}}\left(t\right)$. It is defined by:

$${\mathbf{v}}_{\mathbf{ref}}\left(t\right)=\left[\begin{array}{c}{\mathbf{u}}_{\mathbf{ref}}\left(t\right)\\ {\mathbf{q}}_{{\mathbf{a}}_{\mathbf{ref}}}\left(t\right)\end{array}\right]={\left[\begin{array}{ccccccc}{u}_{l_{ref}}& u_{m_{ref}}& {\omega }_{ref}& {\dot{q}}_{1_{ref}}& {\dot{q}}_{2_{ref}}& {\dot{q}}_{3_{ref}}& {\dot{q}}_{4_{ref}}\end{array}\right]}^T$$

(14)

(i) Kinematic Control of Omnidirectional Mobile Robot. The kinematic model of Equation (2) is considered, therefore, the proposed control law for the omnidirectional robot is:

$${\mathbf{u}}_{\mathbf{ref}}\left(t\right)={\mathbf{J}}_{\mathbf{P}}^{-1}\left(\psi \right)\left({\dot{\eta }}_{\mathbf{ref}}^h\left(t\right)+{\mathbf{K}}_{\mathbf{p}}tanh\left(\dot{\tilde{\eta }}\left(t\right)\right)\right)$$

(15)

where, ${\dot{\eta }}_{\mathbf{ref}}^h\left(t\right)$ represents the desired velocity generated by the human operator (h) through the haptic device; ${\mathbf{J}}_{\mathbf{P}}^{-1}\left(\psi \right)$ represents the Jacobian matrix of the behaviour of the omnidirectional robot; ${\mathbf{K}}_{\mathbf{p}}\in {\Re }^{3\times 3}$ is the gain matrix that weights the velocity errors of the local controller of the omnidirectional robot; and ${\mathbf{u}}_{\mathbf{ref}}\left(t\right)$ represent the manoeuvrability velocities of the omnidirectional robot. The motion control error of the omnidirectional robot can be defined as the difference between the commands generated by human operator and the motion velocity of the omnidirectional robot, i.e:

$$\dot{\tilde{\eta }}\left(t\right)={\dot{\eta }}_{\mathbf{ref}}^h\left(t\right)-\dot{\eta }\left(t\right)$$

(16)

As the operator generates commands only for the mobile robot, then the robotic arm should remain in a fixed position, therefore a control law is defined to keep the robotic arm static during the locomotion task. Let be the desired position of the arm end-effector with respect to the reference frame of the mobile robot ${\mathbf{h}}_{\mathbf{d}}\left(t\right)={\left[h_{xd},h_{xd},h_{xd}\right]}^T$. The controller that generates the reference velocities of the robotic arm joints is given by:

$${\dot{\mathbf{q}}}_{{\mathbf{a}}_{\mathbf{ref}}}\left(t\right)={{\mathbf{J}}_{\mathbf{a}}}^{\#}{\mathbf{K}}_{\mathbf{a}}tanh\left({\tilde{\mathbf{h}}}_{\mathbf{a}}\left(t\right)\right)+\left(\mathbf{I}-{{\mathbf{J}}_{\mathbf{a}}}^{\#}{\mathbf{J}}_{\mathbf{a}}\right)\mathbf{H}tanh\left(\chi \right)$$

(17)

where, ${\mathbf{K}}_{\mathbf{a}}\in {\Re }^{3\times 3}$ is a positive diagonal matrix weighing the control error ${\tilde{\mathbf{h}}}_{\mathbf{a}}\left(t\right)={\mathbf{h}}_{\mathbf{d}}\left(t\right)-{\mathbf{h}}_{\mathbf{a}}\left(t\right)$; $\mathbf{H}\in {\Re }^{4\times 4}$ is a positive diagonal matrix weighing the error of the desired configuration of the robotic arm $\chi ={\left[{\tilde{q}}_1,{\tilde{q}}_3,{\tilde{q}}_3,{\tilde{q}}_4\right]}^T$. As the Jacobian of the arm has more columns than rows, the controller is based on secondary objectives as it has several degrees of freedom; and ${{\mathbf{J}}_{\mathbf{a}}}^{\#}$ is the pseudo-inverse matrix of the robotic arm.

(ii) Robotic Arm Kinematic Control. The position control of the end-effector position of the robotic arm, the kinematic model defined in Equation (4) is considered; therefore, the proposed control law is:

$${\dot{\mathbf{q}}}_{\mathbf{aref}}\left(t\right){=\mathbf{J}}_{\mathbf{a}}^{\#}\left({\dot{\mathbf{h}}}_{\mathbf{ref}}^h+{\mathbf{K}}_{\mathbf{a}}tanh\left({\tilde{\mathbf{h}}}_{\mathbf{a}}\left(t\right)\right)\right)+\left(\mathbf{I}-{\mathbf{J}}_{\mathbf{a}}^{\#}{\mathbf{J}}_{\mathbf{a}}\right)\mathbf{H}tanh\left(\chi \right)$$

(18)

where, ${\mathbf{J}}_{\mathbf{a}}^{\#}\left({\mathbf{q}}_{\mathbf{a}}\right){=\mathbf{J}}_{\mathbf{a}}^{\mathrm{T}}{\left({\mathbf{J}}_{\mathbf{a}}{\mathbf{J}}_{\mathbf{a}}^{\mathrm{T}}\right)}^{-1}$ is the pseudo-inverse matrix of the robotic arm; ${\mathbf{h}}_{\mathbf{ref}}^h\left(t\right)={\int }_{t_0}^{t_f}{\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(\tau \right)$ represents the desired position of the end-effector of the robotic arm; ${\dot{\mathbf{h}}}_{\mathbf{ref}}^h\left(t\right)$ is the desired velocity generated by the operator (h); ${\tilde{\mathbf{h}}}_{\mathbf{a}}\left(t\right)={\mathbf{h}}_{\mathbf{ref}}^h\left(t\right)-{\mathbf{h}}_{\mathbf{a}}\left(t\right)$ represents the error vector of the position of the robotic arm; ${\mathbf{h}}_{\mathbf{a}}\left(t\right)$ represents the actual position of the end-effector of the robotic arm. Finally, in this case, the omnidirectional mobile robot maintains its position, i.e., it does not make any movement ${\dot{\eta }}_{\mathbf{ref}}^h\left(t\right)={\mathbf{0}}_3$, therefore, the reference velocity in this case is:

$${\mathbf{u}}_{\mathbf{ref}}\left(t\right)={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^T$$

(19)

(iii) Kinematic Control of the Mobile Manipulator Robot. Similarly, the kinematic model of Equation (9) is considered, therefore, the proposed control law is:

$${\mathbf{v}}_{\mathbf{ref}}\left(t\right)={\mathbf{J}}^{\#}\left({\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)+{\mathbf{K}}_{\mathbf{1}}tanh\left(\dot{\tilde{\xi }}\left(t\right)\right)\right)+\left({\mathbf{I}}_{7\times 7}-{\mathbf{J}}^{\#}\mathbf{J}\right){\mathbf{K}}_{\mathbf{2}}tanh\left(\zeta \right)$$

(20)

where, ${\mathbf{v}}_{\mathbf{ref}}\left(t\right)\in {\Re }^7$ represent the manoeuvrability velocities of the mobile manipulator robot; ${\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)$ represents the desired velocity of the end-effector of the mobile manipulator robot generated by the human operator (h) through the haptic device; ${\mathbf{J}}^{\#}\left(\psi ,{\mathbf{q}}_a\right)={\mathbf{J}}^T{\left(\mathbf{J}{\mathbf{J}}^T\right)}^{-1}$ is the pseudo-inverse matrix of the mobile manipulator robot; ${\mathbf{K}}_{\mathbf{1}}\in {\Re }^3$; ${\mathbf{K}}_2\in {\Re }^{7\times 7}$ are positive diagonal matrices weighing the robot end-effector position control error and the secondary target control vector errors respectively. The motion control error of the mobile manipulator robot is defined as the difference between the commands generated by the human operator and the motion velocity of the end-effector of the mobile manipulator robot, such that:

$$\dot{\tilde{\xi }}\left(t\right)={\dot{\xi }}_{\mathbf{ref}}^h\left(t\right)-\xi \left(t\right)$$

(21)

For this work, the configuration of the robotic arm is considered as secondary objectives, therefore, the secondary objective vector is defined as:

$$\zeta \left(t\right)={\left[\begin{array}{ccccccc}0& 0& 0& q_{1d}-q_1& q_{2d}-q_2& q_{3d}-q_3& q_{4d}-q_4\end{array}\right]}^T$$

(22)

where, $q_{id}$ con $i=1,2,3,4$ represents the desired position of each joint of the robotic arm; and $q_i$ is the current position of each joint of the robotic arm.

3.3. Dynamic of the Robots

To emulate the behaviour of the robot with its dynamic characteristics. In this section the models of the omnidirectional robot and the robotic arm are defined. Therefore, the robot in the 3D virtual environment allows to replicate the real system behaviour as close to real as possible. With this we simulate in real time the physical performance of the robot. This methodology is a tool that provides a safe and ideal environment for the validation of control techniques in the development of prototypes and for educational purposes for handling robotic prototypes. Figure 8 shows the interpretation of the virtual robot.

3.3.1. Direct Dynamics of the Omnidirectional Robot

The dynamic model is a mathematical formulation that describes the dynamic behaviour of a system. It establishes the relationships between the robot’s joint coordinates or coordinates of interest, velocities, accelerations, forces and torques applied at its joints and the robot’s parameters, such as masses and moments of inertia. The model used in this work is a simplified [23] model, this model is represented in terms of reference velocities, because it is easier to apply velocities than torques, as in traditional robot models. Therefore, to emulate the dynamics of the robot, the direct dynamic model of the robot is used. For this purpose, the inverse dynamic model ${\mathbf{u}}_{\mathbf{ref}}^{\mathbf{d}}\left(t\right)=\mathbf{M}\left(\sigma \right)\dot{\mathbf{u}}\left(t\right)+\mathbf{C}\left(\sigma ,\omega \right)\mathbf{u}\left(t\right)$ is used. Now, the direct dynamic model used in this work is defined by:

$$\dot{\mathbf{u}}\left(t\right)={\mathbf{A}}_{\mathbf{P}}\left(\sigma \right){\mathbf{u}}_{\mathbf{ref}}\left(t\right)+{\mathbf{B}}_{\mathbf{P}}\left(\sigma ,\omega \right)\mathbf{u}\left(t\right)$$

(23)

where, ${\mathbf{u}}_{\mathbf{ref}}\left(t\right)\in {\Re }^3$ is the reference velocity vector of the omnidirectional robot; $\dot{\mathbf{u}}\left(t\right)={\left[\begin{array}{ccc}{\dot{u}}_l& {\dot{u}}_m& \dot{\omega }\end{array}\right]}^T\in {\Re }^3$ is the vector of accelerations of the omnidirectional robot obtained from the direct dynamics; $\mathbf{u}\left(t\right)\in {\Re }^3$ is the actual velocity vector of the robot; ${\mathbf{A}}_{\mathbf{P}}\left(\sigma \right)=\mathbf{M}{\left(\sigma \right)}^{-1}$; ${\mathbf{B}}_{\mathbf{P}}\left(\sigma ,\omega \right)=-\mathbf{M}{\left(\sigma \right)}^{-1}\mathbf{C}\left(\sigma ,\omega \right)$. The matrix $\mathbf{M}\left(\sigma \right)\in {\Re }^{3\times 3}$ is the mass matrix of the robot, is square and positive definite; and $\mathbf{C}\left(\sigma ,\omega \right)\in {\Re }^{3\times 3}$ is a quadratic matrix representing the centripetal and coriolis forces of the robot. The matrices are given by:

$$\begin{array}{cc}\mathbf{M}\left(\sigma \right)=diag\left([{\sigma }_1,{\sigma }_2,{\sigma }_3]\right)& \mathbf{C}\left(\sigma ,\omega \right)=\left[\begin{array}{ccc}{\sigma }_4& -\omega {\sigma }_5& 0\\ \omega {\sigma }_6& {\sigma }_7& 0\\ 0& 0& {\sigma }_8\end{array}\right]\end{array}$$

(24)

where, $\sigma ={\left[{\sigma }_1,{\sigma }_2,\dots {\delta }_8\right]}^T$ is the vector of the dynamic parameters of the omnidirectional robot.

3.3.2. Direct Dynamics of the Robotic Arm

Similar to the omnidirectional robot, the dynamic model used in this work is represented in terms of reference velocities and not in terms of forces and torques. That is to say, the structure of the dynamic model is given by ${\dot{\mathbf{q}}}_{{\mathbf{a}}_{\mathbf{ref}}}\left(t\right)={\mathbf{M}}_{\mathbf{a}}\left(\delta \right){\ddot{\mathbf{q}}}_{\mathbf{a}}\left(t\right)+{\mathbf{C}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right){\dot{\mathbf{q}}}_{\mathbf{a}}\left(t\right)+\mathbf{g}\left(\delta \right)$. Where, ${\ddot{\mathbf{q}}}_{\mathbf{a}}\left(t\right)={\left[{\ddot{q}}_1,{\ddot{q}}_2,{\ddot{q}}_3,{\ddot{q}}_4\right]}^T$ is the vector of accelerations of the robotic arm; ${\mathbf{M}}_{\mathbf{a}}\left(\delta \right)\in {\Re }^{4\times 4}$ is the mass and inertia matrix of the robotic arm; ${\mathbf{C}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)\in {\Re }^{4\times 4}$ is the matrix of centripetal and coriolis forces; y ${\mathbf{g}}_{\mathbf{a}}\left(\delta \right)\in {\Re }^4$ is the gravity vector.

$$\begin{array}{c}\begin{array}{cc}{\mathbf{M}}_{\mathbf{a}}\left(\delta \right)=\left[\begin{array}{cccc}{m}_{11}\left(\delta \right)& m_{12}\left(\delta \right)& m_{13}\left(\delta \right)& m_{14}\left(\delta \right)\\ m_{21}\left(\delta \right)& m_{22}\left(\delta \right)& m_{23}\left(\delta \right)& m_{24}\left(\delta \right)\\ m_{31}\left(\delta \right)& m_{32}\left(\delta \right)& m_{33}\left(\delta \right)& m_{34}\left(\delta \right)\\ m_{41}\left(\delta \right)& m_{42}\left(\delta \right)& m_{43}\left(\delta \right)& m_{44}\left(\delta \right)\end{array}\right]& {\mathbf{g}}_{\mathbf{a}}\left(\delta \right)=\left[\begin{array}{c}{g}_1\left(\delta \right)\\ g_2\left(\delta \right)\\ g_3\left(\delta \right)\\ g_4\left(\delta \right)\end{array}\right]\end{array}\hfill \\ {\mathbf{C}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)=\left[\begin{array}{cccc}{c}_{11}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{12}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{13}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{14}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)\\ c_{21}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{22}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{23}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{24}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)\\ c_{31}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{32}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{33}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{34}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)\\ c_{41}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{42}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{43}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)& c_{44}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)\end{array}\right]\hfill \end{array}$$

(25)

where, $\delta ={\left[{\delta }_1,{\delta }_2,\dots {\delta }_n\right]}^T$ is the vector of dynamic parameters of the robotic arm. Finally, to emulate the robotic arm, the direct dynamics defined by:

$${\ddot{\mathbf{q}}}_{\mathbf{a}}\left(t\right)={\mathbf{A}}_{\mathbf{a}}\left(\delta \right){\dot{\mathbf{q}}}_{{\mathbf{a}}_{\mathbf{ref}}}\left(t\right)+{\mathbf{B}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right){\dot{\mathbf{q}}}_{\mathbf{a}}\left(t\right)+{\mathbf{C}}_{\mathbf{a}}\left(\delta \right)$$

(26)

where, ${\mathbf{A}}_{\mathbf{a}}\left(\delta \right)={\mathbf{M}}_{\mathbf{a}}{\left(\delta \right)}^{-1}$; ${\mathbf{B}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)=-{\mathbf{M}}_{\mathbf{a}}{\left(\delta \right)}^{-1}{\mathbf{C}}_{\mathbf{a}}\left(\delta ,{\dot{\mathbf{q}}}_{\mathbf{a}}\right)$; y ${\mathbf{C}}_{\mathbf{a}}\left(\delta \right)=-{\mathbf{M}}_{\mathbf{a}}{\left(\delta \right)}^{-1}\mathbf{g}\left(\delta \right)$.

4. Virtual Environment

This section presents the development of an immersive environment, a 3D virtual environment is developed in the Unity 3D graphics engine to simulate the teleoperation tasks of a mobile manipulator robot in two completely different scenes. The virtual environment is focused on teaching and learning in the area of robotics, so it allows interaction with different users. To do this, in the first instance, the virtualisation of the 4 DOF robotic arm and the omnidirectional robot is carried out, to subsequently place them in 2 environments where the tasks demanded by the user are carried out through a means of communication between the operator and the virtual environment.

4.1. Virtualization of the Mobile Manipulator Robot

In order to carry out the digitalization process of the mobile manipulator robot, it is necessary to take external resources as a reference. As shown in the diagram above (Figure 9), a 4 DOF robot arm is used for the manipulation of an object in question, while a KUKA omnidirectional robot is used as a reference for the movement of the robot [24]. Then, using CAD software, a 3D solid object is modelled for each of the robots, with the necessary physics for their respective movements and constraints in order to develop the required animation. Once the CAD design of the two robots has been completed, it is exported to Autodesk 3DS-Max 2024 software, which allows the design to be rendered, dimension and rotation changes to be made, and exported in the format required for the graphics engine. In addition, the rotation axes of each joint of the robotic arm are configured and the centre of the axis of the mobile robot is established during the animation phase. At this stage, the movement of each joint and of the platform is checked to ensure that it matches the axes of movement of the real robots. Each component of the robot can then be exported separately to the graphics engine, as each part has its own animation and movement, determined by the kinematic and dynamic model through script programming.

4.2. Scenario Virtualization

For the development of each scenario, it is essential to identify the places that will be used for the creation of the environment. In this case, the project has focused on two scenes in which dangerous objects are handled. To do this, an exhaustive analysis of the objects that make up the scenarios in a real environment is carried out (view Figure 10).

For the development of the virtual environment, The use of prefabricated scenarios that provide a high degree of realism in the objects present in the real environment. This approach facilitates the creation of the virtual scenes and allows for a more efficient development process. Since the main objective of the project is the teaching and learning of service robotics and robot morphology. The use of these prefabricated designs ensures an accurate and functional representation of the different scenarios.

Once the prefabricated scenarios and objects have been selected, we proceed to render the scenes, adjusting to the scale and dimensions of the robot as necessary. For this purpose, 3DS Max software is used, which acts as a link between the prefabricated design and the Unity graphics engine.

Once the export and rendering of the prefabricated scenarios in the Unity graphic engine is completed, we proceed to add the attributes, materials and physical properties to each of the elements present. In this project, two main scenarios have been developed. The first is an environment dedicated to the handling of explosives, where a grenade has been placed near a machine; in this case, the object of interest is the bomb. The second scenario represents a laboratory for handling chemicals, with a flask filled with an explosive liquid as the main object, as shown in the graphic above (Figure 11).

The objective in both scenarios is to move the object of interest to a safe place, away from the risk area, using exclusively the movements of the mobile omnidirectional manipulator robot.

To enable navigation through the environment by moving the mobile manipulator robot, a visual interface has been developed and is shown in the figure (Figure 12). This interface presents the virtual environment from the perspective of the manipulator robot as the main screen, providing an overview of the operating area. Additionally, three auxiliary cameras have been incorporated to provide different perspectives crucial to the task. One of these auxiliary cameras provides a detailed view of the operating end, which is essential for precise manipulations. The other two cameras focus on the objects of interest in each scene, allowing a clear visualization of the target elements to be manipulated. This multi-camera setup ensures that the user has a complete understanding of the environment in order to monitor and control the robot, thus improving efficiency and accuracy in task execution. According to the type of task to be performed, whether it is locomotion or object manipulation by the mobile manipulator robot, the control law is selected by means of the Novint Falcon device, as described in Section 3.2. Additionally, for the movement of the end-effector, there are different types of sensors that help this movement, such as force, optical or piezo-resistive sensors, but for this application, by means of the haptic device, an on/off switching of the gripper is generated, which grips the object as long as it is close to the target.

Additionally, an interface developed in MATLAB software is used, which requests the necessary information to select the predetermined scene and set the duration of the simulation. This interface presents the following data:

• Execution Time
• User
• Scene.

This menu also displays the X, Y, Z coordinates obtained from the Falcon haptic device, as well as the hits and misses related to the manipulation of the objects of interest in the environment. Including Play, Stop, and Save buttons, in order to start or end the simulation.

As mentioned above, the goal of the teaching-learning system is to transport an object of interest to a safe location using the mobile manipulator robot. This safe place is located outside the two main scenes, and is where the dangerous object should be deposited. If the task is performed successfully, the menu increases the number of successes; otherwise, the number of failures is increased (view Figure 13). Otherwise, if the object of interest falls in an area outside the safe area or contacts any other part of the environment, such as walls, machines or other objects present in the scenes, the number of failures is increased.

4.3. Communication Chanel

Finally, once the virtualization of the scenarios and objects in the graphics engine has been completed and the control algorithms have been established in MATLAB, a bilateral communication channel known as Shared Memories is used to facilitate the exchange of information between the virtual environment and the MATLAB mathematical software, using a Dynamic Link Library (DLL) that creates a shared memory in RAM (SM) for the exchange of data between different programs [25].Through the SM, the control actions calculated in the target controller are integrated into the mathematical model of the robotic system. This model calculates the position and velocity outputs, which are then sent to the mathematical software, thus closing the control loop by providing feedback on the robot’s output states. The desired task can be selected by the haptic device, or in the case of using an autonomous control, the user can define the desired task to be performed, e.g., positioning, trajectory tracking or path following, where the computation of the control algorithms is performed by MATLAB mathematical software [21]. Subsequently, the references generated by the controllers are applied to the robot. The Figure 14 indicates the communication scheme used for bilateral communication.

5. Experimental Results

This section presents the results of the use of the virtual environment as a teaching-learning development tool in the field of educational robotics. In addition, a brief usability analysis of the tool on a given number of students is presented. The hardware used for the simulation of the tool is described as follows: Intel i7-7700HQ processor, CPU 2.80 GHz × 8 (Intel, Santa Clara, CA, USA) and Ge-Force GTX 1050 graphics (Nvidia Corporation, Santa Clara, CA, USA). The characteristics of such hardware is not strictly related to the performance of the tool. Nevertheless, for better animation, it is strongly recommended to include specialized graphics hardware. The implemented system for the teleopraticon is shown in Figure 15. Here, you can see the human operator with the haptic device teleoperated to the robot in the developed virtual environment. In [26] you can watch a video showing the operation of the developed application, where the scenarios and tasks developed by the user are shown.

When the objective of the task is accomplished, with each of the limitations presented by the movements of the robot and the environment, the menu shows a success in bringing the object of interest, in this case an explosive, to the safe place. Figure 16 shows how the object is transported to the safe place by the robot commanded by the human operator.

Otherwise, if due to a wrong movement by the user when controlling the mobile manipulator robot, and the object falls somewhere other than the safe zone, the environment presents an explosion animation and the menu indicates an increase in the number of failures due to poor execution of the program, thus resetting the object in its initial position, as shown in the next graphic (Figure 17).

Usability Evaluation

To identify the usability and relevance of the tool, a comprehensive evaluation of the didactic application of teaching and learning is performed. For this purpose, 20 students of the Electronics and Automation Engineering course of the University of the Armed Forces ESPE-L have been chosen to use the virtual environment. These students have basic knowledge in kinematic and dynamic modeling of robots, control systems and virtual reality, which allows them to correctly evaluate the tool. The survey consists of 10 questions focused on evaluating the functionality and intuitiveness of the developed tool. Student responses will be collected using a scale from 1 to 5, where 1 represents “Very difficult”, 2 “Difficult”, 3 “Intermediate”, 4 “Easy” and 5 “Very easy”. The list of questions proposed to the users is presented in

Table 1. List of questions.

Ord.	Questions
1	How easy was it for you to learn how to use our virtual environment tool?
2	How useful have you found our tool in improving your understanding of service robotics?
3	Do you find the user interface of our tool intuitive and easy to use?
4	Do virtual environments offer an adequate variety of resources (videos, images, etc.) to facilitate your learning?
5	Are the instructions provided in the tool clear and easy to follow?
6	Do you consider that the tool is interactive and allows you to actively participate in the learning process about the cinematic model of the robot?
7	Are you satisfied with the functionality and accuracy of the robot’s movements and rotations in the simulation?
8	Has the tool helped you better understand the concepts of movements and rotations of a mobile omnidirectional manipulator robot?
9	Do you consider that the virtual environment has provided you with a valuable educational experience compared to other learning methods?
10	Overall, would you recommend our simulation tool in Unity to other students or teachers interested in learning about robotics?

.

The results of the usability evaluation of the system are presented in the graphic above (Figure 18). For the first question, a mean of 3.25 with a standard deviation of 0.79 was obtained. The second question showed a mean of 3.35 with a deviation of 0.88. The third question registered a mean of 3.40 with a deviation of 0.68, while the fourth question reached a mean of 3.80 with a deviation of 0.52. The fifth question had a mean of 3.40 and a standard deviation of 0.88. For the sixth question, the mean was 3.30 with a deviation of 0.69, and for the seventh question, a mean of 3.30 with a deviation of 0.8 was obtained. The eighth question recorded a mean of 3.65 with a standard deviation of 0.67. The ninth question obtained a mean of 3.35 with a deviation of 0.75, while the tenth question yielded a mean of 4.00 with a deviation of 0.85.

The usability evaluation data, illustrated in the figure above (Figure 18), provide detailed insight into how users perceive the teaching-learning tool. The error bars in the figure show the variability of responses among students, indicating the range within which the responses of the rest of the population fall.

Overall, the results suggest a positive evaluation of the virtual environment tool. The relatively small standard deviations reveal a favorable perception of the users regarding the usability of the system. Thus indicating an ease of use of the tool.

In addition, a traditional theoretical teaching method on kinematic and dynamic modeling of robots was chosen to be used with a different group of 20 additional students, but with similar characteristics to those who used the tool. Subsequently, a comparative evaluation of the knowledge acquired in both groups was carried out. The results indicated that the students who used the teaching-learning system demonstrated greater knowledge about motions, rotations, and the kinematic and dynamic modeling of robots, compared to the group that received conventional theoretical instruction.

6. Conclusions

The kinematic and dynamic modeling of the robot using virtual environments significantly increases the realism of the simulation tool. By replicating with greater precision the movements and behaviors of the robot in question, resulting in the simulation being remarkably close to the behavior of the real robot. This fidelity allows users to interact with the virtual environment in a way that almost identically mimics the operation of the robot in real situations. This results in a much more effective and realistic learning experience, providing users with a more accurate understanding of how the robot behaves in a real situation.

The implementation of virtual environments facilitates the development of new learning tools, offering several significant advantages. First of all, these environments allow a much more accessible and economical system acquisition, by considerably reducing costs compared to the purchase of real systems. In addition, they eliminate the risks associated with the use of physical equipment, such as possible damage and the need to operate with caution. This results in a safe and hazard-free learning experience. Similarly, virtual environments do not require the presence of expert operators for training, which simplifies the process and results in greater accessibility for users. In summary, virtual environments reduce costs, risks and also optimize the teaching-learning process, offering a practical and efficient solution compared to real systems.

The use of the Falcon haptic device in the development of the teaching-learning system proves to be an excellent tool to enhance interaction between the environment and the users. The ability to provide feedback allows students to experience increased realism, manipulation and control of robots, which significantly enriches the learning process. Overcoming the limitations inherent in traditional theoretical teaching methods. In short, the use of the haptic device in combination with virtual environments offers an innovative and highly effective approach to robotics education, benefiting both students and instructors.

The development of a teaching-learning system using virtual environments, presented in this work, facilitates the acquisition of new knowledge on robot modeling and control for users with basic or limited knowledge on the subject. This approach is more effective compared to theoretical, classical or conventional teaching methods, which present limitations for both the instructor and the students in the learning process.