Mobile Visual Servoing Based Control of a Complex Autonomous System Assisting a Manufacturing Technology on a Mechatronics Line

Abstract

The main contribution of this paper is the modeling and control for a complex autonomous system (CAS). It is equipped with a visual sensor to operate precision positioning in a technology executed on a laboratory mechatronics line. The technology allows the retrieval of workpieces which do not completely pass the quality test. Another objective of this paper is the implementation of an assisting technology for a laboratory processing/reprocessing mechatronics line (P/RML) containing four workstations, assisted by the following components: a complex autonomous system that consists of an autonomous robotic system (ARS), a wheeled mobile robot (WMR) PeopleBot, a robotic manipulator (RM) Cyton 1500 with seven degrees of freedom (7 DOF), and a mobile visual servoing system (MVS) with a Logitech camera as visual sensor used in the process of picking, transporting and placing the workpieces. The purpose of the MVS is to increase the precision of the RM by utilizing the look and move principle, since the initial and final positions of the CAS can slightly deviate from their trajectory, thus increasing the possibility of errors to appear during the process of catching and releasing the pieces. If the processed piece did not pass the quality test and has been rendered as defective, it is retrieved from the last station of the P/RML and transported to the first station for reprocessing. The control of the WMR is done using the trajectory-tracking sliding-mode control (TTSMC). The RM control is based on inverse kinematics model, and the MVS control is implemented with the image moments method.

1. Introduction

The main contribution of this paper is the modeling and control of a complex autonomous system (CAS) equipped with a robotic manipulator (RM) and a mobile visual sensor (MVS) to achieve precise positioning in a processing technology on a laboratory mechatronics line (ML). The presented technology allows the recovery of products, through reprocessing, if they partially satisfy the quality requirements. The mechatronics line handles the full manufacturing process of a workpiece, and it has buffering, handling, processing (drilling, boring), sorting, and quality testing facilities. Due to the fact that this technology allows the product retrieval operation for reprocessing, the mechatronics line will be further referred to as P/RML.

This paper presents the integration of multiple subsystems for a laboratory mechatronic line: the P/RML is served by a CAS consisting of an ARS equipped with a 7 DOF RM Cyton 1500 and uses an eye in hand type Mobile Visual Servoing (MVS) located on the end-effector. Task planning, simulation, monitoring, and real-time control are some of the tools used to fulfill the objectives of this study, like workpiece recovery for reprocessing or the increase of the automation level o for the P/RML so that a human operator is no longer needed after the program is started. Moreover, the CAS improves the reusability feature of the production line [1,2,3,4,5].

The manufacturing line is characterized by the operations to be performed in the workstations using the monitoring and control systems, CAS, ARS, RM, MVS which give reparability and flexibility features for processing and reprocessing operations [4,5,6,7,8]. The structure of the P/RML highlights the system’s overall functions: the automatic processing of parts, transport, handling, storing and the automatic control of all the devices integrated in the system using the PLCs in different configurations: centralized or distributed. The processing/reprocessing tasks are the P/RML actions (drilling, reaming, extrusion, bending), operations which are executed in serial or parallel for manufacturing the workpiece until the final product meets the quality requirements. For this, the processing/reprocessing planning also requires complex operations to be performed such as performance measurement and evaluation, kinematics control, system planning, and resources management. For a P/RML assisted by CAS, a preparing strategy adapted to the characteristics of the system is more effective than the techniques used from domain-independent methods [6,7,8,9,10,11]. If the final workpiece does not pass the quality test, the task planner will provide the optimal sequence to reprocess the product.

Visual servoing systems are robust systems that enhance the accuracy and adaptability of the control architecture based on optical feedback [12]. The purpose of a servoing system is to close the movement control loop for a robotic arm. The robotic arm has to move from an initial point to a destination point. The aim of a MVS is to provide information, based on a visual sensor, regarding the actual position. Based on this information, the robot control is able to improve the interaction with the working environment. The features extracted from the images represent the actual position and also the input for the control architecture. The control architectures corresponding to the servoing systems are divided into three categories:

• Position Based Visual Servoing (PBVS) [13,14,15];
• Image Based Visual Servoing (IBVS) [13,16,17];
• Hybrid Visual Servoing (HVS) [4,11,16].

An important factor for a MVS is the visual features selection, which is related to the speed, accuracy, and performance that control the movement of a RM, taking into consideration the degrees of freedom as shown in [10,18,19].

Other elements of originality and contributions are concentrated in the following areas: assigning, planning, and synchronization tasks of P/RML assisted by CAS, RMs, and MVS, designing the architecture of the entire system to allow flexible manufacturing, multifunctionality, communication, synchronization of signals from sensors, distributed control and image processing for precise positioning eye in hand MVS and real-time control, improving the level of automation and increasing the efficiency of the manufacturing lines using the CAS.

The article is structured as follows:

Section 2 is dedicated to the hardware structure and the functional features of the system:

• In Section 2.1, the hardware structure of the P/RML assisted by CAS is presented;
• In Section 2.3, an eye in hand MVS technology is presented;
• In Section 2.3, modeling, and control of MVS, based on the moments of the image, are presented;
• In Section 2.4, assumptions and task planning are presented.
• In Section 3, direct and inverse kinematics model of RM Cyton 1500 are presented;
• In Section 4, trajectory-tracking sliding-mode control of ARS PeopleBot is presented;
• In Section 5, some results regarding real-time control of P/RML assisted by CAS are laid out;
• In Section 6, some final remarks and conclusions are presented.

2. P/RML Assisted by the CAS Equipped with RM and MVS

2.1. The Hardware Architecture of the P/RML Assisted by CAS

The P/RML Festo MPS 200, presented in Figure 1, is a configurable laboratory mechatronic system consisting of four workstations. The four workstations are controlled by an individual Siemens S7 300 PLC that operates in different stages: buffering, handling, processing, and sorting:

• The first station triggers the color tests and then transports the pieces to the warehouse corresponding to the color;
• The second station is where the pieces are buffered for further processing;
• The third station performs drilling and boring operations;
• The fourth station is where the pieces are stored and organized by their colors.

RM Cyton is used for recovering the pieces which are then transported by the ARS PeopleBot. The RM is equipped with a camera used for the eye in hand subsystem mounted on the end-effector. All the elements are controlled by a computer, the camera via USB connection, ARS PeopleBot, and RM Cyton by Wi-Fi with the help of an Advanced Robotic Interface for Applications (ARIA) package [20,21,22,23]. The combined subsystems are presented in Figure 2.

2.2. Eye in Hand MVS

An eye in hand MVS type refers to a system where the visual sensor is placed on the last joint of the RM and the camera movements also affect RM movements. Another type of MVS commonly used is eye to hand, where, contrary to eye in hand, the sensor is fixed in a position relative to the work environment, where both the RM and the workpieces are observed. For image-based visual servoing (IBVS), 2D image information is being used for an estimation of the desired motion of the robot. Standard tasks such as detection, tracking and positioning are realized by minimizing the error between the features extracted from the current image and the visual features of the desired image [13].

In the case of IBVS architecture, the visual sensor that extracts information about the work environment can be either eye in hand implementation, where the movement of the robot induces the movement of the camera, or eye to hand, where the RM and its motions can be examined from a fixed point. A common method used in object detection, classification, and shape identification is called the moments of the image. This method is based on features extraction from a 2D intensity distribution of the image, together with information about the orientation and coordinates of the gravity center. In Figure 3 is presented the complete CAS’s transport trajectory, from the last to the first P/RML workstation, from the detection and grabbing to the placing of the workpiece.

2.3. Modelling and Control of MVS Based on the Moments of the Image Method

The structure of the eye in hand MVS has the following components: a 7-DOF RM, a visual sensor, and an image-based controller presented in Figure 4.

One of the most important parts of this design implementation, is that the image-based controller, needs information about the environment of the system to minimize error between the current configuration of the visual features, $f$, and the desired configuration $f^{*}$. For modelling the open loop servoing system, the fixed parts components of the RM and the visual sensor must be analyzed separately. The signal related with the input control of the RM is ${\upsilon }_c^{*}$, and it represents the reference speed of the camera with the structure: ${\upsilon }_c^{*}={\left({\upsilon }^{*},{\omega }^{*}\right)}^T$ where ${\upsilon }^{*}={\left[{\upsilon }_x^{*},{\upsilon }_y^{*},{\upsilon }_z^{*}\right]}^T$ and ${\omega }^{*}={\left[{\omega }_x^{*},{\omega }_y^{*},{\omega }_z^{*}\right]}^T$ are defined as the linear and angular speed, respectively. The signal, ${\upsilon }_c^{*}$ is expressed in Cartesian space and must be transformed to be applied to the RM.

The posture is defined by the ${\upsilon }_c^{*}$ integration and is noted with $s={\left[s_1,s_2,s_3,s_4,s_5,s_6\right]}^T$ defining the robot Jacobian as follows:

$$J_r=\left[\partial s_i/\partial q_j\right],\hspace{0.17em}i,j=1,\dots ,7,$$

(1)

where $q_{i,j}=1,\dots ,$ 7 represents the RM’s joints’ states. The transformation of ${\upsilon }_c^{*}$ from Cartesian space to robotic joint space is done with $J_r^{-1}$ and the interaction matrix. The moments $m_{ij}$ are a set of visual features with the analytic form for the time variation, ${\dot{m}}_{ij}$, resultant to the moments of order $\left(i+j\right)$, differing depending on the speed of the visual sensor ${\upsilon }_c^{*}$ corresponding to the following equation:

$${\dot{m}}_{ij}=L_{m_{ij}}{\upsilon }_c,$$

(2)

where $L_{m_{ij}}=\left[m_{{\upsilon }_x}{m}_{{\upsilon }_y}{m}_{{\upsilon }_z}{m}_{{\omega }_x}{m}_{{\omega }_y}{m}_{{\omega }_z}\right]$ is the interaction matrix.

The interaction matrix associated with a set of image moments $f={\left[x_n,y_n,a_n,\tau ,\xi ,\alpha \right]}^T$ for n points is processed in this manner [12,13,24,25,26]

$$L_f=\left[\begin{array}{cccccc}-1& 0& 0& a_n{e}_{11}& -a_n\left(1+e_{12}\right)& y_n\\ 0& -1& 0& a_n\left(1+e_{21}\right)& -a_n{e}_{11}& -x_n\\ 0& 0& -1& -e_{31}& e_{32}& 0\\ 0& 0& 0& {\tau }_{{\omega }_x}& {\tau }_{{\omega }_y}& 0\\ 0& 0& 0& {\xi }_{{\omega }_x}& {\xi }_{{\omega }_y}& 0\\ 0& 0& 0& {\alpha }_{{\omega }_x}& {\alpha }_{{\omega }_y}& -1\end{array}\right].$$

(3)

The analytical form of the parameters from (3) can be found in [27].

2.4. Assumptions and Task Planning

The processing/reprocessing tasks can be simplified into a sequence of actions combined in parallel with workpieces positioning and transportation along the cells of the mechatronics line. Shown in Figure 5 is shown the task planning in the form of a block diagram with the actions done by the CAS when a workpiece has failed the quality test. Next, the workpiece is transported from the storage station (last cell) to the buffer station (first cell). The red lines represent the reprocessing tasks done if the piece has been considered faulty, and the black lines represent the normal processing assignments when an initial workpiece has been supplied.

Because the P/RML on its basis is a flexible industrial production line, it is designed to easily adapt to changes in the type and quantity of the product and it has a high range of products that it can provide. Through reprocessing, the faulty products can be retrieved and reworked to the necessary quality standard. Since the technology utilized by the P/RML can be altered by aspects such as process modes, type of finished workpiece, and operations times, some assumptions are needed for the entire system to work as intended:

Assumption 1.

The manufacturing technology processes the workpieces in pipeline mode;

Assumption 2.

The first station is provided with one piece at a time for processing;

Assumption 3.

The initial conditions and parameters of the technology are known, such as quantity of pieces and task duration;

Assumption 4.

Only one type of piece can be processed or reprocessed with different colors since the proposed processing operations are specific to a type of product;

Assumption 5.

The number of the workstations involved in processing/reprocessing by the P/RML is previously known and unchanged;

Assumption 6.

The workstations of the P/RML have a linear distribution in the following order: buffer, handling, processing, and storing;

Assumption 7.

The processing/reprocessing tasks are executed on the same workstation and the pieces can be processed simultaneously in different stages;

Assumption 8.

A red workpiece will mean that the quality test is not passed and reprocessing is needed;

Assumption 9.

In the storage, the first level from the top is utilized for rejected workpieces;

Assumption 10.

One CAS assists the P/RML, which is used for picking up, transporting, and releasing the workpieces;

Assumption 11.

One eye in hand visual sensor is mounted on the RM Cyton;

Assumption 12.

To avoid conflict, priority is given to the workpiece for reprocessing;

Assumption 13.

The technology includes two quality tests, which detect between a scrap workpiece and one recoverable by reprocessing.

3. Direct and Inverse Kinematics Model of RM Cyton 1500

RM Cyton 1500 offers a robust and precise manipulation for a wide variety of applications. It has been designed to simulate the structure of a human arm; the shoulder has three joints, the elbow has one joint, and the wrist has three joints. All the joints described make up the 7 DOF of the RM. Angle limits of the RM are shown in

Table 1. 7-DOF RM Cyton 1500.

Joint	Lower Limit (in Angles)	Upper Limit (in Angles)
Shoulder Roll	−150°	150°
Shoulder Pitch	−105°	105°
Shoulder Yaw	−105°	105°
Elbow Pitch	−105°	105°
Wrist Yaw	−105°	105°
Wrist Pitch	−105°	105°
Wrist Roll	−150°	150°

.

For testing and simulating the RM, kinematic modeling needs to be performed, with the main objective being the study of RM’s mechanical parts, the direct and the inverse kinematics.

The direct kinematics consist of finding the position of the end-effector in space by knowing the movements of the joints, for example $F\left({\theta }_1,{\theta }_2,\dots \dots ,{\theta }_n\right)=\left[x,y,z,R\right]$, and inverse kinematics consist of determining the value of every joint by knowing the position of the end effector and its orientation, $F\left(x,y,z,R\right)=\left[{\theta }_1,\dots \dots \dots ,{\theta }_n\right]$.

A RM is composed, in general, of three joints, each defined by one or more degrees. In the case of RM Cyton 1500, it is made of:

• One shoulder type joint, characterized by three angles;
• One elbow type joint, characterized by one angle;
• One wrist type joint, characterized by three angles.

Presented in Figure 6 is a compact diagram block of kinematic modeling [28].

The configuration of the RM and its joints is presented in Figure 7 [29].

One convention that is used for the selection of reference frames, especially in robotics applications, is represented by the Deavit&Hartenberg (D&H) convention, illustrated in Figure 8 [30], and parameters of the RM Cyton 1500, presented in

Table 2. RM Cyton 1500 D&H parameters.

i	α	a	d	P
1	90	0	d1	θ1
2	0	a2	0	θ2
3	90	0	d3	θ3
4	0	a4	0	θ4
5	90	0	d5	θ5
6	0	a5	0	θ6
7	−90	a6	0	θ7

.

To determine the direct kinematic model, a fixed coordinate system with an index of 0 for the shoulder type joints and, for the other joints, a system with an index of i, i = 1 … 7 have been placed. In this convention, every homogeneous transformation $A_i$ is represented as a product of four basic transformations, where the four variables ${\theta }_i, a_i, d_i, {\alpha }_i$ are the parameters associated with every link and joint.

$$A_i=R_{z,{\theta }_i}Tran{s}_{z,d_i}Tran{s}_{x,a_i}{R}_{x,{\alpha }_i},$$

(4)

$$A_i=\left[\begin{array}{cccc}c{\theta }_i& -s{\theta }_i& 0& 0\\ s{\theta }_i& c{\theta }_i& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& d_i\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& a_i\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& c{\alpha }_i& -s{\alpha }_i& 0\\ 0& s{\alpha }_i& c{\alpha }_i& d_i\\ 0& 0& 0& 1\end{array}\right]$$

$$A_i=\left[\begin{array}{cccc}c{\theta }_i& -s{\theta }_{i}c{\alpha }_i& s{\theta }_{i}s{\alpha }_i& a_{i}c{\theta }_i\\ s{\theta }_i& c{\theta }_{i}c{\alpha }_i& -c{\theta }_{i}s{\alpha }_i& a_{i}s{\theta }_i\\ 0& s{\alpha }_i& c{\alpha }_i& d_i\\ 0& 0& 0& 1\end{array}\right].$$

(5)

With the help of the D&H table the $A_I$ matrices have been determined for every DOF of the RM:

$$\begin{gathered} A_1=\left[\begin{array}{cccc}{c}_1& 0& s_1& 0\\ s_1& 0& -c_1& 0\\ 0& 1& 0& d_1\\ 0& 0& 0& 1\end{array}\right], A_2=\left[\begin{array}{cccc}{c}_2& -s_2& 0& {\alpha }_2{c}_2\\ s_2& c_2& 0& {\alpha }_2{s}_2\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right], \\ {A}_3=\left[\begin{array}{cccc}{c}_3& 0& s_3& 0\\ s_3& 0& -c_3& 0\\ 0& 1& 0& d_3\\ 0& 0& 0& 1\end{array}\right], A_4=\left[\begin{array}{cccc}{c}_4& -s_4& 0& {\alpha }_4{c}_4\\ s_4& c_4& 0& {\alpha }_4{s}_4\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right], A_5=\left[\begin{array}{cccc}{c}_5& 0& s_5& 0\\ s_5& 0& -c_5& 0\\ 0& 1& 0& d_5\\ 0& 0& 0& 1\end{array}\right], \\ {A}_6=\left[\begin{array}{cccc}{c}_6& -s_6& 0& {\alpha }_6{c}_6\\ s_6& c_6& 0& {\alpha }_6{s}_6\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right], A_7=\left[\begin{array}{cccc}{c}_7& 0& s_7& {\alpha }_7{c}_7\\ s_7& 0& -c_7& {\alpha }_7{c}_7\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]. \end{gathered}$$

(6)

The direct kinematics model of the RM Cyton is determined by the decomposition of the seven matrices:

$$T_1^0=A_1,$$

(7)

$$T_7^0=A_1{A}_2{A}_3{A}_4{A}_5{A}_6{A}_7=\left[\begin{array}{cccc}{s}_x& n_x& a_x& d_x\\ s_y& n_y& a_y& d_y\\ s_z& n_z& a_z& d_z\\ 0& 0& 0& 1\end{array}\right]$$

(8)

Figure 9 illustrates an image representing all the joints and angles composing RM Cyton. The kinematic model obtained is utilized in the control structure of RM Cyton:

4. Trajectory Tracking Sliding-Mode Control of ARS PeopleBot

The ARS PeopleBot is controlled to follow a desired trajectory based on continuous time sliding-mode control. In TTSMC, the real ARS follows the trajectory of a virtual one, with the desired trajectory generated by the virtual ARS denoted with

$$q_d\left(t\right)={\left[\begin{array}{ccc}{x}_d& y_d& {\theta }_d\end{array}\right]}^T.$$

(9)

The kinematic model of the virtual ARS becomes

$$\{\begin{array}{c}\dot{x}=v_d\cos {\theta }_d\\ \dot{y}=v_d\sin {\theta }_d\\ \dot{\theta }={\omega }_d\end{array},$$

(10)

where $x_d$ and $y_d$ represent the cartesian coordinates of the geometric center, $v_d$ represents the linear speed, ${\theta }_d$ represents the orientation, and ${\omega }_d$ represents the angular speed. When the ARS follows the desired trajectory, on X and Y axis appear tracking and orientation errors:

$$\left[\begin{array}{c}{x}_e\\ y_e\\ {\theta }_e\end{array}\right]=\left[\begin{array}{ccc}\cos {\theta }_d& \sin {\theta }_d& 0\\ -\sin {\theta }_d& \cos {\theta }_d& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_r-x_d\\ y_r-y_d\\ {\theta }_r-{\theta }_d\end{array}\right].$$

(11)

The error dynamics are given by the following equations

$$\{\begin{array}{c}{\dot{x}}_e=-v_d+v_r\cos {\theta }_e+{\omega }_d{y}_e\\ {\dot{y}}_e=v_{r}sin{\theta }_e-{\omega }_d{x}_e\\ {\dot{\theta }}_e={\omega }_r-{\omega }_d\end{array}.$$

(12)

Since the ARS orientation is not perpendicular on the desired trajectory, it is assumed that $\left|{\theta }_e\right|<\mathsf{\pi }/2$. Given the errors’ positions (11) and the derivatives (12), the sliding surfaces are

$$\{\begin{array}{c}{s}_1={\dot{x}}_e+k_1{x}_e\\ s_2={\dot{y}}_e+k_2{y}_e+k_{0}sgn\left(y_e\right){\theta }_e\end{array}.$$

(13)

Presented in Figure 10 are the real and virtual ARS with absolute coordinates and orientation following a desired trajectory:

The parameters $k1, k2, k3$ are positive and constant. If s1 converges to zero, then $x_e$ becomes 0 too, and if $s_2$ tends to 0, then $\dot{y}=-k_2{y}_e+k_{0}sgn\left(y_e\right){\theta }_e$. The surface derivatives

$$\{\begin{array}{c}{\dot{s}}_1={\ddot{x}}_e+k_1{\dot{x}}_e,\\ {\dot{s}}_2={\ddot{y}}_e+k_2{\dot{y}}_e+k_{0}sgn\left(y_e\right){\dot{\theta }}_e\end{array},$$

(14)

are written in a compact form:

$$\dot{s}=-Qsgn\left(s\right)-Ps,$$

(15)

where

$$Q=\left[\begin{array}{cc}Q1& 0\\ 0& Q2\end{array}\right], P=\left[\begin{array}{cc}P1& 0\\ 0& P2\end{array}\right], Q\ge 0,P\ge 0,$$

(16)

$$s={\left[\begin{array}{cc}{s}_1& s_2\end{array}\right]}^T,sgn\left(s\right)={\left[\begin{array}{cc}sgn\left(s_1\right)& sgn\left(s_2\right)\end{array}\right]}^T.$$

(17)

Thus, from (11)–(14), the TTSMC law becomes as follows:

$${\dot{v}}_c=\frac{-Q_{1}sign\left(s_1\right)-P_1{s}_1-k_1{\dot{x}}_e-{\dot{\omega }}_d{y}_e-{\dot{\omega }}_d{\dot{y}}_e+v_r\dot{{\theta }_e+{\dot{v}}_d} }{\cos {\theta }_e},$$

(18)

$${\omega }_c=\frac{-Q_{2}sign\left(s_2\right)-P_2{s}_2-k_2{\dot{y}}_e-{\dot{v}}_r\sin {\theta }_e+{\dot{\omega }}_d{x}_e+{\omega }_d{\dot{x}}_e+{\omega }_d}{v_{r}cos{\theta }_e+k_{0}sgn\left(y_e\right)}$$

(19)

5. Real-Time Control of P/RML Assisted by CAS

The assisting technology for P/RML consists of one dynamic robotic system, CAS, used in picking, placing, and transporting the workpieces.

The process is based on 3 main control loops:

• Control loop for the P/RML with Siemens PLCs, programmed in Simatic Step 7;
• Control loop for the ARS PeopleBot based on TTSMC used in workpiece transportation;
• Control loop for the RM Cyton with eye in hand MVS for precise positioning for both picking and placing workpieces.

Presented in Figure 11 are the control loops connected to a single computer that runs the applications which control the P/RML, CAS and its mechanisms with the help of Simatic Step 7, Microsoft Visual Studio, the C programming language, Aria packages, and MATLAB.

Displayed in Figure 12 are a few pictures from real-time with the process of scanning the object, and, if it is detected, it is picked up by the gripper of the RM and transported with the CAS to the first workstation. The process with the main steps done in detecting the objects is presented in Figure 13. In Figure 13a is the conversion from the RGB (Red Green Blue) color model to the HSV (Hue Saturation Value) color model, as it is more robust to changes in light, since, during the experiments, there have been major effects with the type of light used, natural sunlight or laboratory light. Presented in Figure 13b is the detected object if the colors between the HSV limits have been met and the shape (in this case, a circle) has been found using the Ramer–Douglas–Peucker algorithm and Canny Edge detection, which are implemented by using the OpenCV libraries. Finally, in Figure 13c, the object is tracked if the conditions of both the color and shape have been met; the centroid is tracked by using the image moments method for its versatility and efficiency, as deviations appear only in a 2D space, the distance on the Z axis is constant, and the rotations of the object do not exist, since the workpiece always comes to the same place, only the CAS does not have a stable position and a MVS is needed [31].

Likewise, presented in Figure 14 are pictures with a similar process to the previous figure, this time for placing the workpieces rather than pick them up. The biggest difference between Figure 13 and Figure 15 consists of the fact that, in Figure 13, the object itself is detected while, in Figure 15, a reference point specific to the first workstation is detected, and the object is placed. Also, while in the first case the shape detected is a circle, in the second one the shape detected is a rectangle.

In Figure 16a, the complete 3D picking up trajectory is presented: the movement from the home position to the scanning, then above the object, picking it, and back to the home position, so that the CAS can transport it to the first workstation for reprocessing if it fails the quality test. Presented in Figure 16b is the 3D trajectory for placing the object.

Although the MVS starting and ending point are present in Figure 16a,b, they are hard to visualize, which is why Figure 17 is needed, in the case of picking the object, having the X and Z axis in Figure 17a and the Y on Figure 17b. The entire MVS process takes ~7 s and it is most apparent in the case of the Y axis, where there is a deviation that goes up to 10 mm from the desired position.

The time for placing the workpiece, based on MVS control, is about four s due to the fact that difference between the desired and actual features is much smaller than picking up. The X and Z trajectories over time are displayed in Figure 18a. In Figure 18b, only the Y axis trajectory is shown, since the distance is much smaller than the X and Y axis.

The PeopleBot trajectory has some deviations on the X and Y axis that can be observed in Figure 19a. Along the X axis, the error is near zero, while on the Y axis the final error is about 4 mm, as shown in Figure 19b.

The velocity profile of the ARS PeopleBot along the forward trajectory, during transportation of the workpiece, is presented in the Figure 20a, with a maximum of 0.12 m/s. The state transition of the workpiece is shown in Figure 20b. While the RM is active for picking and placing the workpiece, the CAS has zero velocity.

In Figure 21, the X and Y trajectory are displayed, so that the differences between the desired and real trajectories can be identified more easily.

6. Conclusions

The presented research is still in the experimental phase, the final step being a fully automated processing technology, allowing the recovery by reprocessing of a given volume of pieces. By using SIMATIC STEP 7, Visual C ++ platforms and MATLAB, real-time control structures have been implemented, allowing automatic processing/reprocessing, assisted by the CAS, ARS equipped with RM, and MVS. Compared to [4], where two autonomous robotic systems and two type visual servoing systems (eye to hand and eye in hand) have been used, in this paper only one CAS, with one type of MVS (eye in hand), assists P/RML. As a result, the reliability, flexibility, and robustness of the technology have been increased due to the uncertainties that might come from the sensors fails, environment conditions, disturbances in data transfer and communications. Although this is a technology that has been used at the level of a laboratory, it can be extended further to real industry, where high accuracy and positioning are needed. In the future, this technology is expected to be improved to be able to satisfy the standards of the Industry 4.0. Through this perspective, the presented manufacturing technology has a dual purpose, one educational and the other to be as close as possible to the real industrial world. The educational goal aims to familiarize the system designer with everything that defines: Industry 4.0, concepts, design, Cloud and IoT communication, real-time monitoring and control. Therefore, we try to improve the presented technology by the integration of the state of the art in the field of Industry 4.0, including smart manufacturing, SCADA systems, tasks communication and synchronization, sensors data fusion, and high precision actuators.