Next Article in Journal
Performance of Steel Bar Lap Splices at the Base of Seismic Resistant Reinforced Concrete Columns Retrofitted with FRPs—3D Finite Element Analysis
Previous Article in Journal
Damage Tolerance of a Stiffened Composite Panel with an Access Cutout under Fatigue Loading and Validation Using FEM Analysis and Digital Image Correlation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Design and Evaluation of a Learning-Based Vascular Interventional Surgery Robot

1
Research Centre for Medical Robotics and Minimally Invasive Surgical Devices, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
2
CAS Key Laboratory for Health Informatics, Shenzhen Institutes of Advanced Technology, Shenzhen 518055, China
3
Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
4
Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Beijing 100049, China
*
Author to whom correspondence should be addressed.
Fibers 2022, 10(12), 106; https://doi.org/10.3390/fib10120106
Submission received: 22 September 2022 / Revised: 22 November 2022 / Accepted: 7 December 2022 / Published: 13 December 2022

Abstract

:
Interventional therapy is one of the most effective methods for diagnosing and treating vascular-related diseases at present. It relies on achieving precise and safe navigation of intravascular tools within a patient’s vasculature. Vascular Interventional Surgical Robots (VISR) can reduce surgeons’ exposure to operational hazards including radiation. However, the absence of apt position control and force feedback remains a challenge. This study presents an isomorphic master–slave VISR for precise navigation of endovascular tools viz. catheters and guidewires. The master console aids operators in issuing manipulation commands and logs feedback from the force, rotation, and translation data. The slave manipulator uses the commands received from the master platform for actual tool navigation. However, precise master–slave position control and force feedback are precursors for optimal patient outcomes. This study utilized a fuzzy-PID controller for precise tool navigation and a neural network model for resistance force modulation with 50 mN precision. Furthermore, we evaluated the performance of using the learning-based models within our VISR and compared it with the performances from conventional methods. Results show that the models enhanced the proposed robotic system with better navigation precision, faster response speed, and improved force measurement capabilities.

1. Introduction

Cardiovascular diseases are characterized by high morbidity and mortality globally. It is one of the primary contributors to global disease burden and disease-adjusted life years [1]. Minimally invasive vascular intervention is recently gaining attention as it offers health benefits for patients e.g. minor surgical trauma, less blood loss, and quicker recovery. Thus, percutaneous coronary interventions (PCIs) are now a preferred modality for effective diagnosing and treating different heart and cardiovascular diseases [2]. The procedure involves catheterization of endovascular tools along the blood vessel. Such tools are flexible thin tubes and inserted via peripheral ports into the cardiac regions. Despite their advantages, endovascular interventions are manually performed under fluoroscopic guidance, exposing surgeons to X-rays. A countermeasure is wearing a lead protection apron, which leads interventionalists who frequently appear in the operating room to sustain orthopedic injuries. Thus, robotic catheter systems, first introduced in 2004, now offer significant advantages to interventionalists (e.g. reduced exposure to radiation and orthopedic injury) when performing endovascular interventions.
Recently, researchers have designed remote navigation systems to carry out angiography and angioplasty procedures requiring navigation of the different endovascular tools such as the catheter, guidewire, balloon, and stent. Some commercial telerobotic catheter navigation systems are presented, such as CorPath 200® and CorPath GRX® (Siemens Healthineers, MA, USA), Sensei® X robotic system(Hansen Medical Inc., Mountain View, CA, USA), Niobe®(Stereotaxis Inc., MO, USA), and Amigo®(Catheter Precision, NJ, USA) and they have been gradually adopted by some hospitals. These robotic systems usually possess features of master–slave control mechanisms, closed-loop navigation, and procedural visualization; these criteria are considered for selecting the global leading vascular interventional surgical robot (VISR), while these VISRs have shown to relieve interventionalists’ burden from radiation exposure and orthopedic spine injuries, the existing commercial products and underway prototypes succeeded with further merits of navigation precision, guided control, and remote interventions [3]. Thus, further criteria used for accessing VISRs recently are capabilities for multimodal data feedback, intelligent navigation, and task-based automation during endovascular interventions. However, despite the accumulating evidence that supports the clinical safety and technical efficiency of these systems [4,5], robot-assisted vascular interventions are still limited to a very few clinical centers in the world [6]. This is partly due to the lack of acquisition and feedback of tool-vessel force information during robot-assisted intravascular interventions.
In the teleoperated endovascular procedures, interventionalists need to know how the guidewire touches or interacts with the blood vessel to evaluate the safety of the procedure. Conventionally, this is achieved during PCIs at the distal half of guidewires (i.e., about 30–35 cm from the tip of the flexible tool), which is coated with lubricious materials to reduce tool–vessel friction and enhance slippery tool navigation when guidewires cross coronary artery blockage. Interventionalists are trained with the expertise of stimuli interplay to observe the tactile tool–vessel force interaction, especially when the tool’s tip touches the vessel wall. However, this functionality is lacking in existing robotic platforms used for robot-assisted PCIs (R-PCIs). The lack of such effective tactile feedback in the interventional robots has raised medical, ethical, and legal concerns about intravascular damage to the patient’s vasculature during R-PCIs [7]. As a result, several research groups gradually identified sensing and feedback of distal contact force information during R-PCIs to be an important evolving research [5,8,9]. Currently, interventionalists still depend on visual image-based feedback using Digital Subtraction Angiography. Although this technique is employed to estimate the interaction forces between flexible tools (e.g., guidewires and catheter) and the blood vessels. However, this is mostly imprecise and reduces interventionalists’ eye-hand coordination during R-PCIs [10,11].
Tool–vessel contact forces have been studied from the perspectives of proximal and distal force perception and can be found in [12,13]. Proximal force data are obtained with one or more commercial sensors placed around a clamping device within the robotic system used to maintain endovascular tools in a fixed position. Usually, the resistance information obtained is a complex composite force consisting of contact force and tool hysteresis (e.g., backlash, frictional force, and viscous resistance from blood) and directly interpreted to indicate the interventionalists’ tactile feedback during an operation. On the other hand, the distal force perception requires integration of miniaturized sensors to acquire the interaction force between the flexible tools and patient’s blood vessel wall during intravascular catheterization; while this is the most effective approach and has received more research attention, its developments are still preliminary.
Several methods have been proposed for both distal and proximal force sensing during robot-assisted vascular catheterization with guidewires or catheters in recent years. Broadly, these are classified as sensor-based [12,13,14,15,16,17,18] or model-based approaches [19,20,21,22]. Sensor-based methods typically engage the use of commercial (e.g., ATI F/T or Hall sensors [23]) or custom sensors developed with strain gauge and Fiber Bragg Grating (FBG) sensors [5,7,24,25] with special mechanical structures designed for catheters only. Alternatively, model-based approaches involve analyzing the interventional catheterization process to estimate the force interactions between the flexible endovascular tools (i.e., catheters or guidewires) and blood vessels.
This research aims to provide a novel VISR with machine learning-based approaches for precise position-based navigation control and tool–vessel force measurement. In this study, we propose to use fuzzy-PID for precise master–slave position control and a neural network model for resistance force modulation to achieve 50 mN precision. The remainder of the paper is structured as follows. In Section 2, the design of our sensorized VISR with isomorphic master–slave devices is proposed. In Section 3, analysis of the robot’s dynamics model is presented and linked for learning-based navigation control. The latter includes the development and performance evaluation of a fuzzy-PID method. The results obtained are compared with the performance of the conventional PID. Then, Section 4 introduces a learning-based method for real-time force interaction estimation when catheterization is performed with guidewires. Lastly, the study conclusions are given in Section 5.

2. Design and Manufacture of the Vascular Robotic System

2.1. Overview

This study proposes a VISR that can simulate a surgeon’s realistic operation mode. It includes isomorphic master and slave devices with teleoperated control logic, as shown in Figure 1. Both the master and slave mechanisms are capable of three degrees of freedom (3-DoF) navigation. Thus, the robotic catheter system can be employed for tool delivery with axial translation, rotary motion, and interventional angles of endovascular tools. The robotic system is designed so that the surgeons can teleoperate the master console in the room without being exposed to X-rays. For this, the slave devices can be fixed on a robotic arm to ensure catheterization stability. This mechanism can be placed beside a patient’s bed while it is manipulated to drive the endovascular tools along a patient’s vasculature. The operating mechanism of the robotic system adopts a two-finger operation mode. This involves simulating the wire feeding or rotation of the isomorphic master–slave robotic catheter system. The robot is capable of transmitting motion commands issued at the master-side device onto the slave-side with similar mechanism propulsion. Thus, this ensures a reduced time is required for interventionalists to develop sufficient tool manipulation techniques and skills for the surgical system.
Interventionalists control the robot’s slider in the master console to generate axial or radial tool motions. The slave manipulator performs corresponding actions and controls guidewires or catheter motion based on issued commands from the master platform. Furthermore, a force feedback system is designed to improve interventionalists’ perceptual senses for better motor control. The control architecture proposed for the VISR is presented in Figure 2. It demonstrates that the VISR uses multiple layers for intravascular catheterization. The current version is designed for a shared-control paradigm in which the interventionalists and the robotic system collaboratively manipulate the guidewire and catheter. The system includes a network communication based on the transmission control protocol (TCP), and a buffer of 1024 bits on each side of the TCP connection temporarily holds the incoming data. With multithreading coding, the VISR could realize the different commands and the hand motions utilized for robot-assisted catheterization independently.

2.2. Master Console

The master console, otherwise known as the surgeon terminal, comprises Nvidia® Jetson AGX Orin TM Developer Kit, magnetic displacement sensor, rotary encoder, magnetic powder brake, synchronous belt, lithium batteries as power supply, and a force box, as shown in Figure 3. The magnetic displacement sensor provides axial information when the surgeon pushes and pulls the sensor and sends them to Nvidia Orin TM through the I 2 C communication protocol. The rotary encoder detects rotary motion and sends it to the controller through the Modbus RTU protocol. The resolution of the magnetic displacement sensor is 0.05 mm, and the rotary encoder is set to 0.072 . The Nvidia® embedded controller receives the data of the displacement and rotation information via a RS485/USB converter. After the data is processed, the data is sent to the slave robot through the TCP/IP protocol suite. As shown in Figure 4, the robotic system includes a force box used for measuring the tactile force that interventionalists exerted with their fingers. This force is regarded as the interventionalists’ operative force on the guidewire or catheter in regular surgery. The force box includes a 32-prism, 32 tiny flexible strip sensors, PCBs, and batteries. A flexible sensor is folded and wound around the force box surface, and its data is logged as a 32-channel data multiplexed over an Analog Multiplexer (Texas Instruments, Dallas, Texas, USA). Thus, the interventionalists also operate the master device by manipulating the flexible sensor with their finger. The force data is processed in a microcontroller STM32 (STMicroelectronics, Geneva, Switzerland) and transmitted over a HC-04 Bluetooth module (HC Tech, Guangzhou, China) to the slave console in real time. The force box also reflects information about the flexible tool’s firmness as it is held with a clamping mechanism in the slave manipulator. The guidewire is clamped tightly when the surgeon presses hard on the force box and vice versa.
The controller on the master console transforms the guidewire force information obtained from the slave manipulator into the current to determine the usage of the magnetic powder brake. The brake is used to decide when the tool is safe to be navigated by the interventionalists. The corresponding torque can be generated according to the input current of the magnetic powder brake. With the help of the synchronous belt, the force resistance can bring intuitive force feedback information to the surgeon. This procedure allows the surgeon to establish a close surgical presence to the manual guidewire operation [26].

2.3. Slave Manipulator

The slave manipulator is placed beside the patient’s bed in the operating room and remotely controlled by the master console. Based on analysis of the unactuated flexible endovascular tools, the mechanism is designed to actualize the convention of manually steering the guidewire and catheter in the CathLabs. The slave manipulator shown in Figure 5 and Figure 6 comprises a Nvidia® Jetson AGX Orin TM Developer Kit, an electric slider, rotary motor, motor drivers (Orientalmotor®, Tokyo, Japan), gear sets, clamping device, electric gripper, tool orientation adjuster, battery, digital transmitter, and force sensors. A general-purpose input/output (GPIO) current from the Nvidia Orin, limited to 2 mA, is used. Thus, a microcontroller STM32 is connected through the serial port to control the tool orientation adjuster. Typically, the slave robot can meet the requirements of the hand motions utilized for intravascular interventions. The robotic system can actualize actions such as push, pull, clockwise, or anticlockwise rotation of the guidewire and catheter along the blood vessel.
The slave controller receives instructions from the master device to control the slave’s electric slide and the rotary motor independently. The clamping device is mounted on the slide to realize the translation of the guidewire. An S-shape force sensor is mounted at the rear of the clamping device. An artificial neural network is developed and applied to estimate the proximal contact force of a catheter in the vessel, and it is presented in Section 4. We also installed a tool orientation adjuster, which is now used to realize subdegree rotation of the endovascular tools for accurate navigation of the flexible tools along curvy geometry during an intervention. For the mobility of the VISR, a rechargeable battery pack unit is designed for self-powering. The battery pack has a total capacity of 15,000 mAh, so it can keep the robotic system up and running for an average of 3 h.
The clamping device comprises a stepping screw motor, a guiding block, a clamping bracket block, a spring guide shaft, a linear slide rail, and a thin film force sensor. The clamping device achieves the clamping and releasing of the guidewire and catheter in the slave manipulator. When the lead screw of the stepping screw motor is pushed or retracted, the guiding block will respond accordingly with the same navigation action. A thin film force sensor is attached to the clamping device to measure the clamping force on the guidewire. Therefore, the clamping force can be adjusted according to the operation requirements in real-time to ensure that the flexible endovascular tools (viz., catheter and guidewire) are effectively clamped, while the sensor also measures the tool handling force data proximally.

3. Control System of VISR

3.1. Dynamics Modeling

The dynamic model of its axial displacement is shown in Figure 7. The motor’s input shaft is directly connected with the ball screw, and the ball screw is closely matched with the linear slide rail with nuts; bolts tighten the moving base plate. It is fixed on the linear slide rail, and the clamping device is loaded on the moving base plate. When the motor rotates clockwise or anticlockwise, the rotation will be converted into a linear motion of the moving base plate.
According to similar analysis on the electric slider [27], the overall dynamic equation can be written as:
T s l i d e = J s l i d e · 2 θ s l i d e t 2 + B s l i d e · θ s l i d e t + T l e a d s c r e w
The exact relationship between the driving torque of the stepping motor and the axial displacement of the base plate can be expressed as:
T s l i d e = 2 π p ( J s l i d e + J l e a d s r e w + p 2 m s 4 π 2 η 1 ) · 2 x s t 2 + 2 π p ( B s l i d e + μ v p 2 4 π 2 η 1 ) · x s t + p μ c m s g 2 π η 1
In the formula, T s l i d e is the driving torque of the propulsion motor (N·m); J s l i d e and J l e a d s c r e w are inertias of the propulsion stepper motor and lead screw (kg·m 2 ), respectively; B s l i d e is the damping coefficient of the slide; p is the length of the lead screw (mm); m s is the mass of the moving table (kg); η 1 is the positive efficiency of the lead screw; x s is the axial displacement of the moving table (mm); μ v is the viscous friction coefficient and μ c is the Coulomb friction coefficient.
The transfer function of the propulsion can be calculated with the Laplace transform as:
G ( s ) = 0.96 0.96 s 2 + 0.05 s + 2.5

3.2. Fuzzy PID Controller of VISR

Proportional Integral Derivative (PID) is one of the most sophisticated and frequently used control algorithms in the industry and in other applications requiring continuously modulated control. A typical discrete-time PID controller can be expressed as follows:
u ( t ) = k p e ( t ) + k i 0 t e ( t ) d t + k d d e ( t ) d t
Because of the dynamical effects in the proposed VISR, such as undesirable drifts, friction, and backlash of motors [28], it is essential to adjust the parameters of the PID controller dynamically. Fuzzy PID is a control algorithm that combines fuzzy logic for self-tuning of the control gains in a PID-based control system. The fuzzy controller is designed to automatically fix the gain values for a conventional PID and tune the control parameters online according to the instantaneous error value of the system. This process improves the accuracy of position tracking. The fuzzy logic operates on a set of rules built on two system variables: tracking error and change in error, serving as inputs to the fuzzy controller, and three control gains: k p , k i , and k d , which are its output.
The structure of the fuzzy PID controller is shown in Figure 8, where three parameters—proportional gain k p , integral gain k i , and derivative gain k d —and a fuzzy logic system (FLS) are used to tune the control gains [29]. The input of the FLS included the deviation (e) and deviation change rate ( e c = d e / d t ), while its output is the PID parameters k p , k i , and k d . Each of the system inputs and output consists of three membership functions for position error (e), the error rate ( e c ), and corresponding output variables u ( t ) of the fuzzy controller [30,31]. These are converted into seven linguistic variables: Negative Big (NB), Negative Medium (NM), Negative Small (NS), Zero (ZE), Positive Small (PS), Positive Medium (PM), and Positive Big (PB). Inputs and outputs are all normalized in the interval of [−1, 1]. It takes 49 rules to complete the fuzzy rule base for each gain, as shown in Table 1, Table 2 and Table 3.
Based on Equation (3), a conventional and fuzzy PID controller is established in MATLAB SIMULINK®. It provides tools for automatically choosing optimal PID gains to balance performance (response time, bandwidth) and robustness (stability margins). Using the PID Tuner tools based on the transfer function, we set the maximum error and response time as main tuning goals, which are essential parameters in VISR. The proportional gain k p , integral gain k i , and derivative gain k d are adjusted to 30.84, 20.33, and 11.31, respectively. By comparing the simulation of conventional and fuzzy PID shown in Figure 9, we can measure that the overshoot of the traditional PID control system is 11.79%. In contrast, the newly designed fuzzy PID control system is only 5.8%. Compared with the traditional PID system, the settling time and the precision of the fuzzy PID system have been improved. The signal in fuzzy PID reaches balance status within the range of 98% to 102% faster (from 1 s to 2.760 s) than conventional PID (from 1 s to 3.757 s). The sample period is 0.01s.
The experiment of VISR under PID and fuzzy PID algorithm is shown in Figure 10. The master of VISR was pulled or pushed randomly, and the slave manipulator responded accordingly. The red straight line represents the signal received from the master, which is the trajectory in ideal conditions. The blue dot line and green dash-dot line represent the conventional and fuzzy PID controller performance, respectively. It can be seen that the fuzzy PID algorithm has a faster response speed and better following features than the traditional PID and suppresses backlash in displacement effectively.

4. ANN-based Force Estimation

A learning-based method is also developed for estimating distal tool-vessel interaction forces during catheterization. For this, we developed a neural network approach. In theory, an artificial neural network (ANN) is a generalized collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain to obtain knowledge. Such network has the ability to learn effectively from data and compute nonlinear problems, while traditional machine learning methods often have difficulties modeling the complexities of nonlinear behavior. An ANN model can be implemented to encode complex mathematical modeling, parameter identification, and provide solutions to curve fitting problems [32]. The distal contact force of manipulating the guidewire is proposed to be estimated by modeling proximal forces measured with a commercial S-shape force sensor. The sensor is positioned behind the clamping device in the slave device of the robotic system. The prediction force will be displayed as reference information to the interventionalist during the surgery.

4.1. Experiment System Overview

To obtain the relationship between distal and proximal forces, studies involving logging both force data were conducted using the setup shown in Figure 11. The experiment involved catheterizing an SCW-GW-0.035 guidewire (SCW Medicath, Shenzhen, CN) and navigating it so that it moved along the bending areas of a phantom tube. Thus, the guidewire produces both tip contact force and friction force as it moves along the tortuous. We position that this navigation is similar to guidewire navigation in the blood vessels. The distal composite force and the proximal force in VISR are measured by force sensors SBT630 (Simbatouch, Guangzhou, CN), and the data is measured and recorded in real-time with the Nvidia® Jetson AGX Orin. The acquisition system used for recording force and displacement data was developed and customized in Python, and all signals were logged at a sampling rate of around 60 Hz. The robotic setup included spring and smooth slide that allowed the force to change over a period rather than instantaneously so that they could be recorded precisely in an enormous amount. As mentioned in Section 1, the force measurements in VISR are divided into sensor-based and model-based approaches. This method is more suitable for clinical surgical robots than the model-based one. When the mechanism is efficient and accurate, it has a faster response, higher linearity, and is easier to implement.

4.2. Designing and Training the ANN

Before the ANN training, five static and five random trials were conducted, while the proximal force, distal force, and guidewire displacement data were logged in real-time, simultaneously. In the static trials, the Nvidia® Orin controls the guidewire and delivers slowly at a constant speed of 0.1 mm/s. On the contrary, in random experiments, we manipulate the master controller with random displacements, and the slave terminal performs corresponding movements. The guidewire is manipulated like in the actual surgery environment. The force values in static experiments are training datasets (9791 samples), and the values in random experiments are testing datasets (3795 samples). The datasets are used to fit a feed-forward neural network (FFNN) model, i.e., the model was trained to fit the data. As depicted in Figure 12, the developed ANN is a multilayer perceptron (MLP) with an input, an output, and five hidden layers, which have 15, 17, 13, 17, and 15 neurons, respectively.
The relationship between the proximal force obtained with VISR and the distal force from the trained model, as shown in Figure 13. It can be understood that the data hosts features of hysteresis. For instance, when the displacement is taken into consideration, a distinguished pattern can be noted in the data for the cases of force loading and unloading. However, there exists a nonlinear relationship between the proximal and distal force. Thus, a linear fitting method may lead to significant error and this is how we motivated to implement an ANN model to realize the data fitting modeling.

4.3. Force Estimation Evaluation

Several test experiments were conducted to evaluate the performance of the proposed ANN-based algorithm in Nvidia® Orin. Similar to random experiments for testing datasets, we manipulate the master controller like in the actual surgery environment, and the value of the proximal force sensor (4110 samples) is imported into the ANN, so the value of distal force is estimated in real-time. When training the ANN, the loading and unloading are separated, so in the test experiment and actual surgery, the prediction of loading or unloading (viz. push or pull the guidewire) procedures are also separated, and the displacement can be obtained through motor driver. Figure 14 presents one test experiment with real-time force estimation. The difference between real distal force and estimated distal force is presented in the blue curve. The actual and estimated distal force values have a very close trend. However, there are also time delays in the increase and decrease in the force, especially when the guidewire is subjected to a significant force. The main reasons are that the guidewire is bent and soft in the vessels, so it takes time to transmit the force. Moreover, this edge computing algorithm in Nvidia® Orin takes longer to process than in a workstation.
Figure 15 shows the estimation error of five test experiments with random motion. On each box, the central mark indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles of the errors, respectively. The whiskers extend to the 5th and 95th percentiles of the errors. Most estimation errors are under 50 mN, which is within the acceptable range. In traditional surgery without robots, the interventionalist manipulates the guidewire directly by hand, the tactile sense originates from the distal force of guidewire. Hence these errors have no severe effect on PCIs. The results demonstrate the feasibility of the VISR, and the proposed robot can increase the precision and accuracy of guidewire manipulation. The estimated force is displayed on a monitor and sent back to master console and change the resistance force generated from the magnetic brake by controlling the current. Thus, the surgeon can intuitively feel the resistance of the guidewire. Meanwhile, a force-based security system will be activated when the estimated force exceeds a critical value.

5. Conclusions and Future Works

This paper presents a vascular interventional surgery robot with isomorphic master–slave devices and learning-based control models for real-time tool navigation and force prediction. The master device in the VISR is capable of scheduling different tool navigation actions, such as pull or push, while the slave device can respond accordingly. Experimental results show that the VISR supports logging a surgeon’s catheterization behaviors about tool handling force, rotation, and translation. These intravascular methods aid guidewire navigation with resistance-based force feedback when cannulating blood vessels. The control navigation in VISR used conventional and fuzzy PID control techniques. At the same time, recent applications of neural-based and reinforcement learning are underway [31]. In future research, we will use some learning-based control algorithms to enhance the intelligence of the robot and increase levels of autonomy to task autonomy (level 2) or conditional autonomy (level 3) [33]. Furthermore, surgical planning remains an important aspect of robot-assisted interventions [34,35,36]. For instance, robot-assisted intravascular interventions are recommended for selected patients with complex cardiovascular cases because the robot would prevent surgeons’ overexposure to radiation, while patients would also be prevented from disease–related comorbidities. In our future studies, we will integrate a module of imaging data acquisition with deep and reinforcement learning approaches for robot-assisted catheterization in specific case-based vascular catheterization using VISR.

Author Contributions

Conceptualization, X.C.; Methodology, X.C., Y.C. and W.D. (Wenke Duan); Software, G.Y. and W.D. (Wenjing Du); Formal analysis, Y.C.; Validation, J.J. and T.O.A.; Writing—original draft, X.C.; Writing—review and editing, O.M.O.; Supervision, O.M.O. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by National Key Research and Development program of China (2019YFB1311700); National Natural Science Foundation of China (U21A20480 and 61950410618); Shenzhen Natural Science Foundation (JCYJ20190812173205538); and CAS PIFI.

Data Availability Statement

Data is available in a publicly accessible repository.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
VISRVascular Interventional Surgery Robot
PIDProportional Integral Derivative
PCIPercutaneous Coronary Intervention
FBGFiber Bragg Grating
ANNArtificial Neural Network
MLPMultilayer Perceptron
FFNNFeed Forward Neural Network

References

  1. Roth, G.A.; Mensah, G.A.; Johnson, C.O.; Addolorato, G.; Ammirati, E.; Baddour, L.M.; Barengo, N.C.; Beaton, A.Z.; Benjamin, E.J.; Benziger, C.P.; et al. Global burden of cardiovascular diseases and risk factors, 1990–2019: Update from the GBD 2019 study. J. Am. Coll. Cardiol. 2020, 76, 2982–3021. [Google Scholar] [CrossRef] [PubMed]
  2. Grubb, K.J.; Nazif, T.; Williams, M.R.; George, I. Concurrent Coronary Artery and Valvular Heart Disease–Hybrid Treatment Strategies in 2013. Interv. Cardiol. Rev. 2013, 8, 127. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Weisz, G.; Metzger, D.C.; Caputo, R.P.; Delgado, J.A.; Marshall, J.J.; Vetrovec, G.W.; Reisman, M.; Waksman, R.; Granada, J.F.; Novack, V.; et al. Safety and feasibility of robotic percutaneous coronary intervention: PRECISE (Percutaneous Robotically-Enhanced Coronary Intervention) Study. J. Am. Coll. Cardiol. 2013, 61, 1596–1600. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Maor, E.; Eleid, M.F.; Gulati, R.; Lerman, A.; Sandhu, G.S. Current and future use of robotic devices to perform percutaneous coronary interventions: A review. J. Am. Heart Assoc. 2017, 6, e006239. [Google Scholar] [CrossRef] [PubMed]
  5. Omisore, O.M.; Han, S.P.; Ren, L.X.; Wang, G.S.; Ou, F.L.; Li, H.; Wang, L. Towards characterization and adaptive compensation of backlash in a novel robotic catheter system for cardiovascular interventions. IEEE Trans. Biomed. Circuits Syst. 2018, 12, 824–838. [Google Scholar] [CrossRef]
  6. Riga, C.V.; Bicknell, C.D.; Rolls, A.; Cheshire, N.J.; Hamady, M.S. Robot-assisted fenestrated endovascular aneurysm repair (FEVAR) using the Magellan system. J. Vasc. Interv. Radiol. 2013, 24, 191–196. [Google Scholar] [CrossRef]
  7. OMISORE, O.M.; ShiPeng, H.; LingXue, R.; Lei, W. A teleoperated robotic catheter system with motion and force feedback for vascular surgery. In Proceedings of the 2018 18th International Conference on Control, Automation and Systems (ICCAS), PyeongChang, Republic of Korea, 17–20 October 2018; pp. 172–177. [Google Scholar]
  8. Guo, J.; Guo, S.; Yu, Y. Design and characteristics evaluation of a novel teleoperated robotic catheterization system with force feedback for vascular interventional surgery. Biomed. Microdevices 2016, 18, 1–16. [Google Scholar] [CrossRef]
  9. Payne, C.J.; Rafii-Tari, H.; Yang, G.Z. A force feedback system for endovascular catheterisation. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal, 7–12 October 2012; pp. 1298–1304. [Google Scholar]
  10. Pacchierotti, C.; Tirmizi, A.; Prattichizzo, D. Improving transparency in teleoperation by means of cutaneous tactile force feedback. ACM Trans. Appl. Percept. (TAP) 2014, 11, 1–16. [Google Scholar] [CrossRef]
  11. Cao, C.G.; Zhou, M.; Jones, D.B.; Schwaitzberg, S.D. Can surgeons think and operate with haptics at the same time? J. Gastrointest. Surg. 2007, 11, 1564–1569. [Google Scholar] [CrossRef]
  12. Guo, S.; Song, Y.; Yin, X.; Zhang, L.; Tamiya, T.; Hirata, H.; Ishihara, H. A novel robot-assisted endovascular catheterization system with haptic force feedback. IEEE Trans. Robot. 2019, 35, 685–696. [Google Scholar] [CrossRef]
  13. Yin, X.; Guo, S.; Xiao, N.; Tamiya, T.; Hirata, H.; Ishihara, H. Safety operation consciousness realization of a MR fluids-based novel haptic interface for teleoperated catheter minimally invasive neurosurgery. IEEE/ASME Trans. Mechatron. 2015, 21, 1043–1054. [Google Scholar] [CrossRef]
  14. Guo, S.; Wang, Y.; Xiao, N.; Li, Y.; Jiang, Y. Study on real-time force feedback for a master–slave interventional surgical robotic system. Biomed. Microdevices 2018, 20, 1–10. [Google Scholar] [CrossRef]
  15. Zhang, L.; Guo, S.; Yu, H.; Song, Y. Performance evaluation of a strain-gauge force sensor for a haptic robot-assisted catheter operating system. Microsyst. Technol. 2017, 23, 5041–5050. [Google Scholar] [CrossRef]
  16. Bandari, N.M.; Hooshair, A.; Packirisamy, M.; Dargahi, J. Optical fiber array sensor for lateral and circumferential force measurement suitable for minimally invasive surgery: Design, modeling and analysis. In Specialty Optical Fibers; Optical Society of America: Washington, DC, USA, 2016; p. JTu4A-44. [Google Scholar]
  17. Sayadi, A.; Hooshiar, A.; Dargahi, J. Impedance matching approach for robust force feedback rendering with application in robot-assisted interventions. In Proceedings of the 2020 8th International Conference on Control, Mechatronics and Automation (ICCMA), Moscow, Russia, 6–8 November 2020; pp. 18–22. [Google Scholar]
  18. Hooshiar, A.; Payami, A.; Dargahi, J.; Najarian, S. Magnetostriction-based force feedback for robot-assisted cardiovascular surgery using smart magnetorheological elastomers. Mech. Syst. Signal Process. 2021, 161, 107918. [Google Scholar] [CrossRef]
  19. Yaftian, P.; Bandari, N.; Hooshiar, A.; Dargahi, J. Image-based contact detection and static force estimation on steerable rfa catheters. In Proceedings of the 2020 International Conference on Biomedical Innovations and Applications (BIA), Varna, Bulgaria, 24–27 September 2020; pp. 57–60. [Google Scholar]
  20. Zhao, B.; Nelson, C.A. A sensorless force-feedback system for robot-assisted laparoscopic surgery. Comput. Assist. Surg. 2019, 24, 36–43. [Google Scholar] [CrossRef]
  21. Hooshiar, A.; Sayadi, A.; Jolaei, M.; Dargahi, J. Accurate estimation of tip force on tendon-driven catheters using inverse cosserat rod model. In Proceedings of the 2020 International Conference on Biomedical Innovations and Applications (BIA), Varna, Bulgaria, 24–27 September 2020; pp. 37–40. [Google Scholar]
  22. Jolaei, M.; Hooshiar, A.; Dargahi, J.; Packirisamy, M. Toward task autonomy in robotic cardiac ablation: Learning-based kinematic control of soft tendon-driven catheters. Soft Robot. 2021, 8, 340–351. [Google Scholar] [CrossRef]
  23. Chatzipirpiridis, G.; Gervasoni, S.; Berlinger, F.; Ergeneman, O.; Pané, S.; Nelson, B. Miniaturized magnetic force sensor on a catheter tip. In Proceedings of the 2015 Transducers-2015 18th International Conference on Solid-State Sensors, Actuators and Microsystems (TRANSDUCERS), Anchorage, AK, USA, 21–25 June 2015; pp. 1727–1730. [Google Scholar]
  24. Gan, L.; Duan, W.; Akinyemi, T.O.; Du, W.; Omisore, O.M.; Wang, L. Development of a Fiber Bragg Grating-based Force Sensor for Minimally Invasive Surgery—Case Study of Ex-vivo Tissue Palpation. IEEE Trans. Instrum. Meas. 2021. [Google Scholar] [CrossRef]
  25. Akinyemi, T.O.; Omisore, O.M.; Lu, G.; Wang, L. Toward a Fiber Bragg Grating-Based Two-Dimensional Force Sensor for Robot-Assisted Cardiac Interventions. IEEE Sensors Lett. 2021, 6, 1–4. [Google Scholar] [CrossRef]
  26. Li, Y.; Wang, W.; Duan, W.; Mumini, O.O.; Akinyemi, T.; Du, W.; Zheng, Y. Design of Vascular Interventional Surgical Robot with Network Time Delay Analysis for Master-slave Teleoperation. In Proceedings of the 2021 4th International Conference on Intelligent Autonomous Systems (ICoIAS), Wuhan, China, 14–16 May 2021; pp. 406–412. [Google Scholar]
  27. Yu, H.; Wang, H.; Chang, J.; Niu, J.; Wang, F.; Yan, Y.; Tian, H.; Fang, J.; Lu, H. A novel vascular intervention surgical robot based on force feedback and flexible clamping. Appl. Sci. 2021, 11, 611. [Google Scholar] [CrossRef]
  28. Akinyemi, T.O.; Omisore, O.M.; Chen, X.; Duan, W.; Du, W.; Yi, G.; Wang, L. Adapting Neural-Based Models for Position Error Compensation in Robotic Catheter Systems. Appl. Sci. 2022, 12, 10936. [Google Scholar] [CrossRef]
  29. Yang, C.; Guo, S.; Bao, X.; Xiao, N.; Shi, L.; Li, Y.; Jiang, Y. A vascular interventional surgical robot based on surgeon’s operating skills. Med Biol. Eng. Comput. 2019, 57, 1999–2010. [Google Scholar] [CrossRef]
  30. Omisore, O.M.; Han, S.; Ren, L.; Wang, L. A fuzzy-PD model for master-slave tracking in teleoperated robotic surgery. In Proceedings of the 2016 IEEE Biomedical Circuits and Systems Conference (BioCAS), Shanghai, China, 17–19 October 2016; pp. 70–73. [Google Scholar]
  31. Omisore, O.M.; Akinyemi, T.; Duan, W.; Du, W.; Wang, L. A Novel Sample-efficient Deep Reinforcement Learning with Episodic Policy Transfer for PID-Based Control in Cardiac Catheterization Robots. arXiv 2021, arXiv:2110.14941. [Google Scholar]
  32. Bishop, C.M.; Roach, C. Fast curve fitting using neural networks. Rev. Sci. Instrum. 1992, 63, 4450–4456. [Google Scholar] [CrossRef] [Green Version]
  33. Yang, G.Z.; Cambias, J.; Cleary, K.; Daimler, E.; Drake, J.; Dupont, P.E.; Hata, N.; Kazanzides, P.; Martel, S.; Patel, R.V.; et al. Medical robotics—Regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2017, 2, eaam8638. [Google Scholar] [CrossRef]
  34. Grosso, A.A.; Maida, F.D.; Tellini, R.; Mari, A.; Sforza, S.; Masieri, L.; Carini, M.; Minervini, A. Robot-assisted partial nephrectomy with 3D preoperative surgical planning: Video presentation of the florentine experience. Int. Braz. J. Urol. 2021, 47, 1272–1273. [Google Scholar] [CrossRef]
  35. Grosso, A.A.; Marìn, D.M.; Di Maida, F.; Gallo, M.L.; Lambertini, L.; Nardoni, S.; Mari, A.; Minervini, A. Robotic Partial Nephrectomy with En Bloc Removal of a Renal Vein Thrombus for Multiple cT3a Renal Cell Carcinoma Lesions. Eur. Urol. Open Sci. 2022, 44, 33–36. [Google Scholar] [CrossRef]
  36. Grosso, A.A.; Lambertini, L.; Maida, F.D.; Gallo, M.L.; Mari, A.; Minervini, A. Three-dimensional reconstruction and intraoperative ultrasonography: Crucial tools to safely approach highly complex renal masses. Int. Braz. J. Urol. 2022, 48, 996–997. [Google Scholar] [CrossRef]
Figure 1. Control flow of the VISR. Surgeons teleoperate the doctor’s terminal in the room without X-rays, and the slave manipulator with a robotic arm is placed beside the patient’s bed.
Figure 1. Control flow of the VISR. Surgeons teleoperate the doctor’s terminal in the room without X-rays, and the slave manipulator with a robotic arm is placed beside the patient’s bed.
Fibers 10 00106 g001
Figure 2. Layers of the VISR control architecture.
Figure 2. Layers of the VISR control architecture.
Fibers 10 00106 g002
Figure 3. Model of the surgeons’ manipulator for vascular intervention.
Figure 3. Model of the surgeons’ manipulator for vascular intervention.
Fibers 10 00106 g003
Figure 4. The structure of the force box, which can measure the tactile force exerted by the surgeon’s fingers. The force value will be used as parts of multi-modal information to optimize the control algorithm.
Figure 4. The structure of the force box, which can measure the tactile force exerted by the surgeon’s fingers. The force value will be used as parts of multi-modal information to optimize the control algorithm.
Fibers 10 00106 g004
Figure 5. Structure of the patients’ terminal for vascular intervention.
Figure 5. Structure of the patients’ terminal for vascular intervention.
Fibers 10 00106 g005
Figure 6. Slave manipulator of the VISR.
Figure 6. Slave manipulator of the VISR.
Fibers 10 00106 g006
Figure 7. Dynamic model of the propulsion mechanism.
Figure 7. Dynamic model of the propulsion mechanism.
Fibers 10 00106 g007
Figure 8. Structure of fuzzy logic based PID controller.
Figure 8. Structure of fuzzy logic based PID controller.
Fibers 10 00106 g008
Figure 9. Simulation of response curves of step signal under three controllers.
Figure 9. Simulation of response curves of step signal under three controllers.
Fibers 10 00106 g009
Figure 10. Displacement performance under PID and fuzzy PID. The master of VISR was moved randomly, and the slave manipulator response accordingly.
Figure 10. Displacement performance under PID and fuzzy PID. The master of VISR was moved randomly, and the slave manipulator response accordingly.
Fibers 10 00106 g010
Figure 11. Experimental setup of force measurement. A tube is used to contain the guidewire, and the force applied on it is equal to the force on the entire tube, regardless of the bending state of guidewire.
Figure 11. Experimental setup of force measurement. A tube is used to contain the guidewire, and the force applied on it is equal to the force on the entire tube, regardless of the bending state of guidewire.
Fibers 10 00106 g011
Figure 12. Architecture of the feed-forward neural network used to predict guidewire force. The ANN has an input, an output, and five hidden layers.
Figure 12. Architecture of the feed-forward neural network used to predict guidewire force. The ANN has an input, an output, and five hidden layers.
Fibers 10 00106 g012
Figure 13. Hysteresis curve of the force applied to guidewire. The proximal force is measured by proximal force sensor inside VISR, and the distal force is captured by distal force sensor behind the tube.
Figure 13. Hysteresis curve of the force applied to guidewire. The proximal force is measured by proximal force sensor inside VISR, and the distal force is captured by distal force sensor behind the tube.
Fibers 10 00106 g013
Figure 14. Test experiment with real-time force estimation. The black curve is measured by distal force sensor before the tube, and the red curve is the prediction result of the ANN. The trained neural network provides reasonable force estimates within about 50 mN precision.
Figure 14. Test experiment with real-time force estimation. The black curve is measured by distal force sensor before the tube, and the red curve is the prediction result of the ANN. The trained neural network provides reasonable force estimates within about 50 mN precision.
Fibers 10 00106 g014
Figure 15. Box-plot of errors at five test experiments with real-time force estimation, where the error is defined as the difference between the predicted and measured guidewire distal force.
Figure 15. Box-plot of errors at five test experiments with real-time force estimation, where the error is defined as the difference between the predicted and measured guidewire distal force.
Fibers 10 00106 g015
Table 1. Fuzzy Control Rule table of k p .
Table 1. Fuzzy Control Rule table of k p .
ec
NBNMNSZEPSPMPB
eNBPBPBPMPMPSZEZE
NMPBPBPMPMPSZENS
NSPMPMPMPSZENSNS
ZEPMPMPSZENSNMNM
PSPSPSZENSNSNMNM
PMPSZENSNMNMNMNB
PBZEZENMNMNMNBNB
Table 2. Fuzzy Control Rule table of k i .
Table 2. Fuzzy Control Rule table of k i .
ec
NBNMNSZEPSPMPB
eNBNBNBNMNMNSZEZE
NMNBNBNMNSNSZEZE
NSNBNMNSNSZEPSPS
ZENMNMNSZEPSPMPM
PSNMNSZEPSPSPMPB
PMZEZEPSPSPMPBPB
PBZEZEPSPMPSPBPB
Table 3. Fuzzy Control Rule table of k d .
Table 3. Fuzzy Control Rule table of k d .
ec
NBNMNSZEPSPMPB
eNBPSNSNBNBNMNMPS
NMPSNSNBNMNSNSZE
NSZENSNMNMNSNSZE
ZEZENSNSNSNSNSZE
PSZEZEZEZEZEZENS
PMPBPSPMPSPSPSPB
PBPBPMPMPMPSPSPB
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Chen, X.; Chen, Y.; Duan, W.; Akinyemi, T.O.; Yi, G.; Jiang, J.; Du, W.; Omisore, O.M. Design and Evaluation of a Learning-Based Vascular Interventional Surgery Robot. Fibers 2022, 10, 106. https://doi.org/10.3390/fib10120106

AMA Style

Chen X, Chen Y, Duan W, Akinyemi TO, Yi G, Jiang J, Du W, Omisore OM. Design and Evaluation of a Learning-Based Vascular Interventional Surgery Robot. Fibers. 2022; 10(12):106. https://doi.org/10.3390/fib10120106

Chicago/Turabian Style

Chen, Xingyu, Yinan Chen, Wenke Duan, Toluwanimi Oluwadara Akinyemi, Guanlin Yi, Jie Jiang, Wenjing Du, and Olatunji Mumini Omisore. 2022. "Design and Evaluation of a Learning-Based Vascular Interventional Surgery Robot" Fibers 10, no. 12: 106. https://doi.org/10.3390/fib10120106

APA Style

Chen, X., Chen, Y., Duan, W., Akinyemi, T. O., Yi, G., Jiang, J., Du, W., & Omisore, O. M. (2022). Design and Evaluation of a Learning-Based Vascular Interventional Surgery Robot. Fibers, 10(12), 106. https://doi.org/10.3390/fib10120106

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop