Trends in the Control of Hexapod Robots: A Survey

Abstract

The static stability of hexapods motivates their design for tasks in which stable locomotion is required, such as navigation across complex environments. This task is of high interest due to the possibility of replacing human beings in exploration, surveillance and rescue missions. For this application, the control system must adapt the actuation of the limbs according to their surroundings to ensure that the hexapod does not tumble during locomotion. The most traditional approach considers their limbs as robotic manipulators and relies on mechanical models to actuate them. However, the increasing interest in model-free models for the control of these systems has led to the design of novel solutions. Through a systematic literature review, this paper intends to overview the trends in this field of research and determine in which stage the design of autonomous and adaptable controllers for hexapods is.

1. Introduction

Mobile ground robots have an important role in the replacement of human beings in tasks such as surveillance, demining, inspection, rescue and exploratory missions [1,2]. Their demand increases with the necessity of working in complex environments, where humans could be exposed to hazardous surroundings. Since these robots must walk across unforeseen scenarios, their success relies on their ability to autonomously adapt their motion according to the terrain topology and obstacles encountered without damaging their components. Therefore, the interest in the design of adaptable and fully autonomous devices has increased throughout the years.

Among the existent solutions, the legged solutions are adequate for walking across complex environments due to their discrete footholds and capability of generating trajectories in arbitrary directions [3,4]. From this type of robot, insect-inspired systems have been studied to navigate autonomously in complex environments because of their inherent static stability, which is the capacity for keeping the body stable and upright when only reaction forces are applied to the system [2,5]. Their over-actuated design allows the generation of different gait patterns, which can potentially increase their adaptability to the environment [6]. The control of these hexapods implies correct synchronization between the actuation of the limbs. The most traditional approaches follow predefined routines to generate and adapt locomotion. This type of control relies on the correct definition of the mathematical models of the robot to evaluate the influence of the surroundings in its internal state [7].

These systems have shown their versatility throughout the years, being capable of navigating through different terrain topologies and autonomously executing various chores. Nonetheless, recent advances in artificial intelligence (AI) potentially provide an increase in the autonomy of hexapods [8]. This technique can be implemented for multiple objectives, such as providing accurate perception of the environment, planning the motion of the robot or mimicking the adaptive behavior of animals. Another potential application of AI in these robots is for self-learning of how to walk considering their surroundings through a trial-and-error process.

This paper aims at providing insight into the control of hexapods and current trends implemented in these systems and emphasize the potential that these robots have for navigating in complex environments. The information presented throughout the paper consists of a systematic literature review of publications written in English from conference proceedings and indexed papers about the design and control of hexapods within a period of 10 years. The outline of the contents of this paper is as follows. Section 2 discusses the methods adopted for the searching process of the systematic literature review. Section 3 presents the publications found during the literature review, discussing specific cases of hexapods to understand the most important contributions in this field of research. Finally, Section 4 presents the discussion of the results, and Section 5 contains the final considerations of this paper.

2. Materials and Methods

The objective of this research is to evaluate how the design of hexapods is evolving to increase the adaptability of these systems in complex environments. To identify the recent advances in this area, an insight into the most commonly adopted methods for the control of six-legged robots is required. Through this analysis, it is possible to conclude the restrictions and advantages of each methodology and which ones may potentially provide better results for navigation across ever-changing environments. Consequently, this paper is based on a systematic literature review about the design and control of hexapod robots. Since there are no significant advances in the commercialization of hexapods for navigating in complex environments, this research considers only scientific publications related to the topic under discussion. Therefore, all the processed data was collected from the following data platforms: Science Direct, IEEE Xplore, Google Scholar, SpringerLink and Wiley. Among the keywords used during the systematic search, the terms “Hexapod”, “Six-legged robot”, “Navigation”, “Complex environment” and “Adaptive gait” were the most adopted ones. To understand the design of bio-inspired controllers, the keyword “Central Pattern Generator” was also searched. Moreover, to restrict the number of publications, only papers written in English were selected. The timespan selected for this study was 10 years, since the publications from the previous decade may not reflect the real stage of these research works. If in the established timespan there was more than one publication about a specific project, then only the most recent one was analyzed, except when the previous pieces of research contained useful information for this study. Moreover, if several papers proposed similar control methods and obtained similar results, then only the most recent one was not rejected. By the end of the systematic literature review, 57 papers were analyzed.

3. Control of Hexapods

The scientific publications obtained from the systematic literature review were categorized according to the methodology proposed for planning and controlling the locomotion of the designed hexapod (e.g., traditional controllers, bio-inspired architectures and Reinforcement Learning (RL)). For each category, the publications were presumed to provide an answer to the following questions:

• How was the hexapod tested? The control systems can be tested through simulations or experiments. However, testing the robot under real circumstances allows for concluding that the proposed system is reliable in uncontrolled conditions;
• What was the type of environment? Since the surroundings of the hexapod influence its locomotion, it is important to understand if there are robots already capable of autonomously navigating in extreme environments or if these systems have been mainly tested in controlled conditions. This research divides the type of environment into indoor with regular ground, indoor with irregular ground, which mainly consists of a household scenario with objects randomly displaced on the floor, stairs, depressions or ramps, and outdoor, which can contain the same obstacles seen in the previous case but also has different types of soil, with various friction coefficient values and more asperities;
• Does the hexapod generate an adaptive behavior? Despite some publications being mainly focused on the generation of stable locomotion, it is important to understand the limits of the adaptability in these robots;
• What type of sensors does the hexapod have? The selected sensors provide insight into the control system (e.g., if they rely mainly on proprioceptive or exteroceptive information to control and adjust the locomotion of the robot);
• Which computer vision algorithm was adopted? If the hexapod contains vision sensors to gather exteroceptive data, it also requires analyzing this information for the decision-making process.

3.1. Traditional Controllers

Traditional control systems consider the hexapod as a rigid body connected to six robotic manipulators and analyze the actuation and control of each leg individually [7]. Moreover, the locomotion is fully described through kinematic and dynamic models, with the robotic legs considered as open-chained mechanisms. The most common implemented gaits for these robots are the tripod, the ripple and the metachronal patterns [9], which are considered symmetrical due to their periodic phases. Due to the vast number of adopted methodologies for the design of these control architectures, the gathered information was grouped into kinematic-based, dynamic-based control and real-time path and gait planning.

3.1.1. Kinematic-Based Control

This methodology relies on the calculation of the desired angular position of the joints or the torque of their actuators according to the desired motion of the limbs. For instance, although the Euler–Lagrange dynamic model was implemented to study the interactions of a robot, the actuation of a hexapod was controlled using inverse kinematics and the current angular position of the joints as feedback [5]. Considering a different application, Zu et al. [10] calculated the position of the center of mass (CM) of a robot using its kinematic model to adjust its turning radius for obstacle avoidance. Nonetheless, the environment influences the stability of the robot, and to generate adaptive locomotion, it is required to obtain a perception of the surrounding. One of the most observed methods to generate adaptive gaits considers an evaluation of the contact forces of the feet to detect the asperities of the ground, such as in [11,12]. Irawan and Nonami [13] also detected the contact forces of the feet, and if they overcame a certain threshold, the actuation of the legs was controlled by the model of a linear spring to increase impact absorption during the stance phase. Likewise, ref. [14] used contact detection to adapt the locomotion of a hexapod. The novelty of this research consists of the searching algorithm, which aimed at determining new footholds when the limbs could not detect contact forces during the swing phase. Considering the kinematic constraints, the feet first exerted a downward trajectory, and if the sensors did not detect contact forces, they executed several searching trajectories until the limits of the joints were achieved. Moreover, another difference of this research was the implementation of the Static Stability Margin (SSM) to evaluate the stability of the gait. This method evaluates the distance of the ground projection of the CM of the robot toward the edges of the Support Polygon (SP). If the CM is within the SP, then the robot is statically stable.

The SSM has been widely used for the control of the stability of hexapods, mainly to evaluate if the torso is upright during the transition between the phases of the gait. In [15], the usage of the SSM aimed at ensuring that the robot generated a stable gait while walking across steps and ramps and that the machining tool remained centered with the body. Likewise, ref. [16] used the SSM to adjust the posture of a hexapod while walking across ramps with a maximum slope of 30 degrees. Along with aiming to reduce the energy consumption of the actuators of Noros-III, considering the desired motion of the object intended for manipulation, Ding and Yang [17] also used the SSM to evaluate the stability of the four-legged gait generated to carry the load. Zhao et al. [4] concerned themselves with the ability to overcome obstacles and calculated the SSM at each gait cycle to determine if the trajectory defined to surmount an object was stable. Additionally, Liu et al. [6] took advantage of the joint redundancy of hexapods and tested their ability to generate stable gaits when one of their legs malfunctioned, using the SSM to obtain the most stable configuration of the robot. Considering a different approach, ref. [18] adopted the Longitudinal Stability Margin (LSM), which is similar to the SSM, but it calculates the minimum distance considering the direction of motion. This method was implemented to generate stable locomotion when a hexapod walked across terrain with several forbidden places and was required to adjust its footholds to avoid them.

Along with the ability to navigate across unstructured environments and overcome obstacles, some pieces of research were concerned with the ability to climb surfaces. Henrey et al. [19] included dry adhesive mechanisms on each foot of the hexapod Abigaille-III and evaluated the required adhesion forces to climb a vertical surface and determine the desired torque of the joints. The ability to climb in confined spaces, such as chimneys, using the estimation of the stiffness of the robot and the deformation between the feet and the wall to control the actuation of the joints was studied in [20]. The experiments that were carried out also used an Inertial Measurement Unit (IMU) to detect and control the tilting of the body during its ascending motion. Using a different strategy, the authors of [21] implemented micro spines inspired by the microscopic hairs of cockroaches in the legs of the T-RHex, which allowed it to climb different surfaces such as brick walls or plywood with slopes of 135 degrees.

Table 1. Publications regarding kinematic-based controllers (1: indoor regular ground; 2: indoor uneven ground; and 3: outdoor).

Reference (Year)	Simulation/ Experiment	Environment	Adaptive Behavior	Sensors	Computer Vision Algorithm
[5] (2012)	Yes/No	1	-	-	-
[13] (2012)	Yes/No	2	Adjust to the terrain topology	Force sensors	-
[11] (2014)	No/Yes	2	Adjust to the terrain topology	Tactile sensors	-
[20] (2014)	No/Yes	2	Climb surfaces	-	-
[15] (2015)	Yes/No	2	Climb ramps and steps	-	-
[18] (2015)	Yes/No	2	Avoid forbidden zones
[17] (2016)	Yes/Yes	1	Cargo transportation	Force and infrared sensors and a camera	-
[10] (2017)	Yes/Yes	2	Obstacle avoidance	Attitude sensor and Kinect	-
[16] (2017)	No/Yes	2	Walk across ramps	IMU and encoders	-
[4] (2018)	No/Yes	2	Obstacle avoidance	RGB-D camera, IMU, compass, LiDAR, GPS, force sensors and encoders	-
[20] (2018)	Yes/Yes	2	Climb in confined spaces	IMU and force sensors	-
[12] (2019)	Yes/No	2	Adjust to the terrain topology	Force sensors	-
[14] (2019)	Yes/No	2	Adjust to the terrain topology	Force sensors	-
[21] (2019)	No/Yes	3	Climb surfaces	-	-
[6] (2020)	Yes/Yes	1	Damage recovery	Force sensors and gyroscope	-

summarizes the publications previously discussed.

3.1.2. Dynamic-Based Control

Although some of the previously described papers resorted to force torque measurements to detect the stance phase and changes in trajectory of the legs [11,12,13,14], the control of the deviation of the position of the feet or the angular position of the joints relied only on kinematic variables (e.g., position, velocity and acceleration). On the contrary, the dynamic formulation of the robot allows for obtaining an overview of its interaction with the environment through the analysis of the deviation of motion caused by the external forces and torques applied on the hexapod.

Table 2. Publications regarding dynamic-based controllers (1: indoor regular ground; 2: indoor uneven ground and 3: outdoor).

Reference (Year)	Simulation/ Experiment	Environment	Adaptive Behavior	Sensors	Computer Vision Algorithm
[25] (2012)	Yes/No	1	-	-	-
[31] (2013)	Yes/Yes	1	Cargo transportation	-	-
[24] (2016)	Yes/Yes	2	-	Force sensors	-
[9] (2017)	No/Yes	2	Adjust to the terrain topology and carry objects	Force sensors, LiDAR, IMU and encoders	-
[22] (2017)	Yes/No	1	-	-	-
[28] (2018)	No/Yes	3	Adjust to the terrain topology	Stereo camera, encoders, current sensors and IMU	Visual inertial odometry
[23] (2019)	Yes/No	1	-	-	-
[29] (2019)	No/Yes	3	Walk across confined spaces	RGB-D sensor	Visual inertial odometry
[30] (2019)	No/Yes	3	Adjust to the terrain topology	-	-
[26] (2020)	Yes/Yes	3	Adjust to the terrain topology	Torque sensors and encoders	-
[27] (2020)	Yes/Yes	2	Wall walking	Force sensors, IMU and encoders	-

summarizes all the pieces of research that contained this type of control. While Khudher, Powell and Abbod [22] used this formulation to generate a torque controller based on the desired acceleration of the feet for a hexapod meant to aid in humanitarian demining, ref. [23] focused on the generation of dynamic stable gaits. This piece of research presented the full dynamic model of a hexapod and tested the Dynamic Gait Stability Margin (DGSM), which evaluates the stability of a gait through the generated angular momentum and the edges of the SP. In this case, the hexapod was stable if the balance between the total angular momentum and the minimum angular momentum required to tumble the hexapod over an edge of the SP was positive.

Nonetheless, one of the most important applications of these models is the evaluation of the contact forces between the feet and the ground, as in [24], which is important for the adjustment of the hexapod’s behavior when navigating across an environment with different types of soil and asperities. Soyguder and Alli [25] simulated a Spring-Loaded Inverted Pendulum (SLIP) model for the dynamic stability control of the locomotion of a hexapod. Like the animals’ behavior, this system estimated the torque values of the actuators, which ensured that the virtual dynamic model of each limb behaved as a linear spring during the stance phase, absorbing some of the impact caused by the landing of the feet while considering an increase of their stiffness during the swing phase. To adjust the posture of the HITCR-II in outdoor environments, Liu et al. [26] designed a virtual suspension dynamic model based on a foot force compensation model to control the height, pitch and yaw angles of the body in mild rugged terrain and an adjustment method based on the polygon formed by the non-adjacent limbs in the stance for walking across ground with more asperities. The irregularity of the soil was identified during the swing phase by comparing the obtained position of the feet and the expected foothold in flat terrain in the Z-axis. Ref. [9] designed a compliance controller to adjust the position of the limbs based on the error between the measured and the expected contact forces. Since the hexapod must generate a stable gait to eccentrically carry objects with two of its limbs and to walk across unstructured ground, a Kalman filter was implemented to predict the position of the Zero-Moment Point (ZMP) and check if it was within the SP. Like for the DGSM, the advantage of this method, in comparison with the SSM and the LSM, is considering the influence of the motion of the robot in the capacity of keeping the torso upright instead of assuming quasi-static circumstances. Ref. [27] also used a Kalman filter to estimate the pose of the hexapod when walking across surfaces perpendicular to the ground. This algorithm estimated and corrected the body posture through the data provided by an IMU, the contact forces and the position of each joint and obtained better results than the respective dynamic model of the robot, even when foot slippage occurred.

Using a different method than the one proposed in [25] to adjust the stiffness of the limbs during actuation, Bjelonic et al. [28] implemented impedance controllers on Weaver to increase the stability of the motion and its energy efficiency. This publication also discusses the gathering of data from the surroundings both through visual inertial odometry to detect obstacles and through an IMU and the torque of each joint to calculate the roughness of the ground. In addition, in [29], Weaver was studied for navigating in complex environments. This piece of research used a Kalman filter to estimate the floor and ceiling of confined spaces so that Weaver could walk across them. The hexapod successfully adjusted the elevation of its body to pass over or under obstacles. The analysis of interactions of the surroundings through proprioceptive information is also discussed in [30], which adopted the Euler–Lagrange method to determine the deviations between the desired and real positions of the joints through the contact forces and the generated torques of the actuators.

Another advantage of using the dynamic model of the robot instead of kinematics is the possibility of evaluating the energy consumption of the actuators. Jin, Chen and Li [31] minimized the energy consumption of a hexapod for different payloads through its Euler–Lagrange formulation and the torque distribution of the joints. Hence, the model could adjust its gait parameters with the additional mass that it must carry.

3.1.3. Real-Time Path and Gait Planning Methods

The pieces of research presented in the previous subsections mainly discuss the ability to adapt or generate a specific walking pattern. However, in a complex environment, the usage of complex and accurate mechanical models to define the adequate behavior for a hexapod may be time-consuming. This subsection presents some publications that discuss this issue but do not refer to whether the overall control architecture is kinematic or dynamic-based.

Some of the publications gathered discussed the usage of computer vision algorithms to calculate safe trajectories. For instance, ref. [32] implemented the Rapidly Exploring Random Tree (RRT) algorithm to estimate a safe path in an unknown outer-space environment. Likewise, ref. [33] studied the RRT method for path planning in a simulated environment with different types of obstacles. Deepa et al. [34] also proposed a Large-Scale Direct Monocular SLAM methodology to map the surrounding of the hexapod and plan its path, predicting threatening zones, but this method was not tested in a locomotion planning algorithm.

Aside from the generation of safe routes, the hexapod needs to know how to adapt its behavior. The method presented in [35] aimed at reducing the time required for the controller of a hexapod to plan its gait through an Artificial Neural Network (ANN) and Fuzzy Logic. The Fuzzy Logic was responsible for the determination of the correct actuation of the limbs in an unknown environment, while the ANN decided which was the most adequate locomotion pattern when the hexapod walked across an already known environment. The advantage of this method was that the robot did not need to evaluate the terrain because it knew from experience what was the most adequate gait. Considering a different issue, Tennakoon [36] included a Support Vector Machine to detect if the terrain collapsed with the contact forces of the hexapod when walking across brittle surfaces. Using this information, the robot adjusted the position of its CM and its footholds to avoid unsafe ground.

3.2. Bio-Inspired Controllers

Bio-inspired control architectures aim at mimicking the process of generation of locomotion through the implementation of an ANN [37]. Their implementation intends to provide an optimal adaptation of the behavior of the hexapod to the environment [6,38]. For mammals, the excitatory and inhibitory locomotor commands descend from the Mesencephalic Locomotor Region through the spinal cord to activate or inhibit the Central Pattern Generator (CPG) of each limb, causing rhythmic excitations of the motoneurons of the flexor and extensor muscles [39]. Using the same biological principles, these systems have a higher control center, like the animal brain, which sends the motor commands to the ANN responsible for the actuation of the legs (i.e., the CPG network). This ANN usually contains a neural oscillator per leg (i.e., two neurons which inhibit one another) to bio-mimic the extension and contraction of the muscles [40].

Table 3. Publications regarding bio-inspired controllers (1: indoor regular ground; 2: indoor uneven ground; and 3: outdoor).

Reference (Year)	Simulation/ Experiment	Environment	Adaptive Behavior	Sensors	Computer Vision Algorithm
[41] (2012)	Yes/Yes	1	-	-	-
[42] (2013)	Yes/Yes	1	-	-	-
[43] (2013)	Yes/Yes	1	-	-	-
[44] (2014)	Yes/No	2	Climb ramps	-	-
[40] (2014)	No/Yes	2	Adjust to the terrain topology	IMU	-
[45] (2014)	No/Yes	2	Adjust to the terrain topology	Infrared, force and ultrasonic sensors, scanning laser range finder and an IMU	-
[46] (2015)	Yes/No	3	Adjust to the terrain topology	Force sensors	-
[47] (2016)	Yes/Yes	1	-	-	-
[48] (2017)	Yes/Yes	2	Obstacle avoidance	Vision sensor	AI
[49] (2017)	Yes/No	1	Obstacle avoidance	Ultrasonic sensor, color camera and microphone	-
[50] (2017)	No/Yes	2	Adjust to the terrain topology	Infrared sensors	-
[51] (2018)	Yes/Yes	3	Amphibious behavior	-	-
[52] (2018)	No/Yes	3	Adjust to the terrain topology	Torque sensor	-
[53] (2018)	No/Yes	2	Obstacle avoidance	Kinect	-
[54] (2019)	Yes/No	1	-	Force sensors	-
[38] (2020)	Yes/Yes	2	Adjust to the terrain topology	Force sensors	-
[55] (2020)	No/Yes	1	-	-	-
[56] (2020)	No/Yes	1	-	-	-
[57] (2020)	Yes/Yes	2	Climb ramps	Gyroscope	-

summarizes the publications analyzed during the literature review.

To generate symmetrical gaits, the most common approach for the design of a bio-inspired architecture consists of a CPG network with six coupled oscillators to rhythmically activate the swing and the stance phases of the limbs and a lower control layer to convert the output from the coupled oscillators into the angular positions of the joints, as described in [42]. The desired gait pattern is obtained through tuning the parameters of the neural oscillators. Nonetheless, not all the analyzed publications used the same type of oscillators. While the authors of [42,47] used non-linear modified Van der Pool (VDP) oscillators due to the stability of their output signals, which were not affected by external disturbances, refs. [41,44] proposed the usage of Hopf oscillators due to their stability and simplicity, considering that it was not required to bio-mimic the actuation of an insect’s limb since the robotic legs are already simplifications of these biological systems. Grzelczyk, Stańczyk and Awrejcewicz [49] studied the implementation of VDP, Hopf, Rayleigh and stick-slip oscillators in a CPG model and concluded that the latter was the most adequate due to its simplicity and low energy consumption. On the other hand, refs. [55,56] used spiking neurons instead of oscillators for the design of a CPG architecture to improve the computational efficiency and due to the possibility of using temporal events as activation functions. Despite the method adopted, all these publications provided similar results in terms of generating tripod, wave and metachronal gaits and safely transitioning between them.

The main interest of using bio-inspired architectures is to generate adaptive locomotion through the implementation of sensory feedback by adjusting the parameters and output signal of the oscillator or the spiking neuron when a perturbation occurs in the system [55,58]. The simplest method combined an IMU with a CPG model with three Matsuoka oscillators to control the attitude of a hexapod [40]. In this research, the data provided by the IMU, which was placed on the CM of the robot, were used to adapt the output of the oscillators, adjusting the height of the feet and ensuring that the body remained horizontal with the ground. However, as observed in other research, the perception and classification of the terrain is important for the generation of adaptive locomotion:

• Liu et al. [45] used the data provided by an IMU, the force sensors placed on the feet, an ultrasonic sensor and a scanning laser range finder as input for the CPG network to analyze the irregularity of the terrain. Nonetheless, this method was highly influenced by the posture of the hexapod;
• A different strategy was presented in [52], using a radial basis function ANN for the online classification of the ground through the torque generated in each joint. The value obtained adjusted the parameters of the six VDP oscillators of the CPG layer. The hexapod was tested in fine and coarse gravel and on a smooth surface;
• Yu, Gao and Deng [38] implemented reflexive and sensitive neurons to bio-mimic the reflexive behavior of animals. The input of these neurons is the contact force of each limb. If the system detects early contact with the ground due to collision with an obstacle, then the stance phase is activated. On the contrary, when the reflexive neuron does not detect any contact force after the swing phase, the limb executes several swing trajectories to search for a new foothold;
• The implementation of reflexive neurons is also discussed in [50], where they were used in a Hopf-based CPG network to generate self-adaptable crab-inspired locomotion. In this research, the hexapod had an infrared sensor on the tip of each foot to detect the ground and execute a similar behavior to the one presented in [38];
• Although the generation of reflexive mechanisms is also discussed in [46], this research used a reservoir-based Recurrent Neural Network (RNN) in each leg to implement these behaviors. The RNN predicts the state of the limb by processing both the sensory feedback from the joint’s control and the data from the force contact sensor and adjusts the motor commands that were sent by the CPG model. This control architecture was implemented in AMOS-II and tested in complex environments, such as surmounting gaps, walking across ground with variable topology and climbing surfaces, providing good results in terms of adaptability;
• Using a gyroscope to evaluate the attitude of the hexapod, Wang et al. [57] studied the adaptability of the motion of the limbs to adjust the body posture and execute transitional motions between flat ground and slopes. The robot was able to climb ramps with inclinations up to 16 degrees;
• Opposite to the other pieces of research, the AmphiHex-II took advantage of its variable stiffness limbs to adjust its posture while climbing ramps and stairs, swimming and walking across unstructured ground [51]. Like the studies that evaluated the generation of symmetrical gaits, the analysis of the Hopf-based CPG model consisted of tuning its parameters to generate a tripod pattern, which highlights the simplicity of this solution for navigating in complex environments.

The ability to adjust the generated gait for surmounting obstacles has also been tested for these control architectures. Although the experiments were carried out in an indoor regular ground, ref. [48] discussed the integration of an online path planning in a CPG controller. This research used a Lobula giant movement detector neural network to process the data provided by vision sensors and to determine the adequate turning radius of the robot’s trajectory. However, since the vision sensors were placed in the front of the hexapod, and this neural network processed only the data retrieved at each time instant, the system did not have information about the previously avoided obstacles, causing the robot to collide with them when the turning radius was not perfectly adjusted. Zhong et al. [53] also presented an obstacle avoidance method with computer vision, but in this case, the output of the CPG was adjusted to pass over or surmount objects. Although its objective was space exploration, in [54], an SNN to bio-mimic the actuation of the limbs of LAURON V was implemented, and the adaptation of its trajectory to overcome obstacles and pass over them was studied.

3.3. Reinforcement Learning (RL)

Despite being mainly tested for computer gaming [59], RL provides a control system with the ability to learn how to behave to achieve the desired goal through several trial-and-error interactions of the agent (i.e., the robot) with the environment [8]. Therefore, the implementation of this learning method allows the robot to achieve a higher level of autonomy by not requiring any previous information about the environment or human supervision to learn how to walk. For instance, Lele et al. [7] implemented RL in an SNN to generate a stable tripod gait. With this method, the hexapod learned to coordinate its legs without pre-programmed gait sequences. Using a gyroscope and a camera as the input data from the SNN, the reward of the system and adjustment of the network weights was based on the balance of the robot. Since the goal was to walk forward, the camera provided visual confirmation of the motion generated. In comparison with other learning methods, this approach required less time spent during training, where 70% of the cases converged to a tripod gait. However, the recent interest in this area comes with the possibility of allowing the hexapod to self-learn how to cope with its surroundings [38]:

• Obstacle avoidance: Ref. [60] combined Fuzzy Logic with Q-learning to generate real-time control for obstacle avoidance. The Fuzzy Logic is used to organize and group the data provided by the sonars placed on the hexapod into a set of finite states, which simplifies the learning process of the algorithm. This method had fast convergence and learned an optimal strategy, being able to change the hexapod’s direction to avoid different obstacles;
• Adaptive locomotion: In [61], the Monte Carlo method was used to detect the transition between the gait phases through the force sensors placed on the tip of each foot. This data were used along with the SSM for the determination of which leg needed to be actuated to ensure the stability of the robot. In this case, the algorithm evaluates its results at the end of each episode, and there is no assurance that the agent visits all states, which can provide a greedy policy. The generation of adaptive gaits is also discussed in [62]. The proposed method contains a CPG model with two layers. While one is responsible for the inter-coordination of the limbs, to generate tripod, wave or metachronal gaits, the other must adjust the behavior of each limb through the correct actuation of the knee and ankle joints. Hence, to avoid the manual tuning of the oscillators of the second layer, a Deep Deterministic Policy Gradient is implemented. This algorithm used the position and velocity of the robot and the torque, angular position and velocity of the joints as observations to obtain the correct parameters (e.g., amplitude and phase) of the oscillators. The reward function of the algorithm penalized high energy consumption but rewarded high heading velocity values. This method converged to a solution after 1400 episodes, and the robot could successfully adjust its locomotion to different surfaces with different values for the coefficient of friction;
• Damage recovery: Verma et al. [63] proposed a method based on the proximal policy optimization for damage recovery using a supervised learning NN for the self-diagnosis of the damages. This algorithm could find a gait policy when the hexapod had one or two limbs injured. On the contrary, Chatzilygeroudis and Mouret [64] defended that the model-based policy search algorithms were more efficient for the control of robots and designed a reset-free trial-and-error algorithm for the recovery of internal damages. In this piece of research, the hexapod could learn an optimal walking policy by itself when one or two limbs malfunctioned in less than a minute, despite the computational issues presented. Both methods have the advantage of not requiring the agent to return to its initial position after each episode during training. In [65], the issue of damage recovery was also focused on, and the authors proposed a map-based multi-policy algorithm. This method stored and mapped all possible policies to select the one which provided the maximum expected reward. Despite discussing the self-recovering capacity, this research tested only the generation of locomotion in a climbing stairs environment, in which some features of the model, such as the dimensions of some toes, were changed to induce some damage.

The comparison between the pieces of research previously described is presented in

Table 4. Publications regarding the implementation of RL (1: indoor regular ground; 2: indoor uneven ground; and 3: outdoor).

Reference (Year)	Simulation/ Experiment	Environment	Adaptive Behavior	Sensors	Computer Vision Algorithm
[60] (2017)	No/Yes	2	Obstacle avoidance	Ultrasonic sensors	-
[65] (2017)	Yes/No	2	Damage recovery	-	-
[64] (2018)	Yes/Yes	1	Damage recovery	-	-
[61] (2019)	Yes/No	2	Walk across depressions	-	-
[63] (2020)	Yes/No	1	Damage recovery	-	-
[7] (2020)	Yes/Yes	1	-	Vision sensor and gyroscope	-
[62] (2021)	Yes/Yes	3	Adjust to the terrain topology	External camera	-

.

4. Discussion

Considering the information gathered during the literature review, the papers were analyzed according to the final application of the described hexapods, the method adopted to validate the results, the type of environment, adaptability and control and the selection of sensors and computer vision algorithms. Figure 1 presents the distribution of the analyzed papers over the timespan of the literature review. This evaluation highlights the increasing popularity of the design of model-free controllers over the last five years, which can be related to the recent advances in AI applied to robotics. Furthermore, these data also show that the implementation of RL is still a recent trend for the control of hexapods, having only started in 2017. However, the design of traditional controllers maintains its popularity, which can be motivated by the fact that it does not require long and complex simulations to obtain an optimal configuration of the model itself.

From the data provided in Figure 2a, most pieces of research did not present the final application of the designed hexapods. A possible justification of this fact could be that these systems are still proofs of concepts designed mainly to evaluate the feasibility of their control architectures and are not in an advanced stage of their development. Nonetheless, the publications that identified a specific purpose for their robots described tasks that must be executed in complex environments, such as humanitarian demining [22], rescue [9] and outer space missions [17,32,34]. Moreover, these chores require a high level of autonomy to avoid human intervention. The abilities of the presented hexapods are summarized in Figure 2b. Although 25.00% of the studies did not discuss the adaptation of the parameters of the gait to execute a specific task, 35.09% of the analyzed hexapods could change their locomotion with the asperities of the ground, which enhanced the importance of these mobile robots for navigation in complex environments. Considering the information provided by

Table A1. Benchmarking of the types of obstacles and dimensions of the robots.

Reference	Obstacle Dimensions and Type of Terrain	Mass	Robot Dimensions (mm)	Leg Dimensions (mm)
[3,26]	Max. step height: 57 mm [3]; steps with heights of 34 and 51 mm [26]; rugged terrain paved with angular gravel and stones, with a max. height of 210 mm [26]	3.62 kg [3,26]	430 × 529 × 244 (length × width × height) [3,26]	48 × 140 × 122 (coxa × femur × tibia) [3,26]
[4]	Stairs with a max. slope of 23° [4]; max. step height: 200 mm [4]; Ditches with a max. longitude of 600 mm [4]; max. radius of the obstacles: 268 mm [4]	-	1300 × 900 (Feet span area) [4]	-
[9]	Bricks 20 mm high	130 kg	8200 (height)	-
[11]	Wooden blocks 10 mm high	2.50 kg	160 (height)	44 × 70 × 137 (coxa × femur × tibia)
[12]	Obstacles with a size of 50 × 2.5 × 180 mm (length × width × height)	-	200 × 120 × 40 (length × width × height)	38 × 82 × 71 (coxa × femur × tibia)
[14]	Max. slope: 25°; step height: 250 mm; three ditches with the respective dimensions of 600 × 407, 8000 × 150 and 2000 × 800 mm	5 tons	-	-
[16]	Max. slope: 30° (carpet and rubber) and 25° (plywood)	-	980 × 120 (width × height)	-
[19]	Vertical whiteboard	0.635 kg	-	-
[20]	Max. wall distance: 1200 mm	10.3 kg	-	57 × 195 × 375 (coxa × femur × tibia)
[21]	Max. slope: 30° (plywood), 50° (corkboard), 35° (brick) and 55° (textured concrete)	2.5 kg	254 (length of the torso)	-
[23]	Payload: 4.24 kg	0.65 kg (Torso)	-	-
[24]	Different types of soil, such as tile, rubber, expandable polyethylene and soft blankets	7 ton	3000 × 5000 (width × length)	300 × 1500 × 1500 × 150 (coxa × femur × tibia × foot)
[28,29]	Flat concrete [29]; ramp with a slope of 10° [29]; wooden blocks of different heights [29]; mixture of sand, pebbles, stones and crumbled concrete [29]; pass over an obstacle of 22 cm and under an overhanging barrier of 25 cm [30]; walk over a thin gap 70 cm wide [30]	10.3 kg [29]	620 × 630 × 20 (length × width × height) [29]	-
[30]	Different types of soil, such as wooden blocks, stairs and flat ground	2.3 kg	-	52 × 66 × 138 (coxa × femur × tibia)
[31]	Max. payload: 4 kg	2 kg	220 (height)	-
[32,54]	Field with rocks, different types of sand and step trenches [32]; Max. payloadL 10 kg [32]; Step 50 mm high [54]	42 kg [32]	-	-
[36]	Collapsible terrain: damp peat scab, thin ice and styrofoam with a thickness of 5–10 mm; Non-collapsible terrain: brick, gravel, hardwood, styrofoam with a thickness of 15 mm, and hard ice	9.51 kg	500 × 280 (width × length)	-
[38]	Steps with dimensions of 26 × 250 mm (width × height); Ditches with depths of 18, 40 and 80 mm	-	-	180 × 500 × 500 × 25 (coxa × femur × tibia × foot)
[40]	Cuboids 30 mm high; Hill with a height of 75 mm	1.6 kg	353 × 364 × 170 (length × width × height)	-
[45]	Max. slopeL 35°; Cuboids 80 mm in height	-	-	-
[46]	Ditches with widths of 110–150 mm; Climb stairs; Steps 80 mm in height	-	-	-
[50]	Wavy terrain made of rubber plates	-	-	-
[51]	Ramps with 30° slopes; Stairs with slopes of 20°; Different types of soil, such as muddy substrate, grass, and sand; Underwater scenarios	14 kg	510 × 330 × 100 (length × width × height)	175 (length)
[52]	Different types of soil, such as fine and coarse gravel	-	-	-
[57]	Max. slope: 16°	5.64 kg	298 × 120 × 65 (length × width × height)	41 × 81.49 × 150 (coxa × femur × tibia)
[61]	Rugged soil with 200-mm steps and ditches 150 mm in depth	-	-	-
[62]	Max. slope: 10°; Different types of terrain, such as sandpaper, flat, and soft sand	2 kg	240 × 185 × 45 (length × width × height)	45 × 75 × 135 (coxa × femur × tibia)

, the type of soil in which the hexapods were tested ranged from flat concrete [28,29] and tiles [24] to rocks, sand [32,52,54] and grasslands [51], which scopes most of the terrains encountered in outdoor environments. Hence, these systems already provide versatile controllers to deal with the possible variation of the coefficient of friction of the ground. Despite that, the study of the behavior of hexapods in soft terrain is less frequent [28,29,32,53,54,62]. The ability to overcome different terrain topologies also considered scenarios in which the hexapods had to detect and pass over steps and ditches. To overcome steps, the robot must lift up its limbs high enough to avoid collisions with the obstacles. By taking into consideration the height of the obstacles and the size of the legs, which are portrayed in Table A1, the authors of [3,38] presented the solutions which overcame the highest steps, with respective values of 57 and 250 mm. Considering the capacity of overcoming obstacles, 17.54% of the papers proposed solutions to generate alternative gaits for contouring objects. Among the designed barriers, most consisted of cuboid or prismatic shapes, with a maximum height of 180 mm [12]. From Table A1, only [4] presented obstacles like those that could be found in real conditions with irregular shapes and higher dimensions, having a maximum radius of 258 mm. In terms of cargo transportation, which was discussed in 5.26% of the publications, the maximum payload observed, considering the weight of the robot, is presented in [31], in which the hexapod could carry double its own weight. Another important remark is the inclusion of climbing abilities, which is a complex behavior that requires accurate control of both the hexapod’s stability and the adhesion forces of the feet. By mimicking the behavior of insects, robots can potentially overcome large obstacles and confined spaces. Regardless of the results presented in [19] for climbing a vertical white board, the highest slope which a hexapod could climb was 55° for ground made of textured concrete [21].

Concerning the method adopted to validate the results, it can be concluded from Figure 3a that, in general, the usage of experiments, simulations and both methodologies was almost equally distributed. Of the publications, 35.08% conducted tests in real conditions, which means that the proposed systems were already in an advanced stage of research, being able to perform in the desired circumstances. The experiment or simulation choice could be also evaluated according to the type of control, as is presented in Figure 3b. From this data, 42.86% and 35.48% of the papers regarding the usage of RL and traditional controllers, respectively, resorted to simulations for studying their hexapods. On one hand, this methodology tends to simplify both the model, by considering it non-deformable, with its CM centered and without friction in its joints, and the environment. Therefore, using simulations is advantageous to evaluate the designed controller and obtain preliminary results before using the prototype to test them through experiments. On the other hand, transferring the system from the simulation to the real world may not provide an accurate replication of the behavior obtained during tests, due to an over-simplification of the system and its interactions. This issue is of great importance for models with RL because they must learn from their own experience and may generate behaviors that are not physically possible in real conditions or not consider the effect of some physical variables which are assumed as constant during simulations.

In terms of the environment in which the experiments were carried out, 33.33% of the publications tested their hexapods on indoor regular ground, as is portrayed in Figure 4a. Most of these pieces of research aimed at studying the generation of symmetrical and stable locomotion, so the robots had to be analyzed in controlled environments. Nonetheless, most of the experiments (47.37%) were conducted on indoor uneven ground, which means that it was required to adjust the locomotion of the hexapods to overcome obstacles, ramps or steps. This behavior implies more complex control systems to adapt the trajectory of the hexapod while ensuring its stability. Among the most recurrent methods used for these circumstances was the detection of contact forces to switch between the swing and the stance phase and to correct the posture of the torso [9,11,12,13,14,24,25,26,36,38,45,46,50]. Due to requiring a more accurate representation of the change of behavior of the actuation of the legs with the roughness of the ground, only 19.30% of the papers presented hexapods designed for outdoor environments, with some of the tests carried out in testbeds with different types of soil [26,28,30]. Considering this analysis, the ability to walk in highly complex surroundings remains an open question, since some of the studies remain in an early stage. The scenario in which the hexapods were tested also gives insight into the state of the art and the adaptability of the designed controllers. For instance, in analyzing the data gathered in Figure 4b, the fact that traditional controllers were applied more in indoor uneven terrain highlights the capacity of adjusting mechanical models to iteratively adapt the actuation of the limbs and posture of the body. Furthermore, in outdoor experiments, the traditional controllers were adopted with more frequency (22.58%) in comparison with the rest, which also emphasizes the more advanced stage of these models. For bio-inspired controllers, the same percentage of publications regarding indoor regular and irregular terrains is presented, which means that the state of the art provided sufficient results to motivate transitioning the capacity of generating stable gaits from controlled environments to more complex scenarios. On the contrary, 57.14% of the research regarding RL only tested their systems in indoor environments with regular ground. The fact that the potential adaptability of these systems has not been fully applied points out not only the novelty of this methodology but also the higher complexity required for the learning algorithms to generate adaptive motion.

To discuss the adopted models to control hexapods, considering the information provided in Figure 5a, 54.39% of the studies designed traditional control architectures. The main feature of these controllers is the usage of the kinematic or dynamic models of the robots to evaluate the generated locomotion [7]. From Figure 5b, 55.56% of these systems used kinematic models to describe the actuation of the limbs, because it is only required to establish a connection between the angular position of the joints and the coordinates of the feet. Moreover, most of the kinematic-based controllers use geometrical constraints to evaluate the stability of the hexapod, such as the SSM [4,6,14,15,16,17]. The simplicity of these models has been successfully implemented for complex maneuvers like walking across steps and ramps. Despite that, to walk in unstructured environments, it is required to evaluate how the interactions of the hexapod with the surroundings influence its gait. Hence, dynamic controllers have a more important role for navigation in outdoor environments, due to the possibility of both adjusting the robot’s posture with the terrain topology and the foot force interactions and adapting the actuation of the joints to increase the impact absorption during the stance phase. The methods proposed in [25,26,28] aimed at bio-mimicking the change of stiffness of the limbs of animals during locomotion to increase the stability of the hexapods and optimize the energy consumption of their actuators.

To compare the adaptability between kinematic- and dynamic-based systems, Figure 6 portrays the distribution of these models according to the type of environment in which the hexapod was tested. From the presented data, 36.36% of the dynamic controllers were tested in outdoor scenarios, while only 6.67% of the experiments for kinematic systems were carried out in these conditions. Nonetheless, almost 74% of the latter type of controllers provided good results for indoor environments with uneven ground. Hence, despite the adaptability presented by kinematic models, these systems may not provide an accurate representation of the robot in conditions where the type of soil changes, causing the contact between the limbs and the ground to vary. However, the disadvantage of dynamic controllers is requiring accurate mechanical models to control the actuation of the limbs [3]. More complex environments require more accurate and complex mechanical models of the robot to generate adaptive gaits, which increases the required computational efficiency. Most of the designed dynamic-based controllers mainly contained the definition of the interactions of the feet and used kinematic models for the rest of the control. One emergent solution for this problem was the implementation of predictive models to estimate the stability of the hexapod [9], its posture [27] or the safety of the ground through the value of the contact forces [29].

This also motivated the implementation of model-free controllers, such as bio-inspired ones, which use an ANN to generate the trajectory of the limbs. The correct tuning of the parameters of the designed ANN potentially defines a more natural actuation of the joints of the hexapod with optimal energy consumption [6,38]. The implementation of bio-inspired controllers for hexapods takes advantage of the gathered sensory information as an input of the ANN, adjusting the motion of the limbs with the terrain topology and generating stable transitions between gaits. Therefore, bio-inspired architectures can potentially overcome some of the issues of traditional controllers. Nonetheless, the manual adjustment of the neural oscillators or the SNN is considered a disadvantage, due to being a time-consuming process. Therefore, some research published in the past five years studied the implementation of RL to self-learn how to generate locomotion based on the interactions of the robot with the environment [7,60,61,62,63,64,65]. The advantage of this approach is not requiring previous knowledge about the robot or its surroundings, since it learns how to generate gaits through a trial-and-error process. Despite the few publications concerning this topic, RL has been successfully implemented to generate novel gaits when internal failures occur, which increases the level of autonomy of hexapods. Nonetheless, only the authors of [62] considered the possibility of using this learning method to adjust a bio-inspired controller for outdoor environments. The stability of the outputs generated by neural oscillators combined with RL can be an emergent solution for the generation of autonomous adaptive locomotion for unstructured environments.

The sensors adopted are influenced by the type of environment. For instance, hexapods tested in indoor environments with regular ground do mention the usage of sensors. Although it is not mentioned in multiple papers, the actuation of joints usually requires their angular position as an input, and thus the use of encoders is considered common for the control of these robots. From Figure 7a, almost 27.00% of the papers referred to the implementation of force torque sensors, which are required to obtain an estimation of the contact detection and to adjust the robot to the terrain topology. Another possibility is the usage of indirect methods to obtain this information, such as the current sensors described in [28] to estimate the changes in torque of the actuators with the interaction with the ground. Along with these sensors, the implementation of an IMU or similar unit to estimate the posture of the torso is also important for controlling the stability. Therefore, the internal state of the robot can provide accurate information of the surroundings, as it is important to highlight the main usage of proprioceptive sensors for the generation of adaptive gaits in unstructured environments in several publications. Despite that, resorting to vision and other exteroceptive sensors, such as ultrasonic sensors, is also required to detect obstacles and plan the trajectory of the hexapod. In Figure 7b, it can be observed that 92.98% of the pieces of research did not mention or use computer vision algorithms to process data from the surroundings. For model-free controllers (e.g., bio-inspired systems and RL), only one piece of research provided information about a computer vision algorithm [48], which may be justified by the possible increase of the complexity of the ANN. Although some of the potential applications of these robots require visualization of the environment to identify victims, objects or damages and collect data, it can be concluded that this ability has not been fully implemented for hexapods.

5. Conclusions

The static stability of hexapods motivates their design for tasks in complex environments, relying on the autonomy of their control systems to replace humans in hazardous environments. However, more than 85.00% of the studies did not refer to a final application for the designed hexapods, and only 5.26% mentioned the applicability of these robots for space missions, while 3.51% mentioned them for rescuing tasks. For the validation of results, 35.09% used experiments in real conditions, 31.58% resorted only to simulations and 33.33% considered both methods. Most pieces of research were still concerned with adapting stable symmetrical gaits for indoor environments with uneven ground (47.37%), while only 19.30% presented solutions for outdoor environments. Although 25.00% did not discuss the generation of any adaptive behavior, more than 35.00% aimed to generate locomotion adaptable to the terrain topology. By combining these results with the ones obtained for the type of environment, it can be concluded that most research still presents case studies for posture control and adaptation of the gait, using steps, depressions or ramps placed along a plane to verify the feasibility of their control systems. This means that a large number of pieces of research is in an intermediate stage to reach an adaptive control for outdoor surroundings. The adaptability of hexapods also has been tested for obstacle avoidance (17.54%), climbing abilities (10.53%), damage recovery (7.02%) and eccentrically transporting loads using one or two limbs (5.26%). These complex maneuvers emphasize the versatility of hexapods and their capacity to navigate and execute tasks in complex environments without human intervention.

Considering the time span of 10 years, the design of hexapods has gained more popularity since 2017, which also coincides with the rising interest in the implementation of model-free controllers. Regardless of that, the data gathered during the systematic literature review show that approximately 54.40% of the publications presented traditional controllers. Even though 55.56% of these systems used kinematic-based models to control the actuation of the limbs, the design of dynamic controllers aimed at obtaining an optimal energy consumption of the actuators by altering the stiffness of the limb according to the gait phase. Hence, 36.36% of these models were implemented in more complex environments, such as outdoor scenarios. While 33.33% of the papers used a bio-inspired approach to adapt the behavior of the legs, the rest implemented RL to improve the autonomy and learning abilities of the control architecture. Along with the recent advances in AI applied to robotics, the interest in these model-free controllers can be justified by the fact that they do not depend on complex mechanical models of the robots to actuate them, providing stable motor command in ever-changing environments, and they can improve the autonomy of hexapods. At the current stage of the state of the art, 42.11% of the bio-inspired systems were tested for indoor irregular terrain, which means that possibly in the next decade, there could be research showing good results for outdoor scenarios. However, for the usage of RL, approximately 57.00% of the papers only discussed the ability to learn to walk on indoor regular grounds, which means that the generation of adaptive locomotion remains an open question for this type of controller. Additionally, 42.86% of the results for this learning method were only based on simulations. Hence, due to the simplification of the environment and the model itself, some of these results may not be successfully replicated in real conditions. Another important remark is that although few papers discussed the combination of bio-inspired architectures with self-learning algorithms, its implementation not only may provide an optimal response to unexpected events but also accelerate the design of model-free controllers for exploration of unforeseen environments.

Due to the main interest in adjusting the actuation of a hexapod with the changes in the terrain, the most implemented sensors have the objective of detecting the contact forces between the feet and the ground, such as force torque sensors (26.32%), with some papers (almost 4.00%) studying the possibility of detecting these interactions using current sensors. Along with the implementation of an IMU or similar methods to detect changes of posture, the proprioceptive information reveals high importance for the control of hexapods. Moreover, almost 20.00% of the publications referred to the usage of vision sensors, but less than 10.00% discussed the implementation of computer vision algorithms. Thus, despite the recent advances in computer vision, there has not been a current implementation of these methods for gathering data from the surroundings.