A Deep Reinforcement Learning Approach to Optimal Morphologies Generation in Reconfigurable Tiling Robots

Abstract

Reconfigurable robots have the potential to perform complex tasks by adapting their morphology to different environments. However, designing optimal morphologies for these robots is challenging due to the large design space and the complex interactions between the robot and the environment. An in-house robot named $Smorphi$, having four holonomic mobile units connected with three hinge joints, is designed to maximize area coverage with its shape-changing features using transformation design principles (TDP). The reinforcement learning (RL) approach is used to identify the optimal morphologies out of a vast combination of hinge angles for a given task by maximizing a reward signal that reflects the robot’s performance. The proposed approach involves three steps: (i) Modeling the Smorphi design space with a Markov decision process (MDP) for sequential decision-making; (ii) a footprint-based complete coverage path planner to compute coverage and path length metrics for various Smorphi morphologies; and (iii) pptimizing policies through proximal policy optimization (PPO) and asynchronous advantage actor–critic (A3C) reinforcement learning techniques, resulting in the generation of energy-efficient, optimal Smorphi robot configurations by maximizing rewards. The proposed approach is applied and validated using two different environment maps, and the results are also compared with the suboptimal random shapes along with the Pareto front solutions using NSGA-II. The study contributes to the field of reconfigurable robots by providing a systematic approach for generating optimal morphologies that can improve the performance of reconfigurable robots in a variety of tasks.

1. Introduction

Reconfigurable robotic systems have gained significant attention due to their versatility and potential applications in a variety of domains. A notable framework for the taxonomy and evaluation of self-reconfigurable robotic systems is introduced in [1]. In recent years, considerable progress has been made in the exploration of advanced control methods, autonomy, and complex mechanisms for reconfigurable robots. For example, Refs. [2,3] introduce a staircase climbing robot capable of performing cleaning tasks such as vacuuming and sweeping by navigating over stairs. Similarly, Refs. [4,5] present a reconfigurable facade-accessing robot with autonomous capability to cross over the frames. Although there are numerous applications for reconfigurable robots, floor or horizontal area coverage tasks for cleaning and maintenance (CnM) are considered to be the most common and important.

The global floor cleaning equipment market is a rapidly growing industry, valued at approximately USD 7.53 billion in 2023 [6]. This growth has catalyzed the introduction of innovative cleaning solutions for both home and industrial uses. Service providers are increasingly adopting green cleaning practices, emphasizing eco-friendly techniques and reducing energy consumption. Advances in research have led to the development of reconfigurable robots that can accomplish a wide range of tasks efficiently. These robots can change their morphology by connecting multiple modules or by varying their size. As a result, they are widely adopted across various domains of maintenance and surveillance [7,8,9,10,11]. In the context of floor cleaning, these reconfigurable robots outperform fixed morphology robotic systems as they can vary their morphology and cover difficult-to-access regions [12]. However, the energy consumption of reconfigurable robots is a major challenge for their widespread adoption. This is because reconfigurable robots often need to change their shape during the area coverage, leading to significant energy use. This underscores the importance of devising energy-optimized strategies for area coverage using reconfigurable robots.

Design heuristics for reconfigurable robotic systems, exploring their applicability and efficiency in real-world scenarios, are highlighted in [7]. The implementation of a reconfigurable system for floor cleaning area coverage application is demonstrated through hTetro, a Tetris-inspired shape-shifting robot reported in [12]. Moreover, ensuring energy efficiency in the design of self-reconfigurable robots, particularly by Tetris blocks, has become an important area of study, as highlighted in [13]. Notably, optimizing area coverage and path planning in these robots presents intriguing challenges. The hTetro robot, as highlighted in [12], demonstrates the ability to reconfigure into seven different morphologies for effective cleaning. However, its limitation of only seven morphologies led to the conceptualization of hTetro-infi [14], which has the ability to adapt its morphologies indefinitely for greater area coverage potential.

The trade-offs between area coverage and energy usage based on user preferences were studied in [15]. Modified A* and genetic algorithms for efficient complete path planning were proposed in [16,17]. Further advancements in energy-efficient path planning in complex scenarios were explored to enhance the capabilities of reconfigurable robots [18]. In [19], the authors proposed a graph theory-based approach for planning paths that ensure complete coverage for reconfigurable robots. This approach uses dynamic programming to find the optimal path with the fewest reconfigurations, but it may be computationally expensive for real-world scenarios. A major contribution is made in [20] for energy-efficient area coverage by identifying a range of optimal morphologies using a metaheuristic algorithm in a reconfigurable robot. This approach generates morphology with an optimal balance between area coverage and energy consumption. Energy efficiency in reconfigurable robots has also been studied in other domains, such as underwater self-organizing robots [21] and uneven terrain [22]. Identifying the optimal morphology based on the map layout, where the system can learn from its environment and experiences to adapt and generate more effective morphologies that improve the performance of the reconfigurable tiling robot, remains an open area of study.

By taking considerations from the above discussion, the methodologies and the objectives of this work are listed below:

• Deep RL-based morphology generation: We investigate a deep reinforcement learning (RL) framework to generate optimal morphologies for the Smorphi robot. This framework allows the robot to modify its morphology by reorienting its blocks relative to one another by changing the angle around the hinge.
• MDP modeling: We model the generation of optimal Smorphi robot morphologies as a Markov decision process (MDP). This provides a structured approach to sequential decision-making for morphology optimization.
• RL training with PPO and A3C: We train the MDP model using proximal policy optimization (PPO) and asynchronous advantage actor–critic (A3C) in simulated environments. These reinforcement learning algorithms enable efficient policy optimization for the Smorphi robot.
• Evaluation and comparison: We evaluate the learned policy in various simulated environments and assess the effectiveness of our approach by comparing it with the non-dominated sorted genetic algorithm (NSGA-II), a metaheuristic optimization scheme. This comparative analysis helps demonstrate the advantages of our deep RL-based methodology in morphology optimization for the Smorphi robot.

The rest of the article is structured as follows. Section 2 discusses related works, and the concept of DRL is introduced in the Section 3. Section 4, presents the problem formulation, including the mechanical design of the robot and the coverage planner. Section 5 describes the Markov decision process and the reinforcement learning algorithms used: proximal policy optimization (PPO) and advantage actor–critic (A3C). Section 6 discusses the training and simulation results, followed by the limitations of the proposed approach. The conclusion is presented in Section 7.

2. Literature Survey

Reinforcement Learning in Robotics: Reinforcement learning has shown great potential for training robots to perform complex tasks in a variety of environments. One major area of research has focused on navigation and mapping. For example, in [23], the authors propose a reinforcement learning-based approach to explore unknown environments with a mobile robot, while [24] uses RL algorithms to map cluttered environments autonomously. In [25], the authors exploited the RL algorithm to solve the maze problem, exploring a modular RL-based approach that decomposes monolithic tasks into several chunks, where agents learn in parallel to achieve the goal. While these approaches have shown promise in mapping and navigation applications, it is important to note that they may require a large amount of data and computational resources, which may limit their applicability in real-time sceanrios. RL is also widely adopted in complex tasks such as object recognition and manipulation. For example, in [26], the authors solved the object grasping problem by applying RL with visio-motor feedback, while [27] applies RL to learn how to manipulate objects in a simulated environment. Furthermore, deep reinforcement learning (DRL) is widely applied in multi-robot collaboration to efficiently coordinate homogenous and heterogeneous systems. For example, in [28], the authors present a framework for multi-robot coordination, enabling multiple mobile robots to perform collectively in a decentralized manner. Also, in [29], the authors propose a cooperative exploration strategy for multiple mobile robots. Their approach uses dynamic Voronoi partitions to minimize duplicated exploration areas and a deep reinforcement learning-based collision avoidance algorithm to avoid sudden obstacles. However, it is crucial to evaluate the scalability and robustness of such DRL-based multi-robot coordination in complex, real-world scenarios.

While these limitations exist, a few studies explore RL approaches for improving energy efficiency in different robotic domains. Algorithms, such as proximal policy optimization (PPO), asynchronous advantage actor–critic (A3C), deep deterministic policy gradient (DDPG), recurrent DQN, and trust region policy optimization (TRPO), have been exploited to solve path planning and navigation tasks in mobile robots [30]. Among these algorithms, PPO and A3C have gained prominence due to their stability, ease of implementation, and ability to handle high-dimensional action spaces. TRPO restricts policy updates by constraints, and DDPG, whose policy is deterministic, is not suitable for complex tasks. Further, few studies have explored RL approaches for improving energy efficiency in different robotics domains. For example, in [31], the authors proposed an energy-efficient path planning approach using a Q-learning algorithm in mobile robots, while [32] use an RL approach with the proximal policy optimization (PPO) approach to produce an energy-efficient gait. However, the paper did not include the real trials of the physical robot to validate the proposed theoretical framework. Similarly, in [33], the authors incorporated a deep inverse RL method in a legged robot, where they introduce a preference label based on trajectory ranking; hence the inferred reward map results in a more energy-efficient policy.

Reinforcement learning (RL) has been studied in reconfigurable robots due to their simplicity and adaptability. For example, in [34], the authors introduce a deep RL approach for the hardware design of a reconfigurable legged robot. This approach identified the best crawling motion for the robot in a given environment by modifying its structure. Similarly, in [35], the authors present an RL-based approach for a staircase-climbing robot. The robot learned to safely climb staircases of different sizes using this approach. Additionally, the authors present a unified approach for locomotion and manipulation in a reconfigurable robotics limb using RL in [36]. This approach allows the skills to be shared between manipulation and locomotion. The RL approach has also been widely adopted for area coverage tasks in reconfigurable robots. For instance, in [37], the author presents strategies to overcome obstacles optimally in a reconfigurable robot named RSTAR using Q-learning. Energy-aware complete coverage for a reconfigurable robot named hrombo is reported in [38]. The author identified the optimal trajectories to cover the given map using RL by modeling the problem as the traveling salesman problem (TSP). Similarly, in [39], the author use the LSTM with the actor–critic experience replay (ACER) RL algorithm to generate optimal trajectories to cover the given environment. Similarly, a few coverage approaches using RL have been explored in reconfigurable robots [40,41]. However, all the RL-based approaches that have been studied in self-reconfigurable robots have primarily reported on identifying optimal trajectories for a set of configurations. Moreover, the exiting approaches did not focused on finding the optimal morphology for a given map that maximizes area coverage and minimizes energy consumption which is the focus of the present work.

Shape Optimization Using Reinforcement Learning: In recent years, reinforcement learning (RL) has been extensively used to address the shape optimization problem. For example, in [42], the authors use deep RL for airfoil shape optimization. The RL agent modifies the shape of the 2D airfoil within geometric constraints, and it is run on a low-fidelity external solver to calculate the reward. Through training, the agent is able to generate high-efficiency airfoils. Similarly, in [43], the authors present a novel optimizer for aerodynamic shape optimization that uses RL. The optimizer, based on reinforcement learning and transfer learning, can significantly reduce the computational time required for aerodynamic shape optimization. In [44], the authors proposed a novel RL-based shape optimization algorithm for aerodynamic mitigation of wind-sensitive structures. The algorithm uses transfer learning and meta-learning to learn a policy that can quickly adapt to new environments. These are just a few examples of the many recent studies that have applied RL to shape optimization in fluid mechanics [45,46,47,48,49]. These studies have showcased the adaptability of RL algorithms in fluid mechanics, demonstrating promising results for shape optimization tasks. However, it is important to note that they have only been applied and studied extensively in fluid mechanics. The effectiveness of RL algorithms in the context of optimal morphology generation in a reconfigurable robot remains an open question.

Our work builds upon the advancements in reinforcement learning (RL)-driven shape optimization, aiming to extend these capabilities into the domain of reconfigurable robotics. By leveraging RL methodologies, our approach aims to address the unique challenge of generating optimal robot morphologies that maximize area coverage while minimizing energy consumption—a problem that remains largely unexplored within this field. We use the proximal policy optimization (PPO) and asynchronous advantage actor–critic (A3C) RL algorithms to achieve this goal.

3. Deep Reinforcement Learning (DRL)

Reinforcement learning is an iterative process in which an agent learns how to behave in an environment by performing a certain set of actions and observing the result or rewards of those actions [50]. At each time step, the agent observes the current state $s_t$ of the environment and chooses an action $a_t$. This action prompts the transition to the next state $s_{t+1}$, and the agent receives the reward $r_t$. This process continues until the agent meets the certain mean reward or termination criteria, as shown in Figure 1. The agent’s main objective is to learn a sequence of steps or actions that maximizes the reward over an episode or scenario.

Deep reinforcement learning (DRL) extends the RL framework by integrating deep learning techniques, typically in the form of deep neural networks (DNNs) [51]. In deep RL, a policy is represented by a neural network. This means that the neural network is used to map states to actions. A state is a representation of the environment at a particular time. It can include information about the agent’s position, the objects in the environment, and the agent’s own state. An action is something that the agent can perform in the environment.

Training a DNN involves repeatedly adjusting the weights and biases to reduce the difference between the network’s output and the expected output [52]. This is typically achieved through a process known as back-propagation. In the context of DRL, the DNN learns to map the relationship between states and actions (inputs) and the corresponding expected rewards or values (outputs). Over time, this allows the agent to learn a policy that can effectively select actions to maximize cumulative reward.

4. Problem Formulation

In this work, we formulate the problem of identifying the optimal morphology of a reconfigurable robot $Smorphi$ out of its vast array of configurations as a reinforcement learning (RL) problem. The overall framework for the RL-based morphology optimization problem is depicted in Figure 2.

Agents and Environments: In this formulation, the $Smorphi$ robot is treated as the agent. The agent begins by observing the current state of the environment, which is represented by the morphology of the robot (specifically, the configuration of the three hinges, denoted as ${\theta }_1,{\theta }_2,{\theta }_3$) and the 2D map of the environment, which is represented by the binary values 0’s (obstacle space) and 1’s (free space). Based on this observation, the agent chooses to perform an action in this environment by modifying the robot morphology. Subsequently, the modified morphology of the $Smorphi$ robot is run by an external footprint-based coverage planner. This planner computes the percentage of the area covered $P_c$ and the associated path length $P_{len}$. These metrics serve as a basis for calculating the reward for the agent, as detailed in Section 4.2. Furthermore, in this work, we assumed that the path length is directly proportional to the energy consumption of the robot, as supported by the study in [53].

The agent continues to cycle through these stages of observation, action, and reward calculation until it either achieves a threshold percentage of area covered or until a predefined number of iterations is reached, which is considered as one episode or scenario. At the end of each episode, the mean reward is computed to evaluate the overall performance of the agent. This process is repeated until the algorithm converges. The agent interaction with the designed environment over one cycle is depicted in Figure 2.

The following subsection details the mechanical design, and the footprint-based complete coverage path planner (FBCPP) in detail.

4.1. Mechanical Design of the Agent

The design of the Smorphi robot is derived from transformation design principles (TDP) and facilitators [54,55]. The authors explore a wide range of reconfigurable products, toys, and biological species and generalized their reconfiguration behaviour into three different principles and twenty facilitators. Transformation principles provide guidelines that, when embodied, create transformation, and facilitators enable the creation of mechanical transformation. Some of the work that exploits these principles for designing reconfigurable robots is reported here [7,56,57,58,59]. For example, by exploiting these facilitators, the authors designed a reconfigurable pavement robot with different subsystems. By adapting these TDP, we developed the $Smorphi$ robot. The list below shows the mapping of transformation principles and facilitators to our Smorphi robot:

• Expose/Cover—Reveal or conceal a new surface to alter functionality [54] ⇒ The Smorphi robot exposes/cover its side surface while reconfiguring to a different configuration.
• Generic connection—Employ internal or external connections (structural, power) that can be used by different modules to perform different functions or perform the same [54] ⇒ The Smorphi robot uses hinges in the corners of each block to allow identical modules to be connected, which can enable more number of configurations.
• Shared Drive Transmission—Transmit power from a common source to perform different functions in different configurations [54] ⇒ The Smorphi robot uses the same drive motor for locomotion and reconfiguration.

The design of Smorphi allows it to generate a plethora of unique configurations based on hinge angles. For instance, with a 180-degree step size, Smorphi can achieve eight distinct configurations, namely: [0, 0, 0], [0, 0, 180], [0, 180, 0], [0, 180, 180], [180, 0, 0], [180, 0, 180], [180, 180, 0], and [180, 180, 180]. As the step size decreases, the number of possible morphologies increases substantially. Specifically, with step sizes of 90, 45, 5, and 1 degree, Smorphi can generate 27, 125, 5.06 $\times {10}^4$, and 5.929 $\times {10}^6$ morphologies, respectively.

The design of the reconfigurable Smorphi robot is shown in Figure 3. The Smorphi robot is designed modularly, where each unit is identical and can be connected serially through hinges, allowing the blocks to reconfigure. Each block carries the control unit, locomotion mechanism, and other electrical accessories. A 7.4 V DC motor drives the robot with an encoder coupled with Meccanum wheels. The Meccanum wheels in this robot facilitate two functionalities: they provide locomotion in one configuration and help to reconfigure the robot in the other configuration by controlling the wheels with different velocities. Block 2 is considered the core unit, which carries the main processing unit, including the NVIDIA Jetson PC to control and the Vectotnav IMU to maintain the robot’s orientation by compensating for the error caused by wheel slippage. Additionally, a 2D lidar is integrated into the robot for mapping and localization.

4.2. Footprint-Based Complete Coverage Path Planner (FBCPP) of the Environment

Our previous work propose a footprint-based coverage planner that can perform complete coverage for the different morphologies of a reconfigurable robot [20]. In this work, we adapt that coverage planner to perform complete coverage in a given environment. The coverage planner outputs the percentage of area covered ($P_c$) and the total path length ($P_{len}$) for every unique morphology, which serves as a basis for calculating the reward function for the RL agent. For clarity, we briefly explain the footprint-based coverage planner in this section. The approach for performing area coverage for any morphology based on their footprint consists of two layers: footprint generation and complete coverage path planning based on the footprint.

4.2.1. Footprint Generation

The Smorphi robot’s footprint is assumed as the surface area it occupies at any given time. It is modelled as four independent square blocks connected by hinges on either the left or right side of the blocks, as shown in Figure 4a. Each square block has four vertices, for a total of 16 vertices for the four blocks. The coordinates of these vertices vary depending on the hinge angles, and they are projected onto the X and Y axes to compute the occupied area, as detailed in Equations (1) and (2).

$$\begin{array}{cc}\hfill {\mathit{p}}_x^m& ={[{\mathit{p}}_1^m·\mathit{x}{\mathit{p}}_2^m·\mathit{x}\dots {\mathit{p}}_m^m·x]}^T\end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathit{p}}_y^m& ={[{\mathit{p}}_1^m·\mathit{y}{\mathit{p}}_2^m·\mathit{y}\dots {\mathit{p}}_m^m·\mathit{y}]}^T.\end{array}$$

(2)

The footprint size of the morphology is determined using the Equations (3) and (4)

$$\begin{array}{c}\hfill w_{fp}=|min\left({\mathit{p}}_{\mathit{x}}^{\mathit{m}}\right)-max\left({\mathit{p}}_{\mathit{x}}^{\mathit{m}}\right)|\end{array}$$

(3)

$$\begin{array}{c}\hfill l_{fp}=|min\left({\mathit{p}}_{\mathit{y}}^{\mathit{m}}\right)-max\left({\mathit{p}}_{\mathit{y}}^{\mathit{m}}\right)|.\end{array}$$

(4)

4.2.2. Complete Coverage Path Planning Based on the Footprint

The wavefront coverage algorithm calculates complete area coverage based on the generated footprint. It covers the given map in a wave-like pattern propagating from the starting point and covers all nodes in a grid or map [60].

Upon receiving the hinge angles (${\theta }_1,{\theta }_2,{\theta }_3$), a new configuration is generated, and the initial map is resized accordingly. The wavefront algorithm then calculates the complete area coverage on the resized map and transforms these waypoints back to the original map. Subsequently, the algorithm sweeps through the footprint matrix to estimate the area covered.

The output includes the path length and percentage of the area covered, which are optimized to identify the suitable robot morphology. Figure 4b–d shows an example of footprint-based coverage planning for the configuration ${\theta }_1,{\theta }_2,{\theta }_3\equiv 0,180,0$. The original map is resized based on the footprint value, and the wavefront complete coverage planner is applied to the resized map. The waypoints are then mapped back to the original map to calculate the $P_c$, and $P_{len}$. The detailed pseudocode for the FBCPP is detailed in Algorithm 1. The following section details the modeling of the problem of optimal morphology generation.

Algorithm 1: footprint-based complete coverage path planner (FBCPP)

5. Markov Decision Process

The Markov decision process (MDP) is a fundamental model in the field of reinforcement learning (RL) [61]. An MDP is a mathematical model used to describe a physical or abstract system that evolves over time. It is typically represented as a tuple $(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma )$.

In this work, we modelled the optimal morphology generation for reconfigurable robot as the Markov decision process (MDP) similar to the conventional RL approach.

$\mathcal{S}$ is the state space. For our task, the state of the system is defined by the three hinge angles of the robot, as well as the binary 2D Map M which represents obstacles (1) and free space:

$$\mathcal{S}=\left\{\begin{array}{c}{\theta }_1,{\theta }_2,{\theta }_3,M.\end{array}\right\}$$

(5)

A is the action space. In our case, the action space is a multi-discrete action space, with separate actions for each hinge. The allowable range for each hinge is between 0 and 180 degrees. Actions are defined as follows:

$$\mathcal{A}=\left\{\begin{array}{c}\Delta {\theta }_1,\Delta {\theta }_2,\Delta {\theta }_3\end{array}\right\}$$

(6)

where each $\Delta {\theta }_i\in -1,0,1$.

As per the MDP, the agent gains a reward from the environment based on the observations and action performed in a state. A real-valued function $\mathcal{R}$ determines the reward based on the current state of the robot, the action taken, and the resulting state. The reward function $\mathcal{R}$ maps every state-action-next state triplet to a real number, formally defined as $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$. The total reward is calculated based on the two main components ${\mathcal{R}}_1$, and ${\mathcal{R}}_2$:

${\mathcal{R}}_1$ is designed to guide the agent’s behavior towards maximizing area coverage while assigning negative reward for higher energy consumption. Weighting factors ($w_1,w_2$) are assigned to these two parameters to optimize the result based on specific tasks and requirements. These factors can vary depending on the task and the specific requirements.

$${\mathcal{R}}_1(\mathcal{S},\mathcal{A},{\mathcal{S}}^{\prime })=w_1\ast P_c-w_2\ast P_{len}.$$

(7)

${\mathcal{R}}_2$ is a penalty term for maintaining the same hinge angles across two consecutive time steps. It penalizes the agent for keeping the same hinge angles in consecutive steps, encouraging it to explore different configurations. This is defined as ($\beta$-Penalizing factor):

$${\mathcal{R}}_2(\mathcal{S},\mathcal{A},{\mathcal{S}}^{\prime })=\left\{\begin{array}{cc}-\beta \hfill & \mathrm{if}\mathcal{S}={\mathcal{S}}^{\prime }.\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

The total reward function $\mathcal{R}$ is calculated as,

$$\mathcal{R}(\mathcal{S},\mathcal{A},{\mathcal{S}}^{\prime })={\mathcal{R}}_1+{\mathcal{R}}_2.$$

(9)

$\mathcal{P}:\mathcal{S}\times \mathcal{A}\times {\mathcal{S}}^{\prime }\to [0,1]$ is the state transition function. Given the deterministic nature of our task, if action a is applied in state s, we always get the resulting state $s^{\prime }$ ($\mathcal{P}\to 1$). Thus, the transition function can be considered as deterministic:

$$\mathcal{P}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{\mathcal{S}}^{\prime }=f(\mathcal{S},\mathcal{A}).\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(10)

In our context, the reconfigurable robot is treated as an agent that learns a behavior or policy that maximizes the reward obtained from the actions. The behavior of the agent is known as a policy $\pi$, which is a function that maps states and actions to real numbers. This is formally represented as $\pi :\mathcal{S}\times \mathcal{A}\to \mathbb{R}$. The policy is a probability distribution over actions given a state, denoted as $\pi \left(a\right|s)$. However, in the deterministic case, the policy function can be simplified to a function that maps from states to actions, denoted as $\pi :\mathcal{S}\to \mathcal{A}$. In this scenario, the policy directly determines the action to be taken given the state. In this work, we utilized two different advanced reinforcement learning techniques, namely proximal policy optimization (PPO) and asynchronous actor–critic Agents (A3C). Both are types of reinforcement learning (RL) algorithms employed to determine the optimal morphology (shape) of a $Smorphi$ with four blocks by optimizing the angles between the blocks. These algorithms are discussed in the following subsection.

5.1. Proximal Policy Optimization (PPO)

PPO is a family of policy gradient methods proposed by Schulman et al. [62]. A PPO algorithm employs multiple stochastic gradient ascents for policy updates over several epochs. PPO also collects experiences from the current policy using multiple parallel agents during training. The objective function of PPO ($J_{PPO}$), the clipped surrogate objective function, is shown in Equation (11):

$$J_{PPO}=\widehat{E}[min(r_t\left(\theta \right)\widehat{A},\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ)\widehat{A})].$$

(11)

where $r_t\left(\theta \right)$ is the policy ratio, defined as the ratio of action probabilities as shown in Equation (12):

$$r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{\theta old}\left(a_t\right|s_t)},$$

(12)

where ${\pi }_{\theta }$ is the action probability under the current policy and ${\pi }_{\theta old}$ is the action probability under the old policy used for the rollout. In Equation (11), $r_t\left(\theta \right)$ is clipped within the interval $[1-ϵ,1+ϵ]$. $\widehat{A}$ in Equation (11) is evaluated using generalized advantage estimation.

5.2. Asynchronous Actor–Critic Agents (A3C)

Mnih et al. propose a method for DRL using asynchronous variants of conventional RL methods [63]. Among these asynchronous methods, A3C demonstrated superior performance in an Atari benchmarking test. The A3C method keeps an advantage function and value estimates as $\pi \left(a_t\right|s_t;\theta )$ and $V(s_t;{\theta }_v)$, respectively. Periodic updates are performed based on Equation (13):

$$\Delta {\theta }^{\prime }=log\pi \left(a_t\right|s_t^{\prime };{\theta }^{\prime })A(a_t,s_t,\theta ,{\theta }_v),$$

(13)

where $A(a_t,s_t,\theta ,{\theta }_v)$ is the advantage function estimate, computed by:

$$A(a_t,s_t,\theta ,{\theta }_v)=\sum _{i=0}^{k-1}{\gamma }^i{r}_{t+1}+{\gamma }^{k}V(s_{t+k};{\theta }_v)-V(s_t;{\theta }_v).$$

(14)

In Equation (14), k is upper-bounded by a fixed number of actions and varies from state to state. A3C employs multiple agents (networks) with multiple copies of weights that interact with the environment in parallel and update the global network periodically. The obtained results using this PPO and A3C approach are discussed in the following section.

6. Training and Simulation Results

This section details the training and validation of the proposed framework for optimal morphologies generation. The training is done using two different 2D maps, which resemble an indoor setting, as shown in Figure 5a,b. The black rectangular primitives in the map represent obstacles, and the remaining spaces are considered free for the robot to explore. Each of the 2D maps has a size of 5 m × 5 m, with a resolution of 0.1 m${}^2$.

The environment for the RL training is created using the OpenAI Gym library [64], which provides a custom Application Programming Interface (API) to communicate between environments and different learning algorithms. It has four core functions:

• Initialization: Initializes the environment, action, and observation space.
• Step Function: Defines an action chosen by the agent, calculates the reward, and observes the agent’s behavior.
• Reset: Resets the robot morphology, reward criteria, and number of iterations at the end of each episode.
• Render: Provides visualization of the environment.

The Gym library only provides the environment to interact with, not the DRL agent. Multiple libraries have been developed to implement DRL. In this work, we adapted Rllib from Ray, which is an open-source library that provides state-of-the-art algorithms for training [65]. It allows for scalable and optimized implementation of various reinforcement learning models. We also used the PyTorch framework to train the model.

Table 1. Hyperparameters for PPO and A3C.

Hyperparameter	PPO	A3C
Learning Rate	$3\times {10}^{-4}$	$7\times {10}^{-4}$
Batch Size	4096	32
Discount Factor ($\gamma$)	0.99	0.99
GAE Parameter ($\lambda$)	0.95	0.95
PPO. Epochs	15	-
Max Gradient Norm	0.5	0.5
Entropy Coefficient	0.01	0.01
Value Loss Coefficient	0.5	0.5
Optimization Steps	5	5

shows the list of hyperparameters used for training using the RLLib library. The hyperparameters used in this study were chosen based on the results of previous studies [62,63]. These hyperparameters have been shown to be effective in a variety of tasks. The model training is done on an Ubuntu 20.04 running machine with an Intel Core i7 CPU and an NVIDIA RTX 3080 GPU.

6.1. Analysis of Simulation Results

We trained the model for optimal morphologies generation by implementing the proximal policy optimization (PPO) and asynchronous advantage actor–critic (A3C) algorithms. The results for Map-I and Map-II are shown in Figure 5c–f, which shows the agent’s cumulative reward over the first $3\times {10}^5$ iterations for both algorithms.

As shown in Figure 5, both algorithms demonstrate a consistent increase in mean reward, indicating successful learning and adaptation by the agent. For Map-I, the PPO algorithm reached convergence in $2.5\times {10}^5$ iterations, and for Map-II, it reached convergence in $2.7\times {10}^5$ iterations. The A3C algorithm also showed increases in mean reward for both maps, but it did not reach steady-state convergence in either map.

The PPO algorithm produces a higher mean reward than the A3C algorithm in both maps. For example, in Map-I, the PPO algorithm achieved a mean reward of 122.5, while the A3C algorithm achieved a mean reward of 120. However, the difference in mean reward between the two algorithms is more significant in Map-II, where the PPO algorithm achieved a mean reward of above 105, while the A3C algorithm only reached a peak reward of less than 90 for $3\times {10}^5$ iterations.

The PPO algorithm outperformed the A3C algorithm in generating optimal morphologies in both maps. This indicates the potential of the PPO algorithm for generating optimal morphologies and shape optimization in reinforcement learning problems.

Furthermore, the PPO algorithm demonstrates a smooth learning curve, which indicates its stability. This is because the PPO algorithm aims to provide reliable updates to the policy, avoiding fluctuations that could destabilize the learning process. The A3C algorithm, in contrast, exhibits a more volatile learning curve, displaying periods of rapid improvement followed by occasional performance drops. This behaviour can be attributed to the A3C algorithm’s asynchronous nature, which can lead to increased variability but potentially faster overall learning.

6.2. Evaluation of Optimal Morphologies

This section examines the performance of the top four morphologies produced by the proximal policy optimization (PPO) and asynchronous advantage actor–critic (A3C) algorithms.

Table 2. Optimal morphologies generated by PPO and A3C for Map-I.

Sl.NO	Algorithm	Map	Hinge Angles $({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3)$ (Degree)	Area Covered (Pc) in %	Path Length (Plen)
1	PPO	Map-I	(95, 59, 103)	98.5136	99
2			(97, 61, 107)	98.446	99
3			(96, 61, 109)	98.1036	99
4			(105, 51, 81)	97.422	92
5	A3C	Map-I	(37, 33, 137)	97.6692	99
6			(41, 31, 129)	97.5516	99
7			(37, 35, 125)	97.438	107
8			(33, 31, 125)	97.1308	102

The % of area covered ($P_c$) mentioned in all the Tables includes the obstacle region in the map. The obstacle density for Map-I is 18.7756%, and for Map-II is 22.2142%.

and

Table 3. Optimal morphologies generated by PPO and A3C for Map-II.

Sl.NO	Algorithm	Map	Hinge Angles $({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3)$ (Degree)	Area Covered (Pc) in %	Path Length (Plen)
1	PPO	Map-II	(15, 15, 39)	96.9414	134
2			(15, 21, 39)	96.6926	134
3			(41, 43, 95)	95.2774	96
4			(43, 39, 95)	95.203	96
5	A3C	Map-II	(75, 69, 83)	90.1382	72
6			(83, 67, 97)	89.9714	79
7			(87, 71, 111)	88.9814	79
8			(85, 71, 103)	88.7038	72

summarize the results, which include the percentage of area coverage ($P_c$), path length ($P_{len}$), and the respective hinge angle for each morphology. Even though all of the morphologies listed in Table 2 and Table 3 can be considered optimal, the objective of this analysis is to identify the most effective morphologies for area coverage tasks.

The PPO algorithm consistently generates the most energy-efficient optimal morphologies for both Map-I and Map-II. Specifically, in Map-I (Table 2, row 1), the morphology with the hinge angles (${\theta }_1,{\theta }_2,{\theta }_3\equiv 95,59,103$) achieved a coverage percentage ($P_c$) of 98.5136% and a path length ($P_{len}$) of 99. In Map-II (Table 3, row 1), the optimal configuration with hinge angles (${\theta }_1,{\theta }_2,{\theta }_3\equiv 15,15,39$) resulted in a $P_c$ of 96.9414% and a $P_{len}$ of 134. Similarly, the A3C algorithm’s most optimal morphology in Map-I (Table 2, row 5) with the ${\theta }_1,{\theta }_2,{\theta }_3\equiv (37,33,137)$ achieved a $P_c$ of 97.6692% with a $P_{len}$ of 99. In Map-II, the most optimal morphology produced by the A3C with the hinge angle (${\theta }_1,{\theta }_2,{\theta }_3\equiv 75,69,83$) (Table 3, row 5) produced a $P_c$ of 90.1382% and a $P_{len}$ of 72.

Comparing the results produced by the PPO and A3C algorithms, the PPO algorithm generates highly energy-efficient morphologies in both maps. For example, the morphology set-1 in Table 2 (row 1) achieved a coverage percentage ($P_c$) of 98.5136% and a path length ($P_{len}$) of 99, while all the morphologies generated by the A3C algorithm for the same path length ($P_{len}$) has a lower percentage of area covered. On the other hand, in Map-II, the morphology produced by the PPO (set-2, Table 3, row 2) achieved a $P_c$ of 96.6926% and a $P_{len}$ of 134, while all the morphologies generated by the A3C in Map-II have significantly higher path lengths and lower percentages of area coverage than PPO. Figure 6 depicts the top optimal morphologies generated by the PPO and A3C algorithms in both of the maps, with the footprint-based coverage planner.

This comparative evaluation indicates that the optimal morphologies produced by the PPO algorithm are more energy-efficient compared to those generated by A3C. Therefore, based on the results obtained, we can conclude that the PPO algorithm demonstrates superior performance for tasks that aim to generate energy-efficient optimal morphologies.

6.3. Optimal vs. Sub-Optimal Morphologies

As highlighted in Section 4.1, the $Smorphi$ has a wide range of configurations. Our RL algorithm generates a handful of optimal morphologies for area coverage tasks. This comparison aims to distinguish the optimal morphologies generated from the RL algorithm from the other morphologies from the design space.

Table 4. Randomly selected morphologies from the design space for Map-I.

Sl.NO	Hinge Angles $({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3)$ (Degree)	Area Covered (Pc) in %	Path Length (Plen)
1	(171, 49, 29)	79.2364	92
2	(70, 170, 86)	88.350	121
3	(120, 171, 27)	88.6871	128
4	(71, 113, 125)	86.518	68
5	(2, 55, 17)	82.1452	75
6	(146, 179, 58)	88.0688	135
7	(165, 93, 2)	83.1008	116
8	(0, 180, 0)	100	224

shows the list of sub-optimal morphologies from the design space.

Comparing Table 2, Table 3 and Table 4 shows that the morphologies generated by the RL algorithm are superior in terms of both percentage of area covered ($P_c$) and path length ($P_{len}$). For example, in Map-I (Table 4, row 2), the random morphology with hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 70,170,86$) has a $P_c$ and $P_{len}$ of 88.350% and 121, respectively. In contrast, the morphology generated by the PPO for Map-1 (Table 2, row 3) has a higher percentage of area coverage ($P_c$ = 98.1036%) and lesser path length ($P_{len}$ = 99). Similarly, the morphology with hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 71,113,125$) has a lower energy consumption ($P_{len}$ = 68), but with a lower percentage of area coverage ($P_c$ = 86.518%), making it a sub-optimal morphology for area coverage tasks (Table 4, row 4).

Furthermore, the morphology set with the hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 0,180,0$), (Table 4, row 8), has the highest percentage of area covered ($P_c$ = 100%) compared to all the morphologies produced by the PPO and A3C; however, the path length ($P_{len}=224$) is significantly higher, making it a sub-optimal morphology for energy efficient area coverage tasks.

These results suggest that the morphologies produced by the RL algorithm are superior in producing energy-efficient optimal morphologies for area coverage tasks, with the optimal balance between area coverage and energy consumption.

6.4. Policy Validation

Policy validation is a crucial step in evaluating the generalization of a trained reinforcement learning (RL) agent. This process involves testing the trained agent in new environments that it was not trained on [66]. For this purpose, we created new 2D environments with varying obstacle positions and densities.

We trained our agents, based on the proximal policy optimization (PPO) algorithm, in three newly created environments, with the trained model and baseline. The results of the training are shown in

Table 5. Policy validation in three new environments.

Environment	Iterations Taken for Convergence
	Trained Model	Baseline Model
Environment-1	$5\times {10}^4$	$2.7\times {10}^5$
Environment-2	$4.5\times {10}^4$	$2.1\times {10}^5$
Environment-3	$6\times {10}^4$	$3.2\times {10}^5$

. In all maps, the trained PPO algorithm demonstrated the ability to converge to an optimal result with a relatively small number of episodes. For example, in Validation Environment-1 (Table 5, row 1), the trained PPO algorithm only took $5\times {10}^4$ iterations to converge to a mean reward, while the baseline took $2.7\times {10}^5$ episodes. Similarly, in Validation Environment-3 (Table 5, row 3), the trained agent only took $6\times {10}^4$ iterations, leading to faster convergence, while the baseline model took $3.2\times {10}^7$ iterations. These results show that the PPO algorithm is able to learn effective policies for a variety of environments.

These policy validation results indicate that the proposed framework, leveraging PPO, exhibits scalability to diverse environments with efficient convergence. Given these results, it is plausible that, with more varied maps included in the training phase, the agents could converge even faster. These results highlight the importance and usefulness of reinforcement learning algorithms for optimal morphologies generation.

6.5. Comparison with the Optimization Approach

To further assess the effectiveness of our proposed approach, we compared RL results with a state-of-the-art metaheuristic optimization scheme, the non-dominated sorting genetic algorithm (NSGA-II) [67]. We treated our problem as a multi-objective optimization task with the objectives of maximizing area coverage ($f_1$) and minimizing path length ($f_2$). The Pareto Frontier [68] results obtained using the NSGA-II algorithm for Map-I (Figure 5a) are presented in Figure 7 and

Table 6. Optimization results for Map-I using the NSGA-II algorithm.

Sl.NO	Hinge Angles $({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3)$ (Degree)	Area Covered (Pc) in %	Path Length (Plen)
1	(97, 12, 150)	86.9368	68
2	(159, 20, 152)	94.9972	92
3	(53, 39, 167)	92.5272	85
4	(62, 96, 65)	82.5484	59
5	(73, 90, 124)	85.1824	66
6	(51, 114, 136)	83.6408	65
7	(70, 27, 136)	90.6232	75
8	(55, 17, 3)	95.8784	107

.

By analyzing the results, it is evident that our RL approach outperforms the multi-objective optimization in all considered scenarios. For example, the morphology with the hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 55,17,3$) has a percentage of area coverage ($P_c$) and path length of $\left(P_{len}\right)$ of 95.8784% and 107, respectively (Table 6, row 8). Whereas, the morphology generated by reinforcement learning has a lower path length and higher % of area covered. For example, morphology with the hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 105,51,81$) (Table 2, row 4) has an $P_c$ of 97.422 and an $P_{len}$ of 92. This is better than the morphology generated by the NSGA-II algorithm. Similarly, another morphology set produced by the optimization algorithm with hinge angles of (${\theta }_1,{\theta }_2,{\theta }_3\equiv 97,12,150$) has a lesser path length of ($P_{len}$) of 68, respectively (Table 6, row 1). However, the % of the area covered is only ($P_c$ = 86.9368%), making it a sub-optimal morphology for area coverage tasks. This indicates that the morphology generated by the reinforcement learning algorithm is superior than optimization.

We can conclude that our approach generates morphology with higher area coverage and consumes less energy, making it a promising choice for real-world applications, which performs better than a metaheuristic optimization scheme.

6.6. Limitations

Despite the promising results of our work, there are some limitations to our approach:

• The proposed framework is only applied and validated in the reconfigurable class of tiling robot $Smorphi$. The results may differ for different classes of robots.
• The current model is trained in a static environment. It does not consider dynamic or unpredictable environments. Therefore, the effectiveness of our proposed approach in such scenarios remains an open question.
• The proposed framework is computationally expensive, as it requires training a reinforcement learning agent. This may limit its applicability to real-world applications with tight time constraints.
• Our simulation-based study does not explicitly consider real-world physical constraints and hardware limitations

While the above limitations exist in the work, we believe that the proposed framework is a valuable contribution to the field of reconfigurable robots. It provides a systematic approach for generating optimal morphologies that can be used to improve the performance of reconfigurable robots in a variety of tasks.

7. Conclusions

The study presented in this paper demonstrates the potential of deep reinforcement learning (RL) in generating optimal morphologies for reconfigurable robots. Our findings reveal that the design of the Smorphi robot, inspired by design principles, enables the robot to produce an array of configurations, which can be exploited to perform energy-efficient area coverage. Furthermore, our study shows that the PPO algorithm outperforms the A3C algorithm in generating energy-efficient optimal morphologies for a given map. The PPO algorithm showcases a consistent learning process, with a continuous increase in the mean rewards. In contrast, the A3C algorithm displayed inconsistent learning patterns due to its asynchronous nature, causing periodic fluctuations.

Our comparison of RL-generated morphologies with random morphologies revealed that the morphologies produced by the RL algorithm are superior in performance. This further reveals the advantage of exploring a vast array of configurations with the RL algorithm. Policy validation revealed that agents trained on one map could efficiently adapt and converge more quickly on an unseen map, underscoring RL’s scalability and its ability to learn and transfer across varying environments. Additionally, the comparative study with the metaheuristic optimization scheme NSGA-II revealed that the morphology produced by the RL algorithm dominated the Pareto Frontier results produced by the NSGA-II algorithm.

The proposed approach in this paper has some limitations, as outlined in Section 6.6. These limitations suggest avenues for future work to enhance the framework’s applicability and robustness across various scenarios and conditions. Furthermore, future work aims to explore the potential of multi-robot collaborative learning for optimal morphology generation using reinforcement learning and to validate the proposed approach in a real-world scenario. Additionally, we aim to explore the sensitivity of our approach to different hyperparameters. This could involve conducting a sensitivity analysis to assess how variations in hyperparameters, such as the generalized advantage estimation (GAE) parameter and discount factor, affect the performance of our method. Such an analysis can provide valuable insights into the robustness and adaptability of our approach under different settings. We will extend the proposed framework to generate optimal robot design using the generative adversarial network (GAN). By incorporating GANs into the design process, optimal and novel reconfigurable robot designs can be generated.