Path Tracking Control for Four-Wheel Independent Steering and Driving Vehicles Based on Improved Deep Reinforcement Learning

Abstract

We propose a compound control framework to improve the path tracking accuracy of a four-wheel independent steering and driving (4WISD) vehicle in complex environments. The framework consists of a deep reinforcement learning (DRL)-based auxiliary controller and a dual-layer controller. Samples in the 4WISD vehicle control framework have the issues of skewness and sparsity, which makes it difficult for the DRL to converge. We propose a group intelligent experience replay (GER) mechanism that non-dominantly sorts the samples in the experience buffer, which facilitates within-group and between-group collaboration to achieve a balance between exploration and exploitation. To address the generalization problem in the complex nonlinear dynamics of 4WISD vehicles, we propose an actor-critic architecture based on the method of two-stream information bottleneck (TIB). The TIB method is used to remove redundant information and extract high-dimensional features from the samples, thereby reducing generalization errors. To alleviate the overfitting of DRL to known data caused by IB, the reverse information bottleneck (RIB) alters the optimization objective of IB, preserving the discriminative features that are highly correlated with actions and improving the generalization ability of DRL. The proposed method significantly improves the convergence and generalization capabilities of DRL, while effectively enhancing the path tracking accuracy of 4WISD vehicles in high-speed, large-curvature, and complex environments.

1. Introduction

Four-wheel independent steering and driving (4WISD) vehicles have gained considerable research attention owing to their highly flexible path tracking control capabilities [1,2,3,4,5]. This control independence enhances the vehicle’s adaptability and robustness in complex environments, enabling it to effectively respond to variable road conditions and disturbances [6,7,8]. However, the complex nonlinear characteristics inherent to 4WISD vehicles present significant challenges in designing path tracking controllers that exhibit fast response, high tracking accuracy, and strong disturbance rejection [9].

In recent years, various control algorithms have been proposed to address these challenges in the path tracking control of 4WISD vehicles. One widely used approach is motion decoupling, which separates lateral and longitudinal motions through independent control loops to track different targets [10,11,12]. This method simplifies the design of the control system, allowing for more precise and efficient control in each motion direction. Another key strategy is the hierarchical control structure, where the upper-layer controller computes the generalized forces for the vehicle, and the lower-layer controller optimally distributes these forces to each tire [13,14,15,16]. This hierarchical structure effectively manages the complex vehicle dynamics, improving both the robustness and response speed of the control system.

In terms of control algorithms, sliding mode control (SMC) has been widely applied due to its robustness against system uncertainties and external disturbances [17,18]. This introduces a sliding surface along which the system states move, to achieve effective control of nonlinear systems. Model predictive control (MPC) has also gained significant attention [19,20] for its ability to handle multivariable coupling and constraint conditions by predicting future system behaviors to optimize current control inputs. Proportional-integral-derivative (PID) control and its improved versions remain vital in engineering practice due to their simplicity and effectiveness [21,22]. However, these traditional path tracking control methods have limitations in fully addressing the complex challenges posed by 4WISD vehicles. Traditional model-based control methods pose challenges in managing unknown and highly dynamic external disturbances. These methods rely on a predefined vehicle dynamics model, which may fail to converge under drastic or unexpected environmental changes. As a result, these methods lack real-time adaptability to nonlinear and unpredictable conditions in 4WISD vehicles. In addition, the computational demands can become prohibitive when adjusting to such complex scenarios. Numerical errors tend to accumulate and propagate with each recursive step, which might not only decrease the computational accuracy of the system but also lead to divergence in control algorithms. In path tracking control, as the iteration number increases, the errors are gradually magnified, reducing control precision, especially when the unknown external disturbances strengthen the error accumulation effect. The randomness and unpredictability of external disturbances force the system to adjust control signals, further amplifying numerical errors and reducing the system’s stability. Ultimately, the continuous accumulation of these errors could destabilize the system, significantly reducing path tracking accuracy and increasing the operational risk in dynamic environments.

Model-free methods for vehicle control have been applied to address the above-mentioned challenges [23,24,25,26,27]. Deep reinforcement learning (DRL) combines the advantages of reinforcement learning and deep learning, demonstrating great capability in real-time control [28,29]. Within the DRL framework, agents are modeled using deep neural networks and continuously interact with the environment and control system. Each action is rewarded based on feedback from the environment, and the training objective is to maximize the cumulative reward over time. Through interaction with continuous state-action spaces [30], DRL can adapt to the decision-making requirements of 4WISD vehicles under different external disturbances. While DRL provides flexibility and adaptability by interacting with continuous state-action spaces, it faces substantial challenges in control stability, convergence speed, and generalization to varying environments. These challenges stem from the tendency of DRL agents to overfit to the specific conditions on which they are trained, limiting their ability to generalize to diverse scenarios. Furthermore, DRL’s dependence on large datasets for training can result in instability during the learning process. This issue is especially pronounced for 4WISD vehicles in dynamic or high-dimensional environments.

Based on the literature review, we propose a compound control framework for path tracking of 4WISD vehicles that uses an improved DRL approach. The main novelties of this study are as follows. Firstly, we propose a compound control framework that enhances DRL’s adaptability and generalization by integrating a model-free DRL auxiliary controller with a model-based dual-layer controller. This compound approach capitalizes on the DRL’s strengths in managing complex, high-dimensional environments, while the model-based controller ensures stability and improves decision-making in dynamic or unforeseen scenarios. As a result, the system is capable of generalizing across diverse operational conditions, maintaining accurate path tracking even under highly uncertain external disturbances. Secondly, we propose a group intelligent experience replay (GER) mechanism that treats the experience buffer as an intelligent entity. The experience buffer is categorized into three groups based on the prioritization of samples: discover, joiner, and risker. Coordination within and between groups is performed using training progress and non-dominated sorting, enabling adaptive balancing of exploration and exploitation. This categorization enables the DRL agent to efficiently explore new strategies in unknown environments, while refining its learned policies in known situations. This leads to faster convergence, enabling the system to manage diverse operational scenarios with higher accuracy and stability. Thirdly, an actor-critic architecture based on the two-stream information bottleneck (TIB) is proposed. An information bottleneck approach is introduced into the critic network to minimize the mutual information between high-dimensional features and the target Q-value, thereby improving the critic’s feature extraction ability and reducing generalization error. Meanwhile, a reverse information bottleneck approach is applied to the actor network to maximize the mutual information between features and actions. This approach balances learning the most compact state representation and preserving highly discriminative state-action correlations, and ensures that the DRL agent can generalize its learned policies to a wide range of unknown scenarios, improving its ability to adapt to diverse high-dimensional environments.

The remainder of this paper is structured as follows. The research background and related work are introduced in Section 2. The problem formulation is presented in Section 3. The implementation of the improved DRL-based compound control framework is elaborated in Section 4. The simulation results are compared in Section 5. The conclusions are summarized in Section 6.

2. Background and Related Work

2.1. DRL-Based Vehicle Control

Liu et al. compared the decision-making strategies for autonomous driving on a highway using different deep reinforcement learning algorithms, focusing on their implementation methods, performance metrics, and impact on driving efficiency and safety [31]. In terms of longitudinal control, Lin et al. found that DRL performs better as errors based on model predictive control increase [32]. Chen et al. integrated the methods of deep reinforcement learning and model predictive control for adaptive cruise control (ACC) of tracked vehicles, which enhanced the performance and efficiency of the ACC system [33]. Selvaraj et al. developed a DRL framework that accounts for passengers’ safety and comfort and road usage efficiency [34]. For lateral control, Li et al. broke the vision-based lateral control system into a perception module and a control module, with the latter being trained with a reinforcement learning controller to improve performance on different tracks [35]. Peng et al. combined a DRL strategy with graph attention networks for autonomous driving planning [36]. In complex urban traffic scenarios, Li et al. used a DRL-based eco-driving strategy to optimize economy and travel efficiency. The authors also integrated safety measures and addressed additional challenges arising from real-time traffic elements, such as varying road conditions [37]. Sallab et al. proposed a DRL framework for autonomous driving to handle complex interactions with other vehicles and roadworks [38].

The complex dynamics in 4WISD vehicles make it difficult for existing DRL methods to achieve stable and efficient control. Current approaches often face challenges such as instability and poor convergence, due to the vehicle’s high-dimensional state and action spaces. Therefore, developing more robust and effective DRL-based compound control frameworks is essential to enhance the path tracking performance of 4WISD vehicles.

2.2. Experience Replay Mechanism in DRL

Recent advancements in experience replay mechanisms for deep reinforcement learning have concentrated on several key areas. For experience selection and sampling strategy optimization, various methods have been proposed to enhance learning efficiency using intelligent selection mechanisms. Wei et al. and Li et al. presented quantum-inspired experience replay (QER) to balance the importance and diversity [39,40]. Zhu et al. developed prioritized experience replay (PER) to adjust sampling probabilities based on temporal difference (TD) errors [41]. Na et al. proposed emphasized experience replay (EER) to prioritize experiences that significantly impact algorithm performance [42]. Ye et al. introduced classified experience replay (CER) to adjust sampling ratios for different types of experiences by classifying them into successful and failed attempts. This enhancement improved the training process, enabling the DRL model to learn more effectively from both positive and negative outcomes [43]. Regarding experience timeliness and freshness, Ma et al. incorporated freshness discount factors to increase the sampling probability of recent experiences [44]. Wang et al. employed annealed biased prioritized experience replay to account for experience timeliness [45]. To improve experience storage and memory management, researchers have focused on the effective utilization of limited experience storage space. Osei et al. proposed an enhanced sequential memory management (ESMM) to optimize replay memory usage by improving experience retention strategies [46]. Liu et al. developed two-dimensional replay buffers to enhance storage structures [47].

The non-stationary environments of these vehicles hinder the efficient utilization of training samples by existing methods. Insufficient adaptability to changing conditions and inefficiency in handling high-dimensional state-action spaces typically result in poor learning performance. It is necessary to develop more adaptive and robust experience replay strategies to optimize sample efficiency and enhance control performance in these complex systems.

2.3. Representation Learning in DRL

Recent advancements in information processing and representation learning are briefly introduced. For information compression and extraction, Xiang et al. employed variational information bottleneck techniques to infer fundamental tasks and learn essential skills [48]. Zou et al. developed the InfoGoal method, utilizing information bottleneck to learn compact goal representations, thereby improving policy optimality and generalization in goal-conditioned reinforcement learning [49]. Schwarzer et al. introduced the Self-Predictive Representations (SPR) approach. And the future state prediction and data augmentation were used to markedly increase sample efficiency [50]. Zhang et al. applied dual similarity metrics to learn robust latent representations that encode only task-relevant information, which demonstrated efficacy across various visual tasks [51]. In the domain of contrastive learning and self-supervision, Laskin et al. developed the CURL methods to extract high-level features from raw pixels, enhancing performance across multiple benchmarks [52]. Stooke et al. proposed the Augmented Temporal Contrast (ATC) task to decouple representation learning and policy learning, surpassing end-to-end reinforcement learning performance in most environments [53]. Regarding structured representations, Wei et al. introduced graph representation learning as an effective method for DRL agents to learn network entity relationships, enhancing path selection performance in network routing problems [54]. Qian et al. developed the DR-GRL framework, and combined disentangled representation learning with goal-conditioned visual reinforcement learning to improve sample efficiency and policy generalization [55]. For model-free reinforcement learning with high-dimensional image inputs, Yarats et al. proposed techniques to enhance training stability, demonstrating robustness to observational noise in control tasks [56].

Existing approaches often have difficulties in effectively capturing the complex dynamics or incur high computational costs, limiting their practical applicability. Developing more efficient state representation learning and improved sample information processing methods is crucial for achieving high DRL performance in controlling 4WISD vehicles.

3. Problem Formulation and Analysis

3.1. 4WISD Vehicle Dynamics Model

The 4WISD vehicle dynamics model comprises a vehicle body dynamics model and a tire model, as shown in Figure 1. The inertial reference frame and the vehicle body frame are represented as $OXYZ$ and $O_v{X}_v{Y}_v{Z}_v$, respectively. Assume that the path tracking occurs on even road. The yaw, pitch, and roll motions of the vehicle body are mainly controlled by the longitudinal and lateral forces at each tire. For ease of notation, the four tires are indexed as $i=\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}$, which represent the front-left, front-right, rear-left, and rear-right tires, respectively, as represented in Figure 1a. In the tire coordinate system, the longitudinal force, lateral force, vertical force, and steering angle of each tire are denoted by $F_{li}$,$F_{Li}$,$F_{Ni}$, and ${\delta }_i$, respectively. The dynamics of the vehicle body, which are analyzed in the vehicle coordinate $O_v{X}_v{Y}_v{Z}_v$, can be written as

$$\begin{array}{cc}\hfill & \left\{\begin{array}{cc}\hfill M({\dot{v}}_x-v_y\gamma )& =\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}(F_{li}cos{\delta }_i-F_{Li}sin{\delta }_i)\hfill \\ \hfill M({\dot{v}}_y+v_x\gamma )& =\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}(F_{li}sin{\delta }_i+F_{Li}cos{\delta }_i)\hfill \\ \hfill I_{\gamma }\dot{\gamma }& =\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}\left[L_{bi}(F_{li}cos{\delta }_i-F_{Li}sin{\delta }_i)\right.+\left.l_{bi}(F_{li}sin{\delta }_i+F_{Li}cos{\delta }_i)\right]\hfill \end{array}\right.\hfill \end{array}$$

(1)

where M and $I_{\gamma }$ represent the vehicle mass and the moment of inertia; $(L_{bi},l_{bi})$ denote the location of each tire in the coordinate $O_v{X}_v{Y}_v{Z}_v$, with the center tied to the center of gravity of the vehicle; and ${\dot{v}}_x$, ${\dot{v}}_y$, and $\dot{\gamma }$ represent the vehicle’s longitudinal acceleration, lateral acceleration, and yaw angle acceleration, respectively.

The vehicle accelerations in the vehicle body coordinate system are calculated through the forces exerted by the tires. In the design of a path tracking controller, the longitudinal and lateral forces at each tire are determined by controlling the steering angles and wheel torques, as presented in the tire dynamics model in Figure 1b.

In the tire coordinate system, we utilize the Magic Formula tire model to accurately capture the complex nonlinear characteristics of forces under varying slip ratios and slip angles. These nonlinear characteristics arise from the relationships between the longitudinal and lateral tire forces and their slip ratios and slip angles, which are influenced by factors such as tire deformation, rubber hysteresis effects, and variations in the contact patch.

The Magic Formula model employs a set of empirical equations to effectively describe these nonlinear relationships, thereby enhancing the performance of the vehicle control system and aiding in the optimization of the vehicle’s handling and stability. The Magic Formula model allows for adjustments according to different tire types and operating conditions, providing strong adaptability. The Magic Formula is well suited for real-time control systems in complex dynamic environments and can be expressed as [58]

$$y=D\sin \left(C\arctan \left\{Hx-E\left[Hx-\arctan \left(Hx\right)\right]\right\}\right)$$

(2)

where y represents either the longitudinal force $F_{li}$ or the lateral force $F_{Li}$, and x represents either the longitudinal slip ratio ${\lambda }_i$ or the slip angle ${\alpha }_i$, respectively. The empirical equation is based on the following curve-fitting coefficients: the peak factor D, the stiffness factor H, the shape factor C, and the curvature factor E.

The longitudinal slip ratio ${\lambda }_i$ is calculated as

$$\begin{array}{cc}\hfill & {\lambda }_i=\left\{\begin{array}{cc}(R_w{\omega }_i-u_{wi})/\left(R_w{\omega }_i\right)\hfill & R_w{\omega }_i\ge u_{wi}\hfill \\ (R_w{\omega }_i-u_{wi})/u_{wi}\hfill & R_w{\omega }_i<u_{wi}\hfill \end{array}\right.\hfill \end{array}$$

(3)

where $R_w$ and ${\omega }_i$ represent the dynamic tire radius and the angular velocity, and $u_{wi}$ represents the actual speed at the center of the $i^{\mathrm{th}}$ tire. Note that $R_w{\omega }_i\ge u_{wi}$ and $R_w{\omega }_i<u_{wi}$ indicate the forward acceleration and braking of the tire, respectively. The tire speed $u_{wi}$ can be calculated as

$$\begin{array}{c}\left\{\begin{array}{c}{u}_{w,\mathrm{fl}}=(v_x-L_{bl}\gamma )\cos \left({\delta }_i\right)+(v_y+l_{bf}\gamma )\sin \left({\delta }_i\right)\hfill \\ u_{w,\mathrm{fr}}=(v_x+L_{br}\gamma )\cos \left({\delta }_i\right)+(v_y+l_{bf}\gamma )\sin \left({\delta }_i\right)\hfill \\ u_{w,\mathrm{rl}}=(v_x-L_{bl}\gamma )\cos \left({\delta }_i\right)+(v_y-l_{br}\gamma )\sin \left({\delta }_i\right)\hfill \\ u_{w,\mathrm{rr}}=(v_x+L_{br}\gamma )\cos \left({\delta }_i\right)+(v_y-l_{br}\gamma )\sin \left({\delta }_i\right)\hfill \end{array}\right.\hfill \end{array}$$

(4)

The longitudinal and lateral dynamics models of the tire can be written as

$$\begin{array}{cc}\hfill & \left\{\begin{array}{c}{T}_i=(F_{li}-F_{Ni}{f}_w)R_w+{\omega }_i{I}_w\hfill \\ {\delta }_i=\arctan ((v_y\pm {\omega }_i{l}_{bi})/(v_x\pm {\omega }_i{L}_{bi}))-{\alpha }_i\hfill \end{array}\right.\hfill \end{array}$$

(5)

where $T_i$ denotes the drive torque, $f_w$ denotes the rolling friction coefficient, $I_w$ denotes the moment of inertia of each tire, and $F_{Ni}$ denotes the tire’s vertical force.

Based on the above analysis, we have established a complex vehicle dynamics model with seven degrees of freedom, which has a maximum of eight controllable inputs (i.e., four drive torques and four steering angles). By properly allocating the input parameters, we aim to effectively control the vehicle state to yield high path tracking performance.

The control input matrix of the vehicle dynamics model is given by

$$\begin{array}{cc}\hfill & \mathbf{U}=\left[\begin{array}{cccccccc}{T}_{\mathrm{fl}}& T_{\mathrm{fr}}& T_{\mathrm{rl}}& T_{\mathrm{rr}}& {\delta }_{\mathrm{fl}}& {\delta }_{\mathrm{fr}}& {\delta }_{\mathrm{rl}}& {\delta }_{\mathrm{rr}}\end{array}\right]\hfill \end{array}$$

(6)

where the matrix $\mathbf{U}$ represents the torque and steering angle of the front left, front right, rear left, and rear right wheels, respectively. The vehicle state matrix is defined as $\mathbf{X}$:

$$\begin{array}{cc}\hfill & \mathbf{X}=\left[\begin{array}{ccc}{v}_x& v_y& \gamma \end{array}\right]\hfill \end{array}$$

(7)

where $v_x$, $v_y$, and $\gamma$ denote the vehicle’s longitudinal velocity, lateral velocity, and yaw velocity. To facilitate the design of DRL-based auxiliary controllers, the dynamic equations of the 4WISD vehicle can be rewritten into an affine nonlinear matrix form:

$$\left\{\begin{array}{c}\dot{\mathbf{X}}={\widehat{\mathbf{A}}}_n+\mathbf{Bu}\hfill \\ \mathbf{u}=\mathbf{CU}\hfill \end{array}\right.$$

(8)

where $\mathbf{U}$ represents the control output matrix of the dual-layer controller;

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{A}}}_n=\left[\begin{array}{ccc}{\dot{v}}_{nx}& {\dot{v}}_{ny}& {\dot{\gamma }}_n\end{array}\right]\hfill \end{array}$$

(9)

denotes the acceleration disturbance matrix of the vehicle body; $\mathbf{C}$ indicates the mapping matrix from the control quantities of the upper-layer controller to the control quantities of the lower-layer controller; and $\mathbf{B}$ indicates the inversion of vehicle mass matrix.

Since the vehicle’s motion behavior is primarily determined by the interaction forces between the tires and the road surface, as the vehicle is a multi-body system where each tire may experience different external disturbances, it is critical to consider disturbances at the individual tire level to accurately capture the vehicle’s dynamics. To realistically reflect the impact of external disturbances, we introduce a control-end force disturbance matrix:

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{F}}}_n=\left[\begin{array}{cccccccc}{\widehat{F}}_{l,\mathrm{fl}}& {\widehat{F}}_{l,\mathrm{fr}}& {\widehat{F}}_{l,\mathrm{rl}}& {\widehat{F}}_{l,\mathrm{rr}}& {\widehat{F}}_{L,\mathrm{fl}}& {\widehat{F}}_{L,\mathrm{fr}}& {\widehat{F}}_{L,\mathrm{rl}}& {\widehat{F}}_{L,\mathrm{rr}}\end{array}\right]\hfill \end{array}$$

(10)

which corresponds to the perturbed lateral and longitudinal forces acting on each tire. Note that ${\widehat{F}}_{l,i}$ and ${\widehat{F}}_{L,i}$ represent the longitudinal disturbance force and the lateral disturbance force acting on each tire, respectively. The external disturbances in Equation (10) can propagate through the tire dynamics model to the vehicle body dynamics model, resulting in the acceleration disturbance matrix ${\widehat{\mathbf{A}}}_n$. This method allows for a direct and accurate representation of how external disturbances affect the vehicle’s dynamics and supports the development of more robust controllers to enhance system resilience against external uncertainties.

3.2. Transition Model for DRL

The state variable $\mathbf{s}$ of the 4WISD vehicle includes measurable states $\mathbf{o}$ and unmeasurable disturbances $\mathbf{d}$. We employ DRL as an auxiliary controller rather than for direct end-to-end vehicle control, and assume that the disturbance $\mathbf{d}$ is known, thus utilizing the Markov decision process $p\left(\dot{\mathbf{s}}\right|\mathbf{s}=\mathbf{o},\mathbf{a})$ to facilitate the training of the DRL [47]. In terms of the auxiliary controller in path tracking control of 4WISD vehicles, the optimization problem can be formulated as follows:

$$\begin{array}{cc}\hfill & \begin{array}{cccc}\hfill & \hfill {\mathbf{a}}^{*}=& \hfill & argmin\sum _{i=1}^N{∥{\mathbf{X}}^{*}-f_H\left({\mathbf{X}}_i,f_l\left([{\mathbf{X}}_i,f_a({\mathbf{s}}_i,{\mathbf{a}}_i)+f_{u_c}({\mathbf{e}}_i,{\mathbf{u}}_{ci})],{\mathbf{U}}_i\right)\right)∥}^2\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & {\mathbf{a}}_{\mathrm{lb}}\le {\mathbf{a}}_i\le {\mathbf{a}}_{\mathrm{ub}}\hfill \\ \hfill & \hfill & & {\mathbf{u}}_{\mathrm{lb}}\le {\mathbf{u}}_i\le {\mathbf{u}}_{\mathrm{ub}}\hfill \\ \hfill & \hfill & & {\mathbf{U}}_{\mathrm{lb}}\le {\mathbf{U}}_i\le {\mathbf{U}}_{\mathrm{ub}}\hfill \\ \hfill & \hfill & & {\mathbf{s}}_{\mathrm{lb}}\le {\mathbf{s}}_i\le {\mathbf{s}}_{\mathrm{ub}}\hfill \\ \hfill & \hfill & & {\mathbf{e}}_{\mathrm{lb}}\le {\mathbf{e}}_i={∥{\mathbf{X}}^{*}-f_H\left({\mathbf{X}}_i,f_l\left([{\mathbf{X}}_i,f_a({\mathbf{s}}_i,{\mathbf{a}}_i)+f_{u_c}({\mathbf{e}}_i,{\mathbf{u}}_{ci})],{\mathbf{U}}_i\right)\right)∥}^2\le {\mathbf{e}}_{\mathrm{ub}}\hfill \end{array}\hfill \end{array}$$

(11)

where $f_H$ denotes the dynamic function of the 4WISD vehicle; $f_a$ denotes the auxiliary controller; $f_{u_c}$ denotes the upper-layer controller; $f_l$ denotes the lower-layer controller; ${\mathbf{X}}^{*}=\left[\begin{array}{ccc}{v}_{x_{\mathrm{ref}}}& v_{y_{\mathrm{ref}}}& {\gamma }_{\mathrm{ref}}\end{array}\right]$ represents the reference path states of the longitudinal velocity, lateral velocity, and yaw velocity obtained from the lateral displacement Y and yaw angle $\varphi$ through differential operation, respectively; and $\mathbf{a}$ represents the DRL action. Since the DRL auxiliary controller is expected to achieve real-time control of the vehicle based on the vehicle’s current state, we define the state of the DRL as

$$\begin{array}{cc}\hfill & \mathbf{s}=\left[\begin{array}{cccccc}{v}_x& v_y& \gamma & \mathbf{e}& {\mathbf{u}}_c& {\widehat{\mathbf{A}}}_n\end{array}\right]\hfill \end{array}$$

(12)

where $\mathbf{e}=\left[\begin{array}{ccc}{e}_{v_x}& e_Y& e_{\varphi }\end{array}\right]$ denotes the errors in longitudinal velocity, lateral displacement, and heading angle between the current and ideal vehicle states. The output action $\mathbf{a}$ of the DRL auxiliary controller has the same form as the control variable $\mathbf{U}$ of the vehicle system:

$$\begin{array}{cc}\hfill & \mathbf{a}=\left[\begin{array}{ccc}{F}_{ax}& F_{ay}& M_{a\varphi }\end{array}\right]\hfill \end{array}$$

(13)

The reward function is designed as

$$\begin{array}{cc}\hfill & r=\left\{\begin{array}{cc}-\left(K_r\mathbf{e}{\mathbf{e}}^T+K_f\dot{\mathbf{a}}{\dot{\mathbf{a}}}^T\right)\hfill & \parallel \mathbf{e}\parallel <{\mathbf{e}}_{\mathrm{ub}}\mathrm{and}{\mathbf{s}}_{\mathrm{lb}}\le {\mathbf{s}}_i\le {\mathbf{s}}_{\mathrm{ub}}\hfill \\ -K_b\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(14)

where $K_r$ is the positive gain parameter for the path tracking error, $K_f$ is the positive gain parameter for the stability of continuous control, ${\mathbf{e}}_{\mathrm{ub}}$ denotes the threshold for tracking error, $K_b$ is the penalty term, and $\dot{\mathbf{a}}$ represents the rate of change of time in the auxiliary control variable.

4. Compound Control Framework Based on Improved DRL

The twin-delayed deep deterministic policy gradient algorithm (TD3) employs two independent Q-networks and adopts the smaller Q-value for policy updates, thereby suppressing overestimation. Moreover, TD3’s deterministic policy gradient is suitable for continuous action space problems [59]. This algorithm provides enhanced stability for path tracking control of 4WISD vehicles. In what follows, the improved control framework is integrated with TD3.

4.1. Compound Control Framework

To address the issue of lateral and longitudinal coupling in path tracking control of 4WISD vehicles, we employ a dual-layer control framework in the model-based controller, which consists of an upper-layer controller based on nonlinear model predictive control (NMPC) and a lower-layer controller based on sequential quadratic programming (SQP).

The current state error $\mathbf{e}$ of the vehicle is calculated using the reference path information and the current vehicle state. In the upper-layer controller, the NMPC algorithm is applied with the cost function defined as follows:

$$\begin{array}{cc}\hfill & \begin{array}{cccc}\hfill & J\left(t_k\right)\hfill & \hfill & =min\left[\sum _{n=1}^N{∥\mathbf{e}\left(t_{k+n}\right|t_k)∥}_{{\mathbf{Q}}_{\left(k\right)}}^2+\sum _{n=1}^N{∥\Delta {\mathbf{u}}_c\left(t_{k+n}\right|t_k)∥}_{{\mathbf{R}}_{\left(k\right)}}^2+\rho {ϵ}^2\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & F_{x,\mathrm{lb}}\le F_x\le F_{x,\mathrm{ub}}\hfill \\ \hfill & \hfill & & F_{y,\mathrm{lb}}\le F_y\le F_{y,\mathrm{ub}}\hfill \\ \hfill & \hfill & & M_{\varphi ,\mathrm{lb}}\le M_{\varphi }\le M_{\varphi ,\mathrm{ub}}\hfill \end{array}\hfill \end{array}$$

(15)

where ${\mathbf{Q}}_{\left(k\right)}$ and ${\mathbf{R}}_{\left(k\right)}$ represent the state weighting matrix and the control increment weighting matrix, $ϵ$ denotes the relaxation factor to avoid the absence of feasible solutions, $\mathbf{e}\left(t_{k+n}\right|t_k)$ represents the error state at time $t_k$, and ${\mathbf{u}}_c\left(t_{k+n}\right|t_k)$ represents the generalized control quantities at the time $t_{k+n}$. Note that the generalized control quantities ${\mathbf{u}}_c$ are determined to calculate the vehicle state at the future moment:

$$\begin{array}{cc}\hfill & {\mathbf{u}}_{\mathit{c}}=\left[\begin{array}{ccc}{F}_{cx}& F_{cy}& M_{c\varphi }\end{array}\right]\hfill \end{array}$$

(16)

where $F_{cx}$, $F_{cy}$, and $M_{c\varphi }$ represent the generalized longitudinal force, lateral force, and yaw moment of the vehicle body, respectively.

Then, we allocate the generalized force ${\mathbf{u}}_c$ from the upper-layer controller into the lateral and longitudinal forces for each tire. A lower-layer controller based on SQP is used to achieve the optimal distribution of the generalized force ${\mathbf{u}}_c$. The cost function in the SQP algorithm is defined as

$$\begin{array}{cc}\hfill & \begin{array}{cccc}\hfill & f\hfill & \hfill & =min\left[w_1\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}{\displaystyle \frac{F_{li}^2+F_{Li}^2}{{\mu }^2{F}_{Ni}^2}}+w_2{\left(max\left({\displaystyle \frac{F_{li}^2+F_{Li}^2}{{\mu }^2{F}_{Ni}^2}}\right)-min\left({\displaystyle \frac{F_{li}^2+F_{Li}^2}{{\mu }^2{F}_{Ni}^2}}\right)\right)}^2\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & F_{l,\mathrm{lb}}\le F_{li}\le F_{l,\mathrm{ub}},F_{L,\mathrm{lb}}\le F_{Li}\le F_{L,\mathrm{ub}}\hfill \\ \hfill & \hfill & & 0\le {\displaystyle \frac{F_{li}^2+F_{Li}^2}{{\mu }^2{F}_{Ni}^2}}\le 1\hfill \\ \hfill & \hfill & & F_x=\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}\left(F_{li}cos{\delta }_i-F_{Li}sin{\delta }_i\right)\hfill \\ \hfill & \hfill & & F_y=\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}\left(F_{li}sin{\delta }_i+F_{Li}cos{\delta }_i\right)\hfill \\ \hfill & \hfill & & M_{\varphi }=\sum _i^{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}}\left[L_{bi}(F_{li}cos{\delta }_i-F_{Li}sin{\delta }_i)\right.+\left.l_{bi}(F_{li}sin{\delta }_i+F_{Li}cos{\delta }_i)\right]\hfill \end{array}\hfill \end{array}$$

(17)

where $w_1$ and $w_2$ represent the weighting coefficients for tire balancing. Using the tire dynamics model, the lateral and longitudinal forces on each tire can be transformed into the drive torques and steering angles, resulting in the control variable $\mathbf{U}$ at the next moment. Then, the control variable is transmitted to the vehicle body dynamics model to achieve closed-loop path tracking control of 4WISD vehicles.

Even though the aforementioned dual-layer controller can be used to decouple the longitudinal and lateral motion, the control capability for a 4WISD vehicle subjected to external disturbances is still limited. The DRL-based controller, which is a data-driven algorithm, possesses super adaptability for the improvement of control performance. As depicted in Figure 2, a compound control framework is proposed, which mainly consists of a model-based dual-layer control loop and a model-free DRL-based auxiliary control loop. The current vehicle state is input into the upper-layer controller to obtain the required generalized forces for the next time step. Simultaneously, a well-trained DRL policy network produces an extra control term $\mathbf{a}$ based on the vehicle state information $\mathbf{s}$ for compensation. As such, a new upper layer control variable $\mathbf{u}$ is obtained as

$$\begin{array}{cc}\hfill & \mathbf{u}={\mathbf{u}}_c+\mathbf{a}=\left[\begin{array}{ccc}{F}_{cx}+F_{ax}& F_{cy}+F_{ay}& M_{c\varphi }+M_{a\varphi }\end{array}\right]\hfill \end{array}$$

(18)

By integrating the DRL-based auxiliary controller with the upper-layer controller, the lower-layer controller transforms the generalized force vector $\mathbf{u}$ into an end control variable $\mathbf{U}$. The control stability proof of the proposed compound control framework is given in Appendix A.

4.2. Group Intelligent Experience Replay

The experience replay mechanism in DRL stores transition samples obtained from the agent’s interactions with the environment in a replay buffer and randomly extracts small batches of samples to update the parameters of the value network or policy network. This process can break the temporal correlation between samples to facilitate the convergence of the agent.

To improve the stability and convergence of the DRL training process and avoid becoming trapped into local optima, the principles of group intelligence optimization are incorporated into the experience replay mechanism. Data in the experience buffer are regarded as an intelligent group [60], and the samples in the experience buffer are divided into three distinct functional groups based on the TD error and advantage function value. Non-dominated sorting and training progress are then introduced for within-group and between-group collaboration to optimize data replay and storage [61]. The discover group provides a better direction for the convergence of the DRL training process by prioritizing the learning of novel state-action pairs. The joiner group focuses on replaying a higher proportion of excellent samples around the discover group, reinforcing and optimizing known high-value policies. The risker group filters out low-quality samples to avoid learning from high-risk samples that may cause overfitting.

During the training process, a new transition sample $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1})$ is obtained from the interaction between the agent and the environment, from which the TD error ${\tau }_i$ and advantage function value $A_i$ are calculated as

$$\begin{array}{}\mathrm{(19)}& {\tau }_i=r_i+\gamma Q_{{\varphi }^{\prime }}({\mathbf{s}}_{i+1},{\mu }_{{\theta }^{\prime }}\left({\mathbf{s}}_{i+1}\right))-Q_{\varphi }({\mathbf{s}}_i,{\mathbf{a}}_i)\hfill \mathrm{(20)}& A_i=Q_{\varphi }({\mathbf{s}}_i,{\mathbf{a}}_i)-V_{\varphi }\left({\mathbf{s}}_i\right)\hfill \end{array}$$

where $Q_{\varphi }$ and ${\mu }_{{\theta }^{\prime }}$ represent the target critic network and actor network, respectively, and $V_{\varphi }$ represents the state value function. Subsequently, the initial priority $P_i$ of the $i^{\mathrm{th}}$ sample is calculated as

$$P_i=|{\tau }_i|+\lambda |A_i|$$

(21)

The sample $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1})$ and the associated parameters $P_i$, ${\tau }_i$, and $A_i$ are stored in the experience replay buffer $\Omega$. When the replay criteria are met, the samples in the experience replay buffer are divided into three groups based on priority $P_i$. As shown in Figure 3, the sample ratio coefficients for each group are denoted as $S_{\alpha 0}$, $S_{\beta 0}$, and $S_{\gamma 0}$, respectively, and they must satisfy the following conditions for $S_{\alpha 0},S_{\beta 0},S_{\gamma 0}\in [0,1]$, and $S_{\alpha 0}+S_{\beta 0}+S_{\gamma 0}=1$:

(1)

Group $D_{\mathrm{discover}}$: Samples with priorities in the top $S_{\alpha 0}$ percentile.

(2)

Group $D_{\mathrm{joiner}}$: Samples with priorities in the top $S_{\alpha 0}+S_{\beta 0}$ percentile but not in the top $S_{\alpha 0}$ percentile.

(3)

Group $D_{\mathrm{risker}}$: Samples with priorities in the bottom $S_{\gamma 0}$ percentile.

To effectively and accurately assess the sample quality without excessive time consumption, we select the top two mini-batch samples from each group $D_k$ based on priority $P_i$. These samples are used for non-dominated sorting to obtain the Pareto frontier $F_k$ for each group, which enables within-group collaboration:

$$F_k=\left\{s_i\in D_k\mid \forall s_j\in D_k,{\tau }_j<{\tau }_i\mathrm{and}{A}_j<A_i\right\}$$

(22)

where $k=\mathrm{discover},\mathrm{joiner}$, or risker.

After that, we use the Pareto frontier $F_k$ to ensure that the following updated priorities are ranked higher than those from other samples:

$${\overline{P}}_i=P_i+\min P_k,i\in F_k$$

(23)

The sampling proportions for the three groups are dynamically adjusted for between-group collaboration according to the current training step t and the total number of training steps T. To prevent probability overflow, the priorities in each group are normalized as

$$\begin{array}{cc}\hfill & \left\{\begin{array}{c}{N}_{\alpha }=\exp \left({\mathrm{avg}}_{\mathrm{discover}}\right)/\sum _i\exp ({\mathrm{avg}}_i)\hfill \\ N_{\beta }=\exp \left({\mathrm{avg}}_{\mathrm{joiner}}\right)/\sum _i\exp ({\mathrm{avg}}_i)\hfill \\ N_{\gamma }=\exp \left({\mathrm{avg}}_{\mathrm{risker}}\right)/\sum _i\exp ({\mathrm{avg}}_i)\hfill \end{array}\right.\hfill \end{array}$$

(24)

where ${\mathrm{avg}}_i$ represents the average priority of samples within group $i\in \left(\mathrm{discover},\mathrm{joiner},\mathrm{risker}\right)$, and $N_{\alpha }$, $N_{\beta }$, and $N_{\gamma }$ represent the normalization factors. Then, the updated priorities are defined as

$$\begin{array}{cc}\hfill & \left\{\begin{array}{c}{S}_{\alpha t}=S_{\alpha 0}(1-t/T)N_{\alpha }\hfill \\ S_{\beta t}=(S_{\beta 0}(1-t/T)+S_{\alpha 0}(t/T))N_{\beta }\hfill \\ S_{\gamma t}=(1-S_{\alpha 0}-S_{\beta 0})N_{\gamma 0}\hfill \end{array}\right.\hfill \end{array}$$

(25)

where $S_{\alpha t}$, $S_{\beta t}$, and $S_{\gamma t}$ denote the updated sampling proportions for three groups, respectively. Samples are drawn from $D_{\mathrm{discover}}$, $D_{\mathrm{joiner}}$, and $D_{\mathrm{risker}}$ according to the values of $S_{\alpha t}$, $S_{\beta t}$, and $S_{\gamma t}$ to obtain replay samples. In the initial stage of training, $S_{\beta t}$ is large, which allows for the exploration of new and valuable state-action pairs. As training progresses, $S_{\beta t}$ gradually decreases while $S_{\alpha t}$ increases, which progressively transfers valuable samples discovered during exploration to $D_{\mathrm{joiner}}$ for sufficient utilization. However, $S_{\gamma t}$ remains small to ensure that a low proportion of low-priority samples participates in training, which enhances the model’s generalization capability.

To correct for biases introduced by priority sampling, it is necessary to apply importance sampling correction to the loss function during training. For each sample $({\mathbf{s}}_i,{\mathbf{a}}_i,r_i,{\mathbf{s}}_{i+1})$ in a mini-batch sample, the importance weight $o_i$ is computed as follows:

$$o_i={(P_i/D)}^{-\zeta }$$

(26)

where D represents the sum of priorities of all samples, and $\zeta$ controls the intensity of the correction. Applying the importance weight $o_i$ to the loss function yields a weighted loss function:

$$L_w={\displaystyle \frac{1}{E}}\sum _i^E{o}_i{L}_i$$

(27)

where $L_i$ represents the mean squared error loss function for the $i^{\mathrm{th}}$ sample, and E represents the number of the mini-batch sample.

At the end of each training step, it is essential to update the sample priorities and substitute these updated priorities into the experience replay buffer. The proposed group intelligence experience replay mechanism adjusts sample priorities, balances exploration and exploitation, and enhances sample utilization efficiency. The proposed method provides greater sample utilization efficiency compared to traditional random experience replay. By employing intelligent sample classification and priority adjustment, GER adaptively prioritizes the most valuable samples for policy improvement, significantly enhancing the performance of reinforcement learning models in complex environments. This allows GER to better accelerate convergence and improve system robustness, particularly when dealing with sparse rewards or high-dimensional environments.

4.3. Actor-Critic Architecture Based on TIB

The integration of information bottleneck (IB) techniques into reinforcement learning establishes a self-supervised learning and an adaptive compression mechanism. By maximizing the mutual information achieved by arbitrary policies within short time windows, this approach generates a dense and immediate learning signal and addresses the sparse reward problem inherent in traditional reinforcement learning. The derivation of a lower bound on mutual information facilitates the adaptive adjustment of Lagrange multipliers, enabling maximal information compression while retaining sufficient task-relevant information. This methodology can provide an informative learning signal, and ultimately generate a compact and effective objective representation. Consequently, it significantly enhances the generalization capability and learning efficiency of goal-conditioned reinforcement learning [49]. This work provides theoretical guidance for integrating information theory into deep reinforcement learning.

In the standard actor-critic architecture, the critic network estimates the state-action value function, while the actor network generates policies. Building upon this framework, we employ the two-stream information bottleneck (TIB) [62] to enhance the algorithm’s generalization capability and policy quality, as shown in Figure 4. In the critic network, we integrate an IB module prior to Q-value estimation, which extracts a D dimensional compact representation ${\mathbf{z}}_t$ from high-dimensional features ${\mathbf{h}}_t^c$. The representation ${\mathbf{z}}_t$ minimizes mutual information with high-dimensional features while preserving essential information for Q-value estimation. For the actor network, we integrate a reverse information bottleneck (RIB) module before the policy generation module, which extracts a K-dimensional expressive representation ${\mathbf{u}}_t$ from high-dimensional features ${\mathbf{h}}_t^a$. The representation ${\mathbf{u}}_t$ maximizes mutual information with actions and retains discriminative state-action correlations. The extracted latent representations ${\mathbf{z}}_t$ and ${\mathbf{u}}_t$ are subsequently utilized in the critic and actor networks, respectively, to formulate the state-action value function and policy as follows:

$$\begin{array}{}\mathrm{(28)}& Q_{\varphi }({\mathbf{s}}_t,{\mathbf{a}}_t)=f_{\varphi }^{\left(l_c\right)}\left({\mathbf{z}}_t\left({\mathbf{h}}_t^c({\mathbf{s}}_t,{\mathbf{a}}_t)\right)\right)\hfill \mathrm{(29)}& {\pi }_{\theta }\left({\mathbf{s}}_t\right)=g_{\theta }^{\left(l_a\right)}\left({\mathbf{u}}_t\left({\mathbf{h}}_t^a\left({\mathbf{s}}_t\right)\right)\right)\hfill \end{array}$$

where $l_c$ and $l_a$ represent the number of network layers after the IB module and RIB module, respectively. The IB modules and RIB modules are simultaneously optimized in a single training session.

In the critic network, we introduce an IB module to enhance the network’s generalization and sample efficiency. By compressing redundant information from high-dimensional features and preserving essential information for Q-value estimation, the IB module enables the network to learn state-action value functions more effectively. We aim to reduce the complexity of high-dimensional inputs, extract the most relevant features, and improve the accuracy of Q-value estimates to enhance the model’s adaptability across complex environments. Based on these design objectives, we formulate the training loss function for the critic network as

$$L\left(\varphi \right)={\mathbb{E}}_{({\mathbf{s}}_t,{\mathbf{a}}_t)\sim E}\left[{(Q({\mathbf{s}}_t,{\mathbf{a}}_t)-y_t)}^2\right]+\xi \mathrm{KL}\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right)\right)$$

(30)

where $\varphi$ represents the parameters of the critic network, $y_t$ is the target value for the TD, $\xi$ is the balancing coefficients, $p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t)$ is the conditional distribution defined by the IB module, and $r\left({\mathbf{z}}_t\right)$ is the prior distribution of ${\mathbf{z}}_t$. Note that the expectation ${\mathbb{E}}_{({\mathbf{s}}_t,{\mathbf{a}}_t)\sim E}\left[{(Q({\mathbf{s}}_t,{\mathbf{a}}_t)-y_t)}^2\right]$ measures the TD error, ensuring accurate Q-value estimation, and the expectation $\mathrm{KL}\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right)\right)$ implements the IB by minimizing the Kullback–Leibler (KL) divergence between the conditional distribution of ${\mathbf{z}}_t$ and prior distribution of $r\left({\mathbf{z}}_t\right)$, and constraining the information flow from high-dimensional features to achieve feature compression and selection. After applying the aforementioned design, the critic network maintains accurate Q-value estimation while improving its generalization capability in complex and dynamic environments.

In the actor network, we introduce an RIB module to enhance the quality and expressiveness of the policy. Using the RIB module, we aim to maximize the mutual information between state representations and actions, preserve critical information necessary for policy generation, enhance the discriminative power of state representations to better differentiate the value of various actions, and improve the policy’s exploratory capabilities to generate more diverse and effective actions. Based on these design objectives, we formulate the training loss function for the actor network as

$$J\left(\theta \right)={\mathbb{E}}_{{\mathbf{s}}_t\sim E}\left[{\iota }^{t}Q({\mathbf{s}}_t,{\mathbf{a}}_t)\right]+\kappa \mathrm{KL}\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right)\right)$$

(31)

where $\theta$ refers to the parameters of the actor network, $\iota \in (0,1]$ is the discount factor, $\kappa$ is the balancing coefficients, $p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t)$ is the conditional distribution defined by the RIB module, and $r\left({\mathbf{u}}_t\right)$ is the prior distribution. In Equation (31), the expectation ${\mathbb{E}}_{{\mathbf{s}}_t\sim E}\left[{\iota }^{t}Q({\mathbf{s}}_t,{\mathbf{a}}_t)\right]$ is the standard policy gradient term, aiming to maximize the expected cumulative reward to ensure the generated policy yields high returns. The expectation $\mathrm{KL}\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right)\right)$ implements the RIB by maximizing the KL divergence between the conditional distribution of ${\mathbf{u}}_t$ and prior distribution of $r\left({\mathbf{u}}_t\right)$, thereby increasing the mutual information between the state representation ${\mathbf{u}}_t$ and actions, preserving more policy-relevant information. Through this design, the actor network is capable of generating more precise and effective policies while maintaining sufficient exploratory capabilities to adapt to complex decision-making environments.

For the IB module in the critic network, the KL divergence can be formulated as

$$\mathrm{KL}(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right))={\mathbb{E}}_{{\mathbf{z}}_t}[log\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t)\right)-log\left(r\left({\mathbf{z}}_t\right)\right)]$$

(32)

where $p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t)$ represents the posterior distribution of ${\mathbf{z}}_t$ conditioned on the given ${\mathbf{s}}_t$ and ${\mathbf{a}}_t$.

For the RIB module in the actor network, the KL divergence can be formulated as

$$\mathrm{KL}(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right))={\mathbb{E}}_{{\mathbf{u}}_t}[log\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t)\right)-log\left(r\left({\mathbf{u}}_t\right)\right)]$$

(33)

where $p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t)$ represents the posterior distribution of ${\mathbf{s}}_t$ conditioned on the given ${\mathbf{u}}_t$, and $r\left({\mathbf{u}}_t\right)$ denotes the prior distribution of ${\mathbf{u}}_t$. The standard normal distributions for $r\left({\mathbf{z}}_t\right)$ and $r\left({\mathbf{u}}_t\right)$ are used to simplify computation and ensure model stability.

We update the network parameters based on the loss function defined above, the general rules for updating the critic and actor networks are presented as follows:

$$\begin{array}{}\mathrm{(34)}& {\nabla }_{\varphi }L\left(\varphi \right)={\mathbb{E}}_{({\mathbf{s}}_t,{\mathbf{a}}_t)\sim E}\left[(Q({\mathbf{s}}_t,{\mathbf{a}}_t)-y_t){\nabla }_{\varphi }Q({\mathbf{s}}_t,{\mathbf{a}}_t)\right]+\xi {\nabla }_{\varphi }\mathrm{KL}\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right)\right)\hfill \mathrm{(35)}& {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\mathbf{s}}_t\sim E}\left[{\nabla }_{\mathbf{a}}Q({\mathbf{s}}_t,{\mathbf{a}}_t){\nabla }_{\theta }{\pi }_{\theta }\left(\mathbf{s}\right)\right]+\kappa {\nabla }_{\theta }\mathrm{KL}\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right)\right)\hfill \end{array}$$

The IB module in the critic network and RIB module in the actor network are

$$\begin{array}{}\mathrm{(36)}& {\nabla }_{\psi }\mathrm{KL}(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right))={\nabla }_{\psi }{\mathbb{E}}_{{\mathbf{z}}_t}[log\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t)\right)-log\left(r\left({\mathbf{z}}_t\right)\right)]\hfill \mathrm{(37)}& {\nabla }_{\omega }\mathrm{KL}(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right))={\nabla }_{\omega }{\mathbb{E}}_{{\mathbf{u}}_t}[log\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t)\right)-log\left(r\left({\mathbf{u}}_t\right)\right)]\hfill \end{array}$$

where ${\nabla }_{\varphi }$, ${\nabla }_{\theta }$, ${\nabla }_{\psi }$, and ${\nabla }_{\omega }$ represent the gradients with respect to the critic network, actor network, IB module, and RIB module, respectively.

To enhance the robustness and efficiency of our improved actor-critic architecture, we integrate the TD3 algorithm to mitigate overestimation bias, improve stability, and enhance exploration in complex reinforcement learning environments. We employ dual critic networks $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ with parameters ${\varphi }_1$ and ${\varphi }_2$, respectively, to combat overestimation bias. The loss functions for these networks are defined in Equation (30), the target Q-value can be formulated as

$$y_t=r_t+min(Q_{{\varphi }_1}^{\prime }({\mathbf{s}}_{t+1},{\mathbf{a}}_{t+1}),Q_{{\varphi }_2}^{\prime }({\mathbf{s}}_{t+1},{\mathbf{a}}_{t+1}))$$

(38)

where $Q_{{\varphi }_1}^{\prime }$ and $Q_{{\varphi }_2}^{\prime }$ represent target networks, and ${\mathbf{a}}_{t+1}$ is generated by the target policy network with added noise:

$${\mathbf{a}}_{t+1}={\pi }_{\theta }^{\prime }\left({\mathbf{s}}_{t+1}\right)+N_t$$

(39)

where ${\pi }_{\theta }^{\prime }$ denotes the target actor network and $N_t$ is clipped Gaussian noise.

We implement delayed policy updates to optimize the actor network every $d_{\mathrm{delay}}$ iterations for stabilization. The objective function for the actor network is Equation (31).

To ensure stability, we employ soft updates for target network parameters:

$$\begin{array}{}\mathrm{(40)}& {\varphi }_j^{\prime }\leftarrow \sigma {\varphi }_j+(1-\sigma ){\varphi }_j^{\prime },j=1,2\hfill \mathrm{(41)}& {\psi }_j^{\prime }\leftarrow \sigma {\psi }_j+(1-\sigma ){\psi }_j^{\prime },j=1,2\hfill \mathrm{(42)}& {\theta }^{\prime }\leftarrow \sigma \theta +(1-\sigma ){\theta }^{\prime }\hfill \mathrm{(43)}& {\omega }^{\prime }\leftarrow \sigma \omega +(1-\sigma ){\omega }^{\prime }\hfill \end{array}$$

where $\sigma$ is the soft update coefficient.

The proposed method can outperform other self-supervised and contrastive learning methods in enhancing generalization capability; while traditional methods aim to improve generalization by increasing sample diversity. They often introduce redundant information, leading to greater model complexity. In contrast, TIB can simplify feature representations by retaining only task-relevant information, which simultaneously improves generalization performance and reduces generalization errors. This enables the model to perform more stably and robustly in complex, dynamic environments. Finally, the complete framework of the improved GT-TD3 will interact with the path tracking self-disturbance rejection control of the 4WISD vehicle. The diagram and the pseudocode are presented in Figure 5 and Algorithm 1, respectively.

The framework mainly consists of three components: the 4WISD vehicle path tracking control environment, the actor-critic network, and the experience replay buffer. The 4WISD vehicle path tracking control environment is composed of a dual-layer controller and a dynamics model of the 4WISD vehicle, which is designed to interact with the DRL to generate training data. The DRL actor consists of an on-line actor neural network and a target actor neural network, designed to generate appropriate actions for the 4WISD vehicle. The DRL critic includes two online critic neural networks and two target critic networks based on TIB, aiming to guide the actor network updates. The extended experience replay buffer, designed based on GER, is intended to store historical data for training the critic and actor.

Algorithm 1 Proposed GT-TD3

1: Initialize critic networks $Q_{{\varphi }_1},Q_{{\varphi }_2}$ and actor network ${\pi }_{\theta }$ with random parameters ${\varphi }_1,{\varphi }_2,\theta$; 2: Initialize target networks $Q_{{\varphi }_1}^{\prime },Q_{{\varphi }_2}^{\prime },{\pi }_{\theta }^{\prime }$ with parameters ${\varphi }_1^{\prime }\leftarrow {\varphi }_1,{\varphi }_2^{\prime }\leftarrow {\varphi }_2,{\theta }^{\prime }\leftarrow \theta$; 3: Initialize experience replay buffer $\Omega$ and sampling proportion coefficients for three groups $S_{\alpha 0}$, $S_{\beta 0}$, $S_{\gamma 0}$; 4: for $episode=1$ to M do 5:     Initialize a random process $R_N$ for action exploration; 6:     Receive initial observation state ${\mathbf{s}}_1$; 7:     for $t=1$ to E do 8:         Select action ${\mathbf{a}}_t={\pi }_{\theta }\left({\mathbf{s}}_t\right)+N_t$ according to the current policy and exploration noise; 9:         Execute action ${\mathbf{a}}_t$ in the environment; 10:         Observe reward ${\mathbf{r}}_t$, new state ${\mathbf{s}}_{t+1}$; 11:         Calculate TD error ${\tau }_t$, advantage function $A_t$, priority $P_t$ of sample; 12:         Store transition sample $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1},P_t,{\tau }_t,A_t)$ in experience replay buffer; 13:         Divide the samples in the experience replay buffer into three groups based on priority $P_t$; 14:         Non-dominated sorting within groups based on ${\tau }_t$ and $A_t$: $$\begin{array}{cc}\hfill & F_k=\left\{s_i\in S_k\mid \forall s_j\in S_k,{\tau }_j<{\tau }_i\mathrm{and}{A}_j<A_i\right\}\hfill \end{array}$$ 15:         Update sampling proportion coefficients for three groups $S_{\alpha t}$, $S_{\beta t}$, $S_{\gamma t}$: $$\begin{array}{cc}\hfill & \left\{\begin{array}{c}{S}_{\alpha t}=S_{\alpha 0}(1-t/T)N_{\alpha }\hfill \\ \hfill S_{\beta t}=(S_{\beta 0}(1-t/T)+S_{\alpha 0}(t/T))N_{\beta }\\ S_{\gamma t}=(1-S_{\alpha 0}-S_{\beta 0})N_{\gamma }\hfill \end{array}\right.\hfill \end{array}$$ 16:         Update critic network by policy gradient: $$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \hfill L\left(\varphi \right)={\mathbb{E}}_{({\mathbf{s}}_t,{\mathbf{a}}_t)\sim E}\left[{(Q({\mathbf{s}}_t,{\mathbf{a}}_t)-y_t)}^2\right]+\xi \mathrm{KL}\left(p\left({\mathbf{z}}_t\right|{\mathbf{s}}_t,{\mathbf{a}}_t),r\left({\mathbf{z}}_t\right)\right)\end{array}\end{array}$$ 17:         if $tmod{d}_{\mathrm{delay}}==0$ then 18:            Update actor network by maximizing the loss: $$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \hfill {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\mathbf{s}}_t\sim E}\left[{\nabla }_{\mathbf{a}}Q({\mathbf{s}}_t,{\mathbf{a}}_t){\nabla }_{\theta }{\pi }_{\theta }\left(\mathbf{s}\right)\right]+\kappa {\nabla }_{\theta }\mathrm{KL}\left(p\left({\mathbf{u}}_t\right|{\mathbf{s}}_t),r\left({\mathbf{u}}_t\right)\right)\end{array}\end{array}$$ 19:            Update target networks: $$\begin{array}{c}{\varphi }_j^{\prime }\leftarrow \sigma {\varphi }_j+(1-\sigma ){\varphi }_j^{\prime },j=1,2\hfill \\ {\psi }_j^{\prime }\leftarrow \sigma {\psi }_j+(1-\sigma ){\psi }_j^{\prime },j=1,2\hfill \\ {\theta }^{\prime }\leftarrow \sigma \theta +(1-\sigma ){\theta }^{\prime }\hfill \\ {\omega }^{\prime }\leftarrow \sigma \omega +(1-\sigma ){\omega }^{\prime }\hfill \end{array}$$ 20:         end if 21:     end for 22: end for

5. Numerical Simulations

To validate the effectiveness of the proposed enhanced DRL algorithm and the DRL-based path tracking controller for 4WISD vehicles, we conducted extensive simulation experiments. These experiments were performed on a hardware platform equipped with an Intel Core i9-13900K processor and an NVIDIA GeForce RTX 4080 graphics card.

5.1. Convergence and Generalization Analyses

The simulation parameters for different DRL algorithms are presented in

Table 1. Simulation parameters used in different DRL algorithms.

DRL Algorithm	AER-TD3	PER-TD3	GER-TD3
Hidden layer dimension	256	256	256
Batch size	256	256	256
Discount factor	0.99	0.99	0.99
Soft update coefficient	0.05	0.05	0.05
Policy noise	0.2	0.2	0.2
Noise clipping range	256	256	256
Policy update frequency	2	2	2
Priority exponent	×	0.6	0.6
Group proportion coefficients	×	×	0.2, 0.7, 0.1
Learning rate	1 × 10−4	1 × 10−4	1 × 10−4

. The discount factor determines the extent of the future rewards relative to the immediate future. The soft update coefficient governs the convergence rate of the target network toward the current network. The exploration noise reflects the changes in exploratory behaviors, which is determined by the clipping noise. The policy network refresh frequency determines the update rate of the associated parameters. To make the proposed algorithms comparable, the hyperparameters are consistently set for all compared algorithms.

We establish a high-speed path tracking task for a 4WISD vehicle in the double-shift line condition. Random parameters are used to simulate the various conditions of vehicle velocities, curvatures, and external disturbances, such that the DRL training environment encompasses a wide range of path tracking scenarios, as specified in

Table 2. Simulation parameters used in the DRL training environment.

DRL Training Environment	Parameters	Unit
Shift line longitudinal position	[40,180]	m
Shift line transition length	[25,75]	m
Longitudinal velocity	[15,20]	m/s
Longitudinal velocity variation range	[0,20]	m/s
Smooth disturbance amplitude	[−100,100]	N
Sudden disturbance amplitude	[−100,100]	N
Smooth disturbance duration	[5,10]	s

. And the simulation parameters for a C-class vehicle are listed in

Table 3. System parameters of the vehicle.

Vehicle Parameter	Parameters	Unit
Vehicle mass	1477	kg
Vehicle yaw inertia	1536.7	$\mathrm{kg}·{\mathrm{m}}^2$
Track width	1.675	m
Distance from CG to front axle	1.015	m
Distance from CG to rear axle	1.895	m
Wheel radius	0.325	m
Wheel mass	22	kg
Wheel moment of inertia	0.8	$\mathrm{kg}·{\mathrm{m}}^2$

.

In the established training environment, experiments on each path tracking control task are repeated 10 times to demonstrate the efficacy of the DRL algorithms. During training, the agent is evaluated every 10,000 time steps, with one evaluation report sent every 100 evaluations. In the following figures, the solid lines represent the average results over 10 trials, while the shaded regions indicate the variance of predictions around that mean across 10 experiments.

To show the effect of the methods proposed in Section 4.2 and Section 4.3 on DRL performance, we conducted two convergence experiments. In the first experiment, reward curves obtained from three different experience replay mechanisms are compared in Figure 6, i.e., average experience replay (AER), prioritized experience replay (PER), and the proposed group intelligent experience replay (GER), with the remaining parameters unchanged. The reward curve of AER tends to converge in the early stages of iterations, but remains with a small reward value at the final stage, showing an inefficiency of convergence. Due to the unevenly distributed samples in the training environment, the low probability of occurrence leads to sample sparsity. These factors make it difficult for AER to effectively replay high-quality samples. The PER can prioritize samples based on the TD error, assigning higher sampling probabilities, which allows for the utilization of information-rich samples to yield more frequent updates for the network, thereby mitigating the impact of uneven and sparse distribution of samples. However, the PER-TD3 algorithm is trapped in local optima in the later training stages, resulting in almost constant reward curve variation. In comparison with simulation results from AER and PER, the reward curve from GER converges more stably, due to higher sample utilization efficiency and better balance between exploration and exploitation. The comparison results from the different DRL algorithms are presented in

Table 4. Maximum average return obtained from different TD3-based DRL algorithms.

DRL Algorithm	AER-TD3 [63]	PER-TD3 [41]	GER-TD3
Maximum average return	−933.3041	−509.6428	−378.1214

. The proposed GER improves convergence performance by 59% compared to AER and by 25% compared to PER.

In the second experiment, we maintained a fixed experience replay buffer across deep deterministic policy gradient (DDPG), TD3, and the proposed TIB-TD3 to evaluate convergence. As shown in Figure 7, the TD3 demonstrates stronger convergence capabilities compared to DDPG, due to the application of double Q-networks, delayed policy updates, and target policy smoothing, which improve learning stability and performance in continuous control. The reward curve of proposed TIB-TD3 shows minimal decline in the early stages of training. So the TIB-TD3 converges more quickly and maintains a stable trend in the reward curve in the subsequent training process. This improvement can be attributed to the removal of irrelevant information with IB, enabling the agent to learn higher-quality information. Additionally, the agent enhances the quality and expressiveness of its policy through the persistent application of the RIB, which is capable of preventing overfitting and reducing the influence of lacking unknown information.

The additive external disturbances, which consist of persistent and abrupt disturbances acting on each tire along the lateral and longitudinal directions, were varied from $-100\mathrm{N}$ to $100\mathrm{N}$ to evaluate the generalization efficiency of different DRL algorithms. The percentage improvement represents the performance improvement of the DRL-based compound controller compared to the dual-layer controller in the path tracking control. As illustrated in Figure 8, the proposed TIB-TD3 exhibits enhanced optimization performance compared with DDPG and TD3, with the results presented in

Table 5. Summary of the simulation results obtained from different DRL algorithms.

DRL Algorithm	DDPG [64]	TD3 [59]	TIB-TD3
Maximum average return	−1136.5222	−509.6428	−202.4948
Mean optimization	34.3016	49.3389	66.4575

. The experimental results demonstrate that TIB improves the generalization efficiency and convergence performance of the original TD3 algorithm by 60% and 26%, respectively, by filtering irrelevant information and preserving critical information necessary for action generation. This improvement facilitates the application of DRL models for the path tracking control of a 4WISD vehicle with high uncertain disturbances and unmodeled dynamics.

5.2. Performance Analysis of an Improved DRL-Based Path Tracking Controller for 4WISD Vehicles

To validate the performance of the proposed GT-TD3-based compound control framework (GTC) for path tracking control of 4WISD vehicles subjected to complex external disturbances, we compared it with the NMPC-SQP dual-layer controller (MLC) [16] and the TD3-based [59] compound control framework (TDC). In both the GTC and TDC, the DRL auxiliary controller is established in the training environment using the parameters specified in Section 5.1. The trained model is combined with the MLC method described in Section 3.2 to form the compound control framework.

The control performance of the aforementioned control methods is compared using the mean absolute error (MAE) and the maximum error metric (MAX), where MAE represents the average deviation from the desired path in this experiment, and MAX indicates the highest deviation observed at any point in the experiment, as shown in Figure 9. MAE reflects the overall steady-state control performance of the system, while the MAX indicates the transient performance. We can observe that the dual-layer controller has limited path tracking control capabilities for 4WISD vehicles subjected to complex external disturbances. The TDC can improve the path tracking performance due to the application of the TD3-based model-free auxiliary controller. The proposed GTC can adapt to abrupt changes in complex nonlinear vehicle dynamics with external disturbances, which improve the path tracking performance further. The proposed GTC reduces MAE error and MAX error by 68% and 63%, respectively, compared to MLC. The proposed GTC reduces MAE error and MAX error by 28% and 9%, respectively, compared to TDC.

To demonstrate the control performance of the three control methods, four groups of parameters for the double-shifted path and longitudinal velocity were established to simulate the vehicle’s driving environment under high-speed and large-curvature conditions. Figure 10 presents a comparison of the actual lateral displacement, yaw angle, and longitudinal velocity of the controlled vehicle using the three control methods. Correspondingly, the control errors with respect to the ideal states for the three control methods are presented in Figure 11. From Figure 10 and Figure 11, we can observe that when the tracking path is relatively stable, each control method can track the vehicle motion with high accuracy. However, when the tracking target undergoes significant changes, the dual-layer controller faces difficulties in mitigating external disturbances, resulting in substantial deviations in the vehicle state. In comparison, the TD3-based compound control framework can further address the impact of external disturbances on path tracking control. The proposed GT-TD3 exhibits the highest capabilities of controlling the external disturbances and the complex tracking targets, keeping the tracking errors within a sufficiently low range. This is because the proposed auxiliary controller based on GT-TD3 is capable of conducting dynamic compensation for the control variables from the upper-layer controller, as shown in Figure 12. Through allocating the compensated upper-layer control variables to the tire dynamics model, we finally obtain eight end-effector control variables, that is, the steering angle and torque of each tire, as shown in Figure 13.

The computation time of each controller is also an important factor for evaluating control performance. Figure 14 presents the computation time of the dual-layer controller, the compound control framework based on TD3, and the proposed compound control framework based on GT-TD3. The proposed GT-TD3 makes an improvement upon the original TD3 within the network architecture, rather than adding an independent structure. Therefore, this modification does not significantly increase the computation time of the controller. The experimental results demonstrate that the proposed compound control framework based on GT-TD3 achieves better control performance compared to conventional control methods, with little sacrifice of computation time.

6. Conclusions

In this study, we investigate the path tracking control problem of 4WISD vehicles under complex external disturbances, which are characterized by high-dimensional nonlinearity, complex coupling, and high uncertainty. A path tracking control method based on deep reinforcement learning (DRL) is proposed. Firstly, a compound control framework consisting of an NMPC-SQP dual-layer controller and a DRL-based auxiliary controller is introduced to ensure stable and efficient path tracking control performance under complex external disturbances. A novel group intelligence collaborative experience replay (GER) mechanism is proposed to improve the sample efficiency and convergence of the DRL. Furthermore, an actor-critic architecture based on a two-stream information bottleneck (TIB) is presented to enhance DRL’s ability in extracting high-dimensional nonlinear features and improving its generalization capability. Numerical simulations with extensive parameter settings are performed to validate the effectiveness of the proposed method. The experimental results demonstrate that the proposed GER improves convergence performance by 59% compared to AER and by 25% compared to PER. The proposed TIB can enhance the generalization efficiency and convergence performance of the original TD3 algorithm by 60% and 26%, respectively. Based on the proposed DRL algorithm, a well-trained DRL-based controller can essentially reduce the tracking error by 63%. Application of the proposed DRL can effectively address the issue of path tracking control of 4WISD vehicles under complex external disturbances. The proposed DRL-based compound control framework can significantly improve the stability and accuracy in path tracking control of 4WISD vehicles.

The compound control framework can be implemented in real-world environments to validate its robustness against uncertain disturbances and varying road conditions. Improvements can be made by incorporating advanced neural network designs or compound reinforcement learning strategies to reduce the complexity of the control framework and further improve its performance in complex 4WISD systems. Additionally, the scalability of the framework across different vehicle types and configurations can be further explored. Developing adaptive strategies that automatically adjust parameters in response to environmental changes is also recommended.