Multi-Dimensional Double Deep Dynamic Q-Network with Aligned Q-Fusion for Dual-Ring Barrier Traffic Signal Control

Abstract

Model-free deep reinforcement learning (MFDRL) is well-suited for real-time traffic signal control (RTSC), as it is a sequential decision problem where the environment is difficult to be a priori modeled, but has performance metrics sufficing as rewards. Previous studies have not ideally employed MFDRL systems at typical intersections with a dual-ring barrier phase structure (DRBPS) and second-by-second signal operation. DRBPS allows phases to time flexibly while satisfying signal timing constraints in engineering, making it complicated yet common in real-world applications. This study proposes an MFDRL method, termed MD4-AQF, to address the RTSC problem under DRBPS. The state can be represented as a 4 × 30 × 4 × 4 array. We define action based on “decision point aligning” to produce a consistent action space that controls dual-ring concurrent phases simultaneously. We developed a training algorithm based on a “multi-dimensional Q-network” that reduces the number of learnable actions from 600+ to 52. We designed action selection based on “aligned Q-fusion” to end two lagging phases simultaneously with a shared compromise sub-action. In simulation experiments, MD4-AQF trains an agent to improve average vehicle delay from 135 s to 48 s. It surpasses another MFDRL ablated method by 14%, and a fully actuated conventional method by 19%.

1. Introduction

Model-free deep reinforcement learning (MFDRL) is used for solving sequential decision problems. MFDRL does not rely on the environmental models pre-determined by humans. Instead, it involves learning by interactive trial and error, along with rewards from the environment. This makes MFDRL especially worth being used for problems characterized by two characteristics: (1) the environment’s dynamics cannot be perfectly a priori modeled; and (2) a series of control decisions as a whole can be assessed with a widely accepted performance metric, based on which, the reward for MFDRL can be readily defined [1].

Real-time traffic signal control (RTSC) at intersections is typically a sequential decision problem suitable for MFDRL. First, the behavior of a vehicle and the reactions of other vehicles to it are both highly stochastic. This complexity complicates the dynamics of intersections, making it difficult for humans to model such dynamics without losing any details. Second, metrics such as vehicle throughput and vehicle delay effectively measure intersection performance over a period of time and can be utilized to define rewards.

Many previous studies of MFDRL-based RTSC have focused on a road network with multiple intersections. From a traffic engineering perspective, there were still unresolved issues at individual intersections. In particular, signals often could not operate within a dual-ring barrier phase structure (DRBPS). This structure organizes phases in a fixed sequence to meet engineering constraints, while also allowing phases to time independently and flexibly, as indicated in Section 3. DRBPS is complex but remains the most common at real-world intersections [2]. RTSC methods incompatible with DRBPS cannot be widely adopted in urban areas. On the other hand, some studies did not permit second-by-second adjustment of phase green time. Instead, the green time could only be a multiple of a specific number of seconds, which suppressed the potential for performance enhancement of their methods.

This study proposes an MFDRL method for RTSC at an individual intersection with DRBPS, termed “multi-dimensional double deep dynamic Q-network with aligned Q-fusion (MD4-AQF)”. We deployed a single agent for the intersection that chose from green time options at one-second intervals for each phase. We address the following three major challenges:

• How to define an action and when to select an action for two concurrent phases in different rings. For this challenge, we present an action definition based on “decision point aligning” in Section 4.
• How to effectively learn hundreds of combinatorial actions for two concurrent phases. For this challenge, we present a training algorithm based on a “multi-dimensional Q-network” in Section 6.2.
• How to ensure that two concurrent lagging phases end simultaneously. For this challenge, we present an action selection mechanism based on “aligned Q-fusion” in the algorithm in Section 6.3.

The remainder of this paper is organized as follows. Section 2 reviews previous studies and points out their shortcomings. Section 3 describes the problems faced by us and the agent, and the difficulties in solving them. The main contents from Section 4, Section 5 and Section 6 and their roles in the reinforcement learning loop are illustrated in Figure 1.

2. Literature Review

Regarding MFDRL-based RTSC, recent research has made significant contributions to (1) multi-agent communication and cooperation in a road network; (2) transfer learning between intersections with different roadway geometry or traffic demand; (3) training algorithm modification for better encoding or decoding traffic state features, etc.

However, those studies generally overlooked some intractable issues at an individual intersection. These issues motivated us to conduct this study. We review them in terms of the two following aspects:

• Phase structure. Nearly all researchers employed a phase structure incompatible with DRBPS. This limitation inhibited the widespread use of their methods in engineering practice.
• Many studies restricted a phase to last only in multiples of a specific number of seconds, such as 5 seconds or 10 seconds. This constraint prevented further fine-tuning of green times to better match traffic demand. There had not yet been a training algorithm developed that fully supports DRBPS with second-by-second signal operation.

2.1. Phase Structure: Incompatibility with DRBPS

Almost all studies deployed an agent for an intersection, except for a few that involved a regional agent managing the signals across a road network [3,4,5,6,7,8]. Researchers have started to train multiple intersection agents synchronously. But from the traffic engineering perspective, there was a remaining issue, that is, the phase structure used was often incompatible with DRBPS at each intersection. This incompatibility widened the gap between research and practical engineering applications. Two main causes of this incompatibility have been identified in the literature.

(1) Some studies took a variable phase sequence [3,4,5,6,9,10,11,12,13,14,15,16,17,18,19], allowing agents to activate any preferred phase combination at every time step, as shown in Figure 2a. But to be frank, an unfixed phase sequence is generally not acceptable in traffic engineering. This is for the sake of safety, as detailed in Section 3.

(2) Some studies maintained a fixed phase sequence [7,8,20,21,22,23,24,25,26,27,28,29,30,31,32,33] but took a single-ring structure usually with four or two phases, as seen in Figure 2b. Such a structure is less flexible than DRBPS because it forces multiple vehicle movements with imbalanced demand to have the same green time. As a consequence, single-ring phase structures are not common in the real world [2].

Figure 2. Typical signal operations in previous studies. Solid one-way arrows represent vehicle movements, and two-way arrows represent adjacent parallel pedestrian movements. (a) Signal operation with variable phase sequence. An arbitrary phase or phase combination is chosen to be served every time step (usually a few or tens of seconds). (b) Signal operation in a single-ring phase structure. Four phases are served in cycles with a fixed sequence.

Figure 2. Typical signal operations in previous studies. Solid one-way arrows represent vehicle movements, and two-way arrows represent adjacent parallel pedestrian movements. (a) Signal operation with variable phase sequence. An arbitrary phase or phase combination is chosen to be served every time step (usually a few or tens of seconds). (b) Signal operation in a single-ring phase structure. Four phases are served in cycles with a fixed sequence.

To the best of our knowledge, there were only three studies compatible with DRBPS.

(1) Han, Lee, and Kim [26,27] developed two MFDRL methods with the deep deterministic policy gradient algorithm. To accommodate DRBPS, a five-dimensional continuous action was defined. Its dimensions determined, respectively, the proportion in a cycle assigned to the phases between barriers, and the split for pairs of these phases. However, such actions required a pre-set cycle length. It could not change according to the variation of total traffic demand at an intersection. Furthermore, such actions caused signal timing to be adjusted only once per cycle, making it difficult to tackle the fluctuating demand on phases.

(2) Ma et al. [33] proposed an MFDRL method based on the advantage actor–critic algorithm. The authors presented a stage-based diagram with eight phases, defining an action as whether to switch to the next stage. However, such actions led to the signal operation within a defective DRBPS. The signals were forced to operate in a pre-set stage order. For example, the order might be “east and west left-turn phases, east through and left-turn phases, east and west through phases”. Consequently, the east left-turn phase was always longer than the opposing west one in every cycle. This prevented the phases from timing flexibly based on their own demand.

2.2. Action Space and Training Algorithm

Signal controllers at real intersections primarily employ RTSC methods that allow a phase to last for any number of seconds, subject to certain constraints [2,34]. However, many researchers have restricted this flexibility to ease training by creating a coarse-grained action space. Specifically, they allowed the green time for each phase to be only in multiples of 5 s, 7 s, 10 s, 20 s, etc. [5,6,7,28,29,30,31,32,35,36,37,38,39,40,41,42,43]. In other words, this means phase green time could only be one of a few numbers, with large intervals between them, e.g., $(7·\delta )$, where $\delta \in \{1,2,3,4,5\}$. This approach has limited the potential of MFDRL methods to achieve greater performance advantages compared to the existing methods operating on a second-by-second basis.

In scenarios where signal operations follow a fixed phase sequence, “phase duration” was a suitable action definition for MFDRL [7,8,9,10,11,20,21,22,23,24,25,26,27,28,29,30]. It means choosing a green time for the current phase. However, it is non-trivial to combine the “phase duration” definition and second-by-second signal operation. In the case of a single-ring phase structure, such a definition leads to a discrete action space containing only tens of green time choices. For example, 26 choices at one-second intervals if minimum and maximum greens are 15 s and 40 s, respectively. But regarding DRBPS, an agent needs to determine the green time for each of the concurrent phases in two rings. As a result, the action becomes the permutation of green times for two phases. The number of selectable actions grows quadratically to ${26}^2=676$. Exploring such a large action space is very challenging for an agent.

The deep Q-network (DQN) and its variants are typical value-based MFDRL training algorithms [44]. They are competitive even with the newer algorithms, such as soft actor–critic when handling discrete actions [45,46]. The DQN variants were very common in the previous RTSC studies. But to the best of our knowledge, this paper is the first to propose an algorithm that expands the Q-dimension of a DQN variant [25] by using the technique described in [47], to address the large action space caused by DRBPS with one-second green time intervals.

3. RTSC Problem Within DRBPS and Its Complexities

This study focuses on the most typical and complex case of DRBPS at a four-leg intersection with protected left turns and permitted right turns [2]. As shown in Figure 3, it uses two rings and two barriers to organize eight phases in a signal cycle. The barriers separate the phases on two streets. Between the barriers, any phase in ring 1 can be served together with any phase in ring 2. This flexibility enables phases in different rings to adjust their timing to meet imbalanced demand, making DRBPS the most commonly used structure in real-world settings.

DRBPS involves some signal timing constraints that are fundamental in terms of safety in engineering practice [2,34,48]. They include (1) a fixed phase sequence, (2) simultaneous ending of lagging phases, (3) minimum and maximum green times, and (4) yellow change and red clearance intervals. The first two constraints are the most overlooked in previous studies, yet they are central to this study. We elaborate on both as follows.

• Fixed phase sequence. The sequence is namely the order of phases in each ring. It should be pre-determined and remain fixed during certain times of the day. Otherwise, the signals for all movements will turn green or red arbitrarily and unpredictably. This can confuse or even surprise drivers, and increase the risks of red-light-running and rear-end collision.
• Simultaneous ending of two concurrent lagging phases. Given that the two lagging phases precede the same barrier, this constraint ensures both rings cross this barrier together. Otherwise, conflicting vehicle movements will occur, leading to an increased risk of angled collisions. This is because each lagging phase conflicts with both phases that follow the barrier on the intersecting street.

The RTSC problem within DRBPS involves how to cyclically determine the green time of each phase in real-time, subject to the signal timing constraints above. In this study, efforts will be made to define an action and develop a training algorithm to solve this problem. Below, we describe the complexities of the problem facing us.

We need to enable the agent to control two concurrent phases in the dual ring, such as phases (1 and 5). They run in parallel but their green times overlap and can differ. They may start green successively and have different minimum greens. The agent should be capable of (1) making fully independent decisions on the green times for concurrent leading phases to accommodate varying demands, and (2) making partially independent decisions on the green times for concurrent lagging phases to balance their demands and end them together at a trade-off moment.

4. Action Definition with Decision Point Aligning

To accommodate DRBPS, we improve the action defined in [25] for a fixed phase sequence, which is originally to choose the remaining green time for the current phase at the end of its minimum green.

We begin with the question of when the agent selects an action. This moment is termed “decision point”. The question is non-trivial in DRBPS. If simply selecting an action whenever any phase reaches its minimum green like in [25], the action space can be inconsistent at different times, making it hard to determine the semantics of network outputs. This is because the two concurrent phases in a dual ring may reach their minimum greens either simultaneously or successively. Thus, the agent may sometimes need to make a decision for a single phase (corresponding to a small action space), but sometimes two decisions for two phases (a large combinatorial action space).

We propose a “decision point aligning”-based action definition to address this challenge. It aims to enable the agent to control the two phases with a consistent action space. Meanwhile, it aims to postpone the moment of decision-making as much as possible, so that the agent can base the action selection on the latest state. Specifically, we align the decision points for two concurrent phases in the dual ring at the earliest moment when either or both phases can terminate. At this point (time step t) to determine the green times for the two phases at once, represented as a 2-tuple:

$$A_t=〈a_t^1,a_t^2〉,$$

(1)

$$\forall k\in \{1,2\},a_t^k\in \mathbb{Z}\mathrm{and}0\le a_t^k\le G_{max}^k-G_{min}^k,$$

where $a_t^k$ denotes the remaining green time for the current phase in the k-th ring; and $G_{max}^k$ and $G_{min}^k$ are the maximum and minimum greens for the current phase in the k-th ring, respectively.

The aligned decision point (ADP) and action definition can be best understood via the examples below (see Figure 4).

• A case of two leading phases, $\Phi 1$ and $\Phi 5$. DRBPS allows an independent termination of each phase, subject to only its own minimum green. For these two phases, ADP is the moment that either phase’s minimum green expires (i.e., the “ADP1” moment in the figure). It is the last possible moment to decide whether to give this phase the time beyond its minimum green. At this point, action $A_t$ represents the remaining green times after the respective ends of the minimum green durations of the two phases. $a_t^1$ and $a_t^2$ can be unequal. They denote the lengths of time marked with ${\mathrm{a}}_4^1$ and ${\mathrm{a}}_4^2$ in the figure, respectively.
• A case of two lagging phases, $\Phi 2$ and $\Phi 6$. DRBPS requires the simultaneous termination of the two phases, subject to both their minimum greens. For these two phases, ADP is the moment when the minimum greens of both phases expire (i.e., moment “ADP2” in the figure). It is the earliest possible moment at which the two phases are allowed to terminate simultaneously. Here, action $A_t$ represents the remaining green times for the two phases after this moment, instead of after the ends of their own minimum greens. $a_t^1$ and $a_t^2$ have to be equal. They denote the lengths of time marked with ${\mathrm{a}}_3{6}^1$ and ${\mathrm{a}}_3{6}^2$ in the figure, respectively.

With the decision point aligning, the agent selects actions four times every cycle. An action always determines the green times for two phases. But according to the typical values of the maximum and minimum greens used in engineering, $a_t^1$ and $a_t^2$, each can have 20–40 possible choices. The result is at least ${20}^2=400$ combinatorial choices of $A_t$. This poses a new challenge of how to enable the agent to learn these many actions. We will address it in Section 6.2.

5. Reward and State Definitions

The definitions of (1) reward, (2) state, and (3) neural network structure are similar to those in [25], except for a few modifications. We describe them as follows:

(1) Reward is defined as the normalized total number of vehicles departing from the intersection between two consecutive time steps:

$$R_t={\displaystyle \frac{1}{LN}}\sum _{l=1}^{LN}V{S}_{t,t+1}^l,$$

(2)

where $R_t$ is the reward for time step t; $V{S}_{t,t+1}^l$ is the count of vehicles crossing the stop line on the l-th approach lane between time steps t and $(t+1)$; $LN$ is the number of approach lanes at the intersection.

The agent follows the standard objective in MFDRL, which is to maximize the expectations of cumulative discounted rewards:

$$\mathbb{E}\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots \right],$$

(3)

where $\gamma \in (0,1]$ is the discount factor. This objective amounts to encouraging the agent to serve more vehicles as early as possible to lower the vehicle delay at the intersection [25].

(2) To construct the state representation shown in Figure 5, we take a 120-meter-long portion of each intersection leg as the detection zone. It is divided into grids that are 4 m in length and are as wide as the corresponding lane widths. The state feature of a grid is a 4-tuple if it is on the approach lanes for a phase, or a 2-tuple if it is upstream from approach lanes:

$$f=\left\{\begin{array}{cc}(o,v,g,c)\hfill & \mathrm{for}\mathrm{approach}\mathrm{lanes},\hfill \\ (o,v)\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(4)

where o, v, g, and c denote the presence of the vehicle on the grid, the normalized speed of the vehicle, whether the corresponding phase is green, and whether the phase is a lagging phase, respectively.

The state features of the grids on the approach lanes for each phase, and those of the other grids upstream from approach lanes, are separately organized as image-like state arrays, as follows:

$$s=\left(\begin{array}{cccc}{f}_{1,1}& f_{1,2}& \cdots & f_{1,W}\\ f_{2,1}& f_{2,2}& \cdots & f_{2,W}\\ ⋮& ⋮& \ddots & ⋮\\ f_{H,1}& f_{H,2}& \cdots & f_{H,W}\end{array}\right),$$

(5)

where $f_{h,w}$ is the state feature at the h-th row and w-th column; and H and W are the number of rows and columns in the corresponding state array, respectively.

Lastly, the state at time step t is represented as a multi-element tuple that includes all the state arrays for the intersection:

$$S_t=〈s_{\mathrm{LT}}^1,s_{\mathrm{TH}}^1,s_{\mathrm{US}}^1,s_{\mathrm{LT}}^2,\cdots ,s_{\mathrm{US}}^B〉,$$

(6)

where $s_{\mathrm{LT}}^b$ and $s_{\mathrm{TH}}^b$ are the state arrays for the approach lanes on the b-th intersection leg for the left and through phases, respectively, $s_{\mathrm{US}}^b$ is the state array for the lanes upstream from the approach lanes on the b-th intersection leg, and B is the number of legs at the intersection. For the test-bed signalized intersection in our experiments, the state can be represented as a high-dimensional array with $4·30·4·4=1920$ elements.

(3) Each state array is input into a convolutional neural network with “ReLU” activations. Its first three layers convolve 32, 64, and 64 filters with stride 1 with zero padding, respectively. The filter size is $3\times 3$, except when the state array corresponds to a single lane, in which case the filter size is $3\times 1$. The convolving results for all the state arrays are concatenated as a vector. But unlike in [25], we feed this vector into several branches of fully connected layers (see Figure 5). Each branch consists of a layer with 128 units and a layer outputting multi-dimensional sub-action values. This modification is required by the training algorithm proposed in the next section.

Data availability for reward and state: The reward defined in this study relies on monitoring the number of vehicles crossing stop lines. The data can be easily collected using conventional traffic detection devices, such as underground inductive loop detectors. The defined state requires periodic measurement of vehicle locations and speeds near an intersection. These data can be captured through advanced video or radar detection systems, as well as emerging internet-of-vehicle or vehicle-to-infrastructure technologies.

6. Training Algorithm with Multi-Dimensional Aligned Q-Fusion

In this section, we introduce the training algorithm for MD4-AQF. It is modified from the double deep dynamic Q-network (D3ynQN) [25] through four steps to better accommodate the RTSC problem within DRBPS. The first step involves recalculating the D3ynQN’s parameter n for DRBPS. The second involves generalizing D3ynQN from uni-dimensional action cases to multi-dimensional ones. The third involves fusing multi-dimensional Q values for lagging phases. The fourth involves deriving the formula for a bootstrapping-based value update in the algorithm.

6.1. Dynamic N-Step Update

D3ynQN is a DQN-style training algorithm proposed in [25]. It has already been used in solving a single-ring RTSC problem with a fixed phase sequence. In this section, we introduce and improve D3ynQN to render it suitable for our action definition.

We follow the D3ynQN’s suggestion of setting a constant time-step size of one second. This ensures the rewards are discounted evenly as time goes on, to reflect the uncertainty arising from the time-varying RTSC environment [25]. Consequently, the actions will be selected every several time steps, instead of every time step. This decouples the action step from the time step. So, we will add another subscript i to the following action:

$$A_{t,i}=\langle a_{t,i}^1,a_{t,i}^2\rangle ,$$

(7)

to denote the action step, representing the i-th time of action selection.

Next, we will conduct the dynamic n-step temporal-difference bootstrapping derived from D3ynQN. It adjusts the interval parameter n to match the time span between two actions so that an action value can exploit the next action’s value to bootstrap and update itself.

We begin by recalculating the aforementioned parameter, n, for DRBPS, which should equal the number of time steps between $A_{t,i}$ and its next action [25]. We add the subscript t to $n_t$, as it is dynamic and depends on $A_{t,i}$. According to the definition of action in Section 4, $n_t$ can be given by the following:

$$n_t=\left\{\begin{array}{c}max\left[G_{min}^1-G_{min}^2+{GTG}_t^1,{GTG}_t^2\right],\hfill \\ \mathrm{if}\mathrm{current}\mathrm{phases}\mathrm{are}\mathrm{leading}\mathrm{and}{G}_{min}^1\ge G_{min}^2;\hfill \\ max\left[{GTG}_t^1,G_{min}^2-G_{min}^1+{GTG}_t^2\right],\hfill \\ \mathrm{if}\mathrm{current}\mathrm{phases}\mathrm{are}\mathrm{leading}\mathrm{and}{G}_{min}^1<G_{min}^2;\hfill \\ min\left[{GTG}_t^1,{GTG}_t^2\right],\hfill \\ \mathrm{if}\mathrm{current}\mathrm{phases}\mathrm{are}\mathrm{lagging}.\hfill \end{array}\right.$$

(8)

where $GT{G}_t^k$ is the time from the end of the minimum green for the current phase (at time step t) to the end of the minimum green for the next phase in the k-th ring:

$${GTG}_t=a_{t,i}^k+{YC}^k+{RC}^k+{NG}_{min}^k.$$

(9)

Note that $G_{min}^k$ and ${NG}_{min}^k$ are the minimum greens for the current phase and its next phase in the k-th ring, respectively; ${YC}^k$ and ${RC}^k$ are the yellow change and red clearance intervals for the current phase in the k-th ring, respectively. Some examples of time lengths represented by $n_t$ are visually shown in Figure 4.

With $n_t$ being known, we can write down the general loss formula used to update the action value $Q_{\theta }(S_t,A_{t,i})$ at time step t, given by the following:

$$L_t={\left[\left({RS}_t+{\gamma }^{\left(n_t\right)}Q{B}_t\right)-Q_{\theta }\left(S_t,A_{t,i}\right)\right]}^2,$$

(10)

where ${RS}_t$ is the sum of discounted rewards in subsequent $n_t$ time steps:

$$R{S}_t=\sum _{k=t}^{t+n_t-1}{\gamma }^{k-t}{R}_k,$$

(11)

and ${QB}_t$ is the bootstrap action value after the $n_t$ time steps:

$$Q{B}_t=Q_{{\theta }^{-}}\left(S_{t+n_t},arg\underset{A_{t+n_t,i+1}}{max}{Q}_{\theta }\left(S_{t+n_t},A_{t+n_t,i+1}\right)\right).$$

(12)

Note that $\theta$ and ${\theta }^{-}$ are the weights of the behavior and target Q-networks, respectively. In the following subsections, we will propose more improvements to the loss formula (10) to further accommodate DRBPS.

6.2. Multi-Dimensional Q-Network

D3ynQN and other DQN-style algorithms typically configure a Q-network to output a uni-dimensional vector with each unit for the value of a specific action. Regarding DRBPS, unfortunately, there will be hundreds of such actions (see Section 4). Each of them means a possible permutation of the remaining green times for the two concurrent phases in the dual ring. It is very difficult for the agent to directly learn this many individual actions [47].

To address this challenge, we propose the “multi-dimensional Q-network”. It improves D3ynQN via the technique of dimensionality expansion explored in [47], aimed at reducing the magnitude of the number of network outputs to be considered. We begin by treating each element of action $A_{t,i}$, referred to as a “sub-action” $a_{t,i}^k$ , where $k\in \{1,2\}$. We modify the network from directly outputting action values $Q_{\theta }(S_t,A_{t,i})$ to a branch structure that outputs multi-dimensional sub-action values (see Figure 5):

$$Q_{\theta }^d\left(S_t,a_{t,i}^d\right),$$

(13)

where $d\in \{1,2,\cdots ,D\}$ denotes the sub-action dimension. Each dimension has E sub-action choices.

In our case, with DRBPS, the number of dimensions, $D=2$, corresponds to the two rings. The number of sub-actions per dimension, $E=26$, corresponds to the remaining green time options for the current phase in each ring. By turning into multi-dimensional outputs, the network now only needs to estimate $26·2=52$ values of sub-actions in two dimensions, instead of the original ${26}^2=676$ values of combinatorial actions. We have the agent separately select the sub-action with the maximum value among each dimension:

$$\left\{\begin{array}{c}{a}_{t,i}^1=arg\underset{a_{t,i}^{1,e}}{max}\left[Q_{\theta }^1\left(S_t,a_{t,i}^{1,e}\right)\right],\hfill \\ a_{t,i}^2=arg\underset{a_{t,i}^{2,e}}{max}\left[Q_{\theta }^2\left(S_t,a_{t,i}^{2,e}\right)\right].\hfill \end{array}\right.$$

(14)

Then, the selected sub-actions in both dimensions can compose the action $A_{t,i}=\langle a_{t,i}^1,a_{t,i}^2\rangle$.

6.3. Aligned Q-Fusion

As proposed in the previous subsection, the multi-dimensional Q-network determines the phase green times for two rings independently. This makes it more challenging to ensure that two concurrent lagging phases end simultaneously, as required by DRBPS. In other words, it raises the challenge of how to ensure that the agent selects the same sub-action for each of the two dimensions in such a situation.

We propose “aligned Q-fusion” to address this challenge. Let $a_{t,i}^{d,e}$ denote a specific choice of sub-action $a_{t,i}^d$ for dimension d, and $Q_{\theta }^{d,e}(S_t,a_{t,i}^{d,e})$ denote its value, where $e\in \left\{0,1,\cdots ,E-1\right\}$ is the ordinal number of this choice in its dimension. Then, $Q_{\theta }^{d,e}(S_t,a_{t,i}^{d,e})$ by definition is the agent’s estimate of the future rewards, conditioned on extending e seconds of green time for the current phase in the d-th ring.

We first align the sub-actions and their values for different dimensions like in Figure 6. If current phases are lagging, we calculate the “fused” value ${FQ}_{t,i}^{\ast ,e}$ for each aligned sub-action $a_{t,i}^{\ast ,e}$. It equals the mean of the values of sub-actions with the same ordinals across all dimensions:

$${FQ}_{t,i}^{\ast ,e}\left(S_t,a_{t,i}^{\ast ,e}\right)={\displaystyle \frac{1}{D}}\sum _{d=1}^D{Q}_{\theta }^{d,e}\left(S_t,a_{t,i}^{d,e}\right).$$

(15)

The resulting ${FQ}_{t,i}^{\ast ,e}(S_t,a_{t,i}^{\ast ,e})$ can represent the average expected returns on extending the same green time for two concurrent phases. We then take the aligned sub-action ${sa}_{t,i}^{\ast }$ that maximizes this fused value, as follows:

$${sa}_{t,i}^{\ast }=arg\underset{a_{t,i}^{\ast ,e}}{max}\left[{FQ}_{t,i}^{\ast ,e}\left(S_t,a_{t,i}^{\ast ,e}\right)\right].$$

(16)

For lagging phases, we make ${sa}_{t,i}^{\ast }$, shared by all sub-action dimensions (i.e., the two rings). Thus, the action becomes $A_{t,i}=\langle {sa}_{t,i}^{\ast },{sa}_{t,i}^{\ast }\rangle$. Essentially, ${sa}_{t,i}^{\ast }$ attempts to seek the best compromise between two lagging phases, upon the condition that DRBPS requires them to end simultaneously. For leading phases, we still take the best sub-action in each dimension independently as usual. In summary, the selected sub-action per dimension is as follows:

$$\left\{\begin{array}{c}{a}_{t,i}^1=a_{t,i}^2={sa}_{t,i}^{\ast },\hfill \\ \mathrm{if}\mathrm{current}\mathrm{phases}\mathrm{are}\mathrm{lagging};\hfill \\ a_{t,i}^k=arg\underset{a_{t,i}^{k,e}}{max}\left[Q_{\theta }^k\left(S_t,a_{t,i}^{k,e}\right)\right]\mathrm{for}k\in \{1,2\},\hfill \\ \mathrm{if}\mathrm{current}\mathrm{phases}\mathrm{are}\mathrm{leading}.\hfill \end{array}\right.$$

(17)

The above Equation (17) in effect rewrites Equation (14) by embedding the aligned Q-fusion into it.

6.4. Bootstrap Action Value

Now, we adapt the bootstrap action value ${QB}_t$ in Equation (12) for multi-dimensional sub-actions. As recommended in [47], we set a global ${QB}_t^{\ast }$ used by all sub-action dimensions. It is the mean of the bootstrap value of the sub-action per dimension, given by the following:

$$Q{B}_t^{\ast }=\left\{\begin{array}{c}{\displaystyle \frac{1}{D}}\sum _{d=1}^D{Q}_{{\theta }^{-}}^d\left(S_{t+n_t},{sa}_{t+n_t,i+1}^{\ast }\right),\hfill \\ \mathrm{if}\mathrm{next}\mathrm{phases}\mathrm{are}\mathrm{lagging};\hfill \\ {\displaystyle \frac{1}{D}}\sum _{d=1}^D{Q}_{{\theta }^{-}}^d\left(S_{t+n_t},arg\underset{a_{t+n_t,i+1}^{d,e}}{max}{Q}_{\theta }^d\left(S_{t+n_t},a_{t+n_t,i+1}^{d,e}\right)\right),\hfill \\ \mathrm{if}\mathrm{next}\mathrm{phases}\mathrm{are}\mathrm{leading}.\hfill \end{array}\right.$$

(18)

Then, we have a loss function modified from Equation (10). It takes the mean squared error across sub-action dimensions to form an aggregated scalar loss $L_t^{\ast }$ below, from which the gradients are backpropagated to all the values of the selected sub-actions:

$$L_t^{\ast }={\displaystyle \frac{1}{D}}\sum _{d=1}^D{\left[\left({RS}_t+{\gamma }^{\left(n_t\right)}Q{B}_t^{\ast }\right)-Q_{\theta }^d\left(S_t,a_{t,i}^d\right)\right]}^2.$$

(19)

We need to optimize $\mathcal{L}$, the expectation of the per-sample loss over the replay buffer $\mathsf{\Omega }$. It is estimated using the mean of $L_t^{\ast }$ across a mini-batch including M samples that are uniformly sampled from $\mathsf{\Omega }$. This adds another averaging operator to Equation (19), as follows:

$$\begin{array}{cc}\hfill \mathcal{L}& ={\mathbb{E}}_{(S_t,A_{t,i},{RS}_t,S_{t+n_t})\sim \mathsf{\Omega }}\left(L_t^{\ast }\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{M}}\sum _{m=1}^M{\displaystyle \frac{1}{D}}\sum _{d=1}^D{\left[\left({RS}_t+{\gamma }^{\left(n_t\right)}Q{B}_t^{\ast }\right)-Q_{\theta }^d\left(S_t,a_{t,i}^d\right)\right]}^2,\hfill \end{array}$$

(20)

where m denotes the ordinal number of a sample in the mini-batch.

Given the formula above, we can eventually sum up the training algorithm for MD4-AQF in a pseudo-code, as shown in Algorithm 1.  

Algorithm 1: Training algorithm for MD4-AQF

7. Experiments

We designed three experiments for the proposed MD4-AQF method. First, we investigated the agent’s learning process for MD4-AQF. Second, we contrasted the training progress of MD4-AQF and another MFDRL method (D3ynQN), which was the same as our method but with its “multi-dimensional network” and “aligned value fusion” ablated. Third, we compared the performance of the agent trained by MD4-AQF and a conventional RTSC method. Table 1 shows the setup of the experiments.

The experiments were conducted via a virtual intersection modeled by the traffic simulation software PTV Vissim 11.00-14. Signal control was implemented through the Vissim COM interface for Python 3.9.13. Neural networks were built using PyTorch 2.0.0.

The detailed geometry of the intersection and its peak-period traffic demand scenarios can be found in [25]. Briefly, a street had three lanes in each direction and was widened to four lanes when approaching the intersection (see Figure 7). The typical DRBPS in Figure 3 was employed. The minimum greens were 15 s for through phases and 5 s for left-turn phases. The maximum greens were 25 s greater than the minimum greens. The yellow change and red clearance intervals were 3 s and 2 s, respectively. Each simulation run lasted for $\mu =4200$ s, with the first 600 s being the warm-up period.

Table 1. Overview of experiment setup.

Experiment	Involved Method	Analytical Tool	Purpose and Expected Outcome
#1: agent learning	MD4-AQF	Line graph (Figure 8)	To verify whether the proposed method enables the agent to learn and improve its policy. The intersection performance is expected to gradually improve throughout the learning process.
#2: versus MFDRL method	MD4-AQF D3ynQN	Line graph (Figure 8) Box plot (Figure 9)	To verify whether the proposed method trains the agent more effectively than another MFDRL method, D3ynQN. The proposed method is expected to achieve better intersection performance than D3ynQN, both during training and at the end of training.
#3: versus conventional method	MD4-AQF FASC	Box plot (Figure 9) Scatter plot (Figure 10) Statistical test ( Table 2. Wilcoxon matched-pair signed rank test on the average vehicle delay caused by MD4-AQF and FASC. FASC − MD4-AQF N Mean Rank Sum of Ranks Negative Ranks 49 a 530.35 25,987.00 Positive Ranks 1951 b 1012.31 1,975,013.00 Ties 0 c Z −37.729 Sig. (two-tailed) 0.000 ** a FASC < MD4-AQF, b FASC > MD4-AQF, c FASC = MD4-AQF. ** at 0.05 significance level. )	To verify whether the policy learned via the proposed method outperforms the conventional method FASC. The learned policy is expected to yield better intersection performance compared to FASC in most of the tested traffic demand scenarios.

Table 1. Overview of experiment setup.

Experiment	Involved Method	Analytical Tool	Purpose and Expected Outcome
#1: agent learning	MD4-AQF	Line graph (Figure 8)	To verify whether the proposed method enables the agent to learn and improve its policy. The intersection performance is expected to gradually improve throughout the learning process.
#2: versus MFDRL method	MD4-AQF D3ynQN	Line graph (Figure 8) Box plot (Figure 9)	To verify whether the proposed method trains the agent more effectively than another MFDRL method, D3ynQN. The proposed method is expected to achieve better intersection performance than D3ynQN, both during training and at the end of training.
#3: versus conventional method	MD4-AQF FASC	Box plot (Figure 9) Scatter plot (Figure 10) Statistical test (Table 2)	To verify whether the policy learned via the proposed method outperforms the conventional method FASC. The learned policy is expected to yield better intersection performance compared to FASC in most of the tested traffic demand scenarios.

Figure 7. Layout of the virtual intersection for experiments.

Figure 7. Layout of the virtual intersection for experiments.

Figure 8. Learning curves for MD4-AQF and D3ynQN. The solid line represents the median of the average vehicle delay across 100 simulation runs for each evaluation, and the shaded area represents the range between the top and bottom quartiles.

Figure 8. Learning curves for MD4-AQF and D3ynQN. The solid line represents the median of the average vehicle delay across 100 simulation runs for each evaluation, and the shaded area represents the range between the top and bottom quartiles.

Figure 9. Box plots of the average vehicle delay caused by D3ynQN, MD4-AQF, and FASC.

Figure 9. Box plots of the average vehicle delay caused by D3ynQN, MD4-AQF, and FASC.

Figure 10. Scatter plot of the average vehicle delay caused by MD4-AQF and FASC in different traffic demand scenarios. Each dot corresponds to a specific scenario, with the x-coordinate and y-coordinate denoting the results for FASC and MD4-AQF, respectively. The greater the distance of a dot from the anti-diagonal line, the larger the difference in average vehicle delay between the two methods.

Figure 10. Scatter plot of the average vehicle delay caused by MD4-AQF and FASC in different traffic demand scenarios. Each dot corresponds to a specific scenario, with the x-coordinate and y-coordinate denoting the results for FASC and MD4-AQF, respectively. The greater the distance of a dot from the anti-diagonal line, the larger the difference in average vehicle delay between the two methods.

8. Discussion

8.1. Experiment 1: Agent Learning

We conducted almost no hyperparameter fine-tuning for MD4-AQF but took the values of the hyperparameters used in some original MFDRL algorithms. We found that this simple approach had already yielded very satisfactory performance. The minimal need for hyperparameter tuning demonstrated the ease of use of MD4-AQF. Given this competitive edge, we had little motivation to invest time in fine-tuning hyperparameters. It is possible that better performance could be achieved if more effort is devoted to hyperparameter tuning in the future.

Specifically, the stochastic gradient descent (SGD) optimizer used was Adam, with a learning rate of $6·{10}^{-5}$, which had been tuned and used by the authors of “Rainbow” (one of the best-performing DQN-style algorithms) to stabilize training [45]. Training started after randomly selecting actions 20,000 times, again as tuned and used in Rainbow [45]. A mini-batch included 64 samples, a commonly used value in many previous studies, and in the “BDQ” algorithm, where the Q-dimensionality expansion technique in this study was involved [47]. The target network weights were smoothly updated toward the exponentially moving average of the behavior network weights:

$${\theta }^{-}\leftarrow \tau \theta +(1-\tau ){\theta }^{-}$$

(21)

where $\tau =5·{10}^{-3}$. This practice was applied by the authors of the “SAC” algorithm consistently across various tasks [49]. $\gamma$ was set to 0.99375 such that the agent’s time horizon $1/(1-\gamma )$ equaled the maximum possible cycle length. Such empirical practice was proven effective in prior work on MFDRL-based RTSC with an environment similar to that of this study [25]. The replay buffer held at most 200,000 samples, which was the maximum we could set, due to the limitation of memory capacity. The exploration probability $ϵ$ in the $ϵ$-greedy behavior policy was 0.1 throughout the training and 0 for the evaluation. The value of 0.1 was the final value of $ϵ$ set by DQN and its many variant algorithms in the later stages of training [44].

To track the agent’s learning progress, we evaluated the behavior network every 10 simulation runs during training. For each evaluation, we performed 100 simulation runs and recorded the quartiles of average vehicle delay at the intersection. In principle, more frequent evaluations allowed for more timely tracking of policy improvements, and more simulation runs per evaluation provided more accurate statistics. The training continued until the median of the average delay had not improved for $\rho =200,000$ action steps. This is essentially the “early stopping” tolerance commonly used in deep learning, where training is terminated after waiting a sufficient amount of time without improving the best score. Strictly speaking, $\rho$ is not considered a hyperparameter, as it does not affect the policy changes during training.

As illustrated in Figure 8, the median of delay for MD4-AQF (marked red) oscillated around 135 s in the first 20,000 action steps. This was due to the random action selection, aimed at collecting initial samples in the replay buffer. Thereafter, the policy iteration began and the median showed a sharp drop to nearly 60 s within the next 20,000 steps. This signified that the agent rapidly found an acceptable policy to time signals. Starting from the 40,000th step, the improvement of performance slowed down. The agent spent the next 60,000 steps to lower the median by 5 s, and another 100,000 steps to further reduce the median from 55 s to about 50 s before the 200,000th step. Afterward, the downward curve tended to be gentle. The agent kept on seeking better performance in fluctuating median values. Eventually, it hit a minimum of 47.89 s at the 398,510th step. Since then, the median instead started to deteriorate gradually over time. This was consistent with the known “forgetting” phenomenon in the late period of training with DQN-style algorithms [50]. In general, the pattern of the learning curve was similar to that exhibited in the previous study [25]. We saved the best-performing network weights for the subsequent experiments.

8.2. Experiment 2: Versus D3ynQN Method (Ablation Study)

We contrasted MD4-AQF with a variant of the double DQN method (D3ynQN) in terms of training progress. D3ynQN is the same as MD4-AQF but removes the components of the “multi-dimensional Q-network” and subsequent “Aligned Q-fusion” for lagging phases. Instead, D3ynQN directly learns each uni-dimensional combinatorial action for the dual ring. This requires $256·{26}^2=160,256$ weights in the network output layer (compared to $128·2·26=6656$ for MD4-AQF). For lagging phases, D3ynQN masks the invalid actions that cause the phases to end non-simultaneously. Meanwhile, D3ynQN retains the practice of dynamic N-step update and uses the same hyperparameters of MD4-AQF. In this sense, Experiment 2 is also an ablation study on the proposed method.

D3ynQN [25] is an MFDRL method that has been demonstrated to control signals within a single-ring phase structure while satisfying the same signal timing constraints as in this study. Its success was achieved under the same conditions as in this study, including the same simulation software, road geometry, and traffic demand. Compared to MD4-AQF, D3ynQN is identical in all factors except for the two major innovations proposed in this study for DRBPS. Thus, D3ynQN serves as a competitive and comparable ablation baseline against MD4-AQF.

Figure 8 shows the learning curve for D3ynQN (marked green), compared with that for MD4-AQF. D3ynQN improved the median of the average delay to only about 80 s before the 40,000th action step. The performance exhibited severe fluctuations ranging from 65 s to 95 s until the 120,000th step. This indicated the difficulty in learning hundreds of combinatorial actions directly. As training progressed to around the 150,000th step, the oscillation of the median delay began to alleviate. The median dropped below 60 s at the 320,000th step. During subsequent training, the agent eventually achieved the best performance (55.15 s) at the 465,181st step. This result lagged behind that of MD4-AQF (47.89 s) by 7.26 s or 15.2%. Overall, D3ynQN utilized significantly more network parameters than MD4-AQF but experienced greater instability and consistently underperformed compared to MD4-AQF throughout the training process.

To further compare the agents trained via D3ynQN and MD4-AQF, we followed the manner in [25] to stochastically generate 2000 traffic demand scenarios for the intersection via the D-optimal design. In each scenario, we performed two simulation runs, with signals being timed by D3ynQN and MD4-AQF, respectively. We then recorded and compared their average vehicle delay at the intersection.

As shown in box plots in Figure 9, the medians of the average vehicle delay caused by D3ynQN and MD4-AQF, across the 2000 traffic demand scenarios, were 56.78 s and 49.62 s respectively. In comparison, the performance disadvantage of D3ynQN was 7.16 s or 14.4% in terms of the median value. Additionally, the top and bottom quartiles (as well as the upper and lower whiskers) produced by D3ynQN were larger than those of MD4-AQF. The inter-quartile range (i.e., the difference between the 75th and 25th percentiles) produced by D3ynQN was 66.91 − 50.42 = 16.49 s, whereas that of MD4-AQF was 57.69 − 44.31 = 13.38 s. This indicated that the variance in average vehicle delay caused by D3ynQN was larger than that of MD4-AQF. A similar inference could be drawn by comparing their whisker ranges (i.e., the difference between the upper and lower bounds in the box plots). The whisker range was 91.56 − 38.51 = 53.05 s from D3ynQN, versus 77.70 − 34.14 = 43.56 s from MD4-AQF. These signified that the policy quality of D3ynQN was inferior to that of MD4-AQF across various traffic load scales. Generally, as an ablated version of MD4-AQF, D3ynQN’s performance was worse than that of MD4-AQF. This result validated the significant impact of “multi-dimensional Q-network” and “Aligned Q-fusion” components in the proposed method.

8.3. Experiment 3: Versus Conventional Control Method

We compared MD4-AQF against a conventional fully actuated signal control (FASC) method in terms of vehicle delay. The technical details of FASC can be found in [25]. Briefly, FASC extended current phases on a second-by-second basis, according to the detected time headways of all vehicles crossing a specific point on each approach lane.

FASC typically utilizes data from inductive loop detectors that are widely deployed at real-world intersections. Such an actuated method has long been applied in traffic engineering. FASC has consistently served as a conventional baseline in many previous studies. Its real-time signal timing strategy could produce highly acceptable intersection performance.

We reused the prepared 2000 traffic demand scenarios via the D-optimal design in the previous experiment. We also performed two simulation runs in each scenario, with signals timed by MD4-AQF and FASC, respectively. We recorded the average vehicle delay at the intersection for the following comparative analyses.

As box plots in Figure 9 show, the medians of the average vehicle delay caused by MD4-AQF and FASC, across the 2000 traffic demand scenarios, were 49.62 s and 61.24 s, respectively. The proposed method achieved an 11.62 s (or 19.0%) advantage over FASC in terms of the median value. Moreover, it produced much lower values for both the top and bottom quartiles. Notably, the upper quartile (57.69 s) for MD4-AQF is even lower than the median (61.24 s) for the conventional method. These signified that the proposed method enabled the signals to operate with overall superior performance. On the other hand, the inter-quartile range (72.90 − 53.36 = 19.54 s) and whisker range (102.16 − 39.47 = 62.69 s) produced by FASC were comparatively wide, meaning that the conventional method resulted in less stable performance than MD4-AQF. In particular, there was an obvious contrast in the upper whisker (102.16 s versus 77.70 s). It was inferred that MD4-AQF expanded the intersection capacity and performed well in some severe demand scenarios.

We then paired the delay data produced by two methods to draw the scatter plot in Figure 10. As shown, the proposed method led to better average vehicle delay than FASC in the great majority of demand scenarios. Such an advantage could be approximately 59 s or 46% in some scenarios. Whereas the conventional method outperformed MD4-AQF in very few scenarios with a relatively small difference in delay. Admittedly, there were still some traffic patterns in FASC’s favor, suggesting its competitiveness as a method that has been widely applied over a long period. A potential direction for future work could be to develop a control mechanism that autonomously switches between the conventional method and MD4-AQF.

We further analyzed the paired average vehicle delay from a statistical perspective. We first performed the Kolmogorov–Smirnov normality test with Lilliefors correction. The resulting statistic, D = 0.073 and P < 0.05, showed no normality in the difference in vehicle delay caused by MD4-AQF and FASC. Thus, we had to resort to the non-parametric substitute of the paired t-test, the Wilcoxon matched-pair signed rank test. The resulting statistic Z was −37.729, as illustrated in Table 2. There was a significant difference in the average vehicle delay at a significance level of 0.05. The number of negative and positive ranks was 1951 and 49, respectively. It meant that MD4-AQF surpassed FASC in terms of vehicle delay in 95.35% of the 2000 demand scenarios.

According to the results of the simulation experiments, the proposed MD4-AQF method enabled an agent to acquire signal timing knowledge effectively. The agent improved the average vehicle delay from more than 130 s to less than 50 s during training. The comparison with another MFDRL method served as an ablation study. It showed that the innovative components in MD4-AQF led to faster, more stable, and more effective training with fewer network parameters. The resulting intersection performance improved by over 14% compared to when these components were removed. Furthermore, the agent trained by MD4-AQF achieved significantly better performance compared to a conventional signal control method in a statistical sense. The quantitative advantage in the median of the average vehicle delay was nearly 20%. Such an improvement stands out when compared to the typical performance gains of 10% to 21% seen in analogous MFDRL studies, relative to conventional actuated control methods, under similar conditions of high-resolution simulation software and peak-period traffic demand.

The motivation for this study lies in unlocking the potential of MFDRL for DRBPS, a topic that has been scarcely addressed in previous studies. Working with DRBPS aims to align with key requirements in traffic engineering, making the research more likely to be implemented in practice. As a consequence, DRBPS brings certain limitations in signal control in real-world intersections and increases the difficulty of agent training. DRBPS presents challenges due to the concurrent yet partially independent decision-making required for the two rings. In contrast to previous studies and our prior work [25], the novelty of this study lies in understanding and addressing these emerging challenges.

Overall, we believe that this study achieves its original intent. The proposed MD4-AQF method successfully implements phase control under DRBPS with signal timing constraints in traffic engineering. The trained agent successfully learns how to make good choices for the phases from a set of fine-grained green time options at one-second intervals.

The prior work, D3ynQN, serves as both a representative method from previous studies and the algorithmic foundation for MD4-AQF. The comparative experiments between D3ynQN and MD4-AQF in Section 8.2 show that directly applying an existing method to DRBPS can yield unsatisfactory results. The extensions proposed in MD4-AQF to adapt D3ynQN for DRBPS have proven effective. They are critical to MD4-AQF significantly outperforming conventional methods.

MD4-AQF achieved a median average vehicle delay of 49.62 s in the tested traffic demand scenarios. In contrast, the medians for D3ynQN and FASC baselines were 56.78 s and 61.24 s, respectively. Here, we explored the practical significance of the performance improvement achieved by MD4-AQF. We consulted the level-of-service (LOS) criteria established in the “Highway Capacity Manual” for motorized vehicles at signalized intersections [51]. MD4-AQF improved vehicle delay from LOS grade E (55–80 s) to grade D (35–55 s). This implies a significant reduction in vehicle stoppage and queuing. The benefits are particularly promising in terms of improving driver experience, alleviating traffic congestion, and reducing exhaust emissions.

9. Conclusions

This study proposes an MFDRL method, termed MD4-AQF, to address the RTSC problem within the context of DRBPS. The purpose is to narrow the gap between research and traffic engineering practices since DRBPS is quite commonly used in real-world intersections. To manage several challenges posed by DRBPS, we make three major contributions:

• We present a “decision point aligning”-based action definition for the concurrent phases in two rings. This approach produces a consistent action space, which is used to determine the green times for both phases simultaneously.
• We present a “multi-dimensional Q-network”-based training algorithm against the large action space containing 600+ permutations of green times for two phases. It can transform hundreds of combinatorial actions into tens of sub-actions.
• We present “aligned Q-fusion” embedded in the algorithm to ensure that two concurrent lagging phases end simultaneously. This approach can fuse paired sub-action values and generate one remaining green time shared by the two lagging phases.

The results of simulation experiments indicate that MD4-AQF can help an agent learn useful knowledge about signal control. Its best-performing policy achieves a median average vehicle delay of 47.89 s during training. The trained agent outperforms another MFDRL method by 14.4%, and a conventional method by 19.0%, in terms of the median delay. More interestingly, we believe MD4-AQF, in essence, provides universal solutions to some general control problems as follows.

• Determining the green time for two concurrent phases is essentially a multi-degree-of-freedom (DOF) control problem, where some DOFs operate in parallel but not strictly simultaneously. Our solution is to align the decision points of DOFs with the latest moment at which any DOF must make a decision. Then, the action should be defined as the move of each DOF after its own decision deadline. Therefore, an agent can control several concurrent DOFs at the same time.
• Learning too many combinatorial actions for two phases is essentially a problem of the “curse of dimensionality” caused by controlling multiple DOFs at once. In such a case, the number of actions grows exponentially with the number of DOFs. Our solution is to perform dimension expansion on a DQN-style algorithm to produce multi-dimensional sub-action values. Each sub-action should correspond to the choice of a specific DOF. By doing so, the number of Q-network outputs can become only linear with the number of DOFs.
• Ensuring simultaneous ends of two lagging phases is essentially a problem where all moves of DOFs are required to be identical in some states. Our solution is to align and fuse sub-action values. This is done by averaging the values of the same sub-action over different dimensions. Then, all DOFs should share and take the sub-action that maximizes the average, allowing them to move in sync.

Although this study focuses primarily on MFDRL-based RTSC at a single intersection, the proposed MD4-AQF method can be seamlessly applied to multiple individually operating intersections within a road network. Given that our MD4-AQF targets the most complex eight-phase DRBPS at a four-leg intersection, it can also be adapted to simpler structures with fewer phases at T-intersections or other types of intersections. However, to achieve more effective coordinated control between multiple intersections, developing communication between the states and the actions of intersection agents may be necessary.

Many challenging issues remain to be adequately addressed before MFDRL-based RTSC methods can be successfully applied in practice; for example, the “reality gap” between the dynamics of simulated and real intersections, as well as the reliable generalization of knowledge across varying intersection geometries and traffic demand patterns. Other promising specific directions for future research include designing reward functions to facilitate coordination across multiple intersections; effectively training from the states and rewards based on partially missing sensor data; and learning robust policies to handle edge cases, such as extreme weather or traffic accidents that an agent encounters infrequently.