Scalable Traffic Signal Controls Using Fog-Cloud Based Multiagent Reinforcement Learning

Abstract

Optimizing traffic signal control (TSC) at intersections continues to pose a challenging problem, particularly for large-scale traffic networks. It has been shown in past research that it is feasible to optimize the operations of individual TSC systems or a small collection of such systems. However, it has been computationally difficult to scale these solution approaches to large networks partly due to the curse of dimensionality that is encountered as the number of intersections increases. Fortunately, recent studies have recognized the potential of exploiting advancements in deep and reinforcement learning to address this problem, and some preliminary successes have been achieved in this regard. However, facilitating such intelligent solution approaches may require large amounts of infrastructure investments such as roadside units (RSUs) and drones, to ensure that connectivity is available across all intersections in the large network. This represents an investment that may be burdensome for the road agency. As such, this study builds on recent work to present a scalable TSC model that may reduce the number of enabling infrastructure that is required. This is achieved using graph attention networks (GATs) to serve as the neural network for deep reinforcement learning. GAT helps to maintain the graph topology of the traffic network while disregarding any irrelevant information. A case study is carried out to demonstrate the effectiveness of the proposed model, and the results show much promise. The overall research outcome suggests that by decomposing large networks using fog nodes, the proposed fog-based graphic RL (FG-RL) model can be easily applied to scale into larger traffic networks.

1. Introduction

With growing global populations, urbanization, and automobile ownership, urban transportation networks continue to experience increasing traffic congestion, with severe consequences that include travel delay, driver frustration, increased emissions, and reduced safety. The control of traffic at urban intersections which can help reduce congestion can be classified broadly as: passive and active. The former entails no explicit control over traffic operations at a given intersection; thus, movement through the intersection relies on driver awareness and compliance to the rules of the road (e.g., traffic signs and unsignalized roundabouts). On the other hand, active intersection control directly restricts some traffic movements to enable others at any given time block. The most prominent of these is the traffic signal. While traffic signals provide a relatively safe method of intersection control, efficiency has been elusive, particularly in intersections with high traffic volumes. According to the Federal Highway Administration (FHWA), poor signal timing can account for up to 10% of total traffic congestion [1,2]. The optimization and control of traffic signals represent a key strategy for the management of traffic congestion and improving traffic conditions in urban areas.

Given the significance of traffic signal control (TSC) in urban mobility and consequently on social and commercial activity, TSC research has received much attention. Broadly, TSC mechanisms can be placed into two categories: fixed-time traffic control and real-time traffic control. The former typically uses a pretimed program that controls the cycle and split times. Webster (1958) was one of the earliest researchers to present a fixed-time control model which had sought to minimize the average delay of vehicles [3]. For traffic flow conditions that are stable with little or no randomness, fixed-time traffic control is well suited. However, fixed-time models are unsuitable for highly stochastic and unstable traffic flow conditions. For these conditions, real-time traffic control that is responsive to traffic conditions is more suited. In real time traffic control, real-time traffic data are used, thereby allowing the signals to adjust accordingly, duly accounting for variations in traffic flow. A widely used real-time traffic controller is the actuated signal, which regulates its cycle and timings based on real-time traffic data received from detectors and sensors. Several applications of actuated signals have been developed and deployed successfully; yet still, actuated signals are not always effective in efficient control of traffic across multiple intersections simultaneously in large road networks This adversity arises when the signal is unable to cooperate with the signals at other intersections. Over the decades, several studies have shown that TSC methods such as actuated and pretimed controls are adequate for a single intersection or small networks [4,5]; however, but they do not lead to optimal solutions when they are applied to large networks.

With the imminent emergence of robust vehicular connectivity and automation technologies, researchers are investigating the efficacy of traffic signal control that leverages such technologies.

Scaling small-intersection TSC solutions to large networks has been a persistent challenge that has been investigated using various optimization algorithms. In recent years, there has been pronounced interest in other solution methods, including those that can take advantage of the plethora of data offered by new and advanced traffic sensors [6,7]. Deep learning and reinforcement learning concepts were introduced several decades ago [8,9]; however, continuing increases in computational capabilities have made their application more feasible, and therefore, fostered a new generation of deep and reinforcement learning algorithms for purposes of continuous and discrete control [10,11]. Parallel with the emergence of smart infrastructure technologies that facilitate real-time data collection and sharing such as roadside units (RSUs) and drones, there have been advancements in machine learning techniques. With these developments, it has become increasingly feasible to solve traffic signal control and other problems associated with intelligent transportation systems.

For these reasons, deep reinforcement learning (DRL) based approaches to TSC solution for large networks has become an increasingly studied topic. Wiering’s study was one of the earliest to propose the use of reinforcement learning algorithms for traffic signal control to minimize city-wide congestion [12]. Prashanth and Bhatnagar proposed reinforcement learning with function approximation for traffic signal control, using Q-learning for adaptive signal control [13]. Chu et al. (2020) proposed a multiagent deep reinforcement learning algorithm that could be applied to large-scale networks; they applied an actor critic network to recurrent neural network with long-short term memory (LSTM) [14]. Wang et al. proposed the cooperative double Q-learning (Co-DQL) model that leverages mean field approximation of all other agents in the network to significantly reduce model complexity and the curse of dimensionality [7]. Guo et al. (2019) presented six traffic control methods that utilize connected and automated vehicle (CAV) capabilities [6].

While the aforementioned studies utilize the state-of-the-art DRL approaches for TSC problems, an oft overlooked topic is the resource constraints that may restrict transportation agencies and other government entities from deploying data-facilitating infrastructure such as RSUs and drones. Additionally, with regard to studies that leveraged CAV capabilities, the realization is that there exists uncertainty on when CAVs will be deployed on public roads.

This study presents an alternative to scalable TSC models that can reduce the number of deployed data-facilitating infrastructure. The proposed model utilizes a graph attention network (GAT) to preserve the topology of the traffic network while focusing on relevant inputs to make traffic control decisions. Doing this allows the model to address large networks as well as variable-sized inputs. The proposed model is applicable where RSUs are deployed in an urban grid-like network, each serving as fog-nodes that collect data via detectors and share with other fog-nodes in its range, utilizing the information to control the phase and duration of the traffic lights in its control. The Q-network utilizes double estimators to approximate $\underset{\mathit{a}}{\mathbf{max}}\mathit{E}\left\{{\mathit{Q}}_{\mathit{t}}\left({\mathit{s}}_{\mathit{t}+\mathit{1}},\mathit{a}\right)\right\}$ instead of maximizing over the estimated action values in the corresponding state to approximate the value of the next state (as is the case in standard Q-learning). Therefore, performance overestimation is avoided. Overall, the model extracts node embeddings from fog node features while also constructing an adjacency matrix that maps the topology of the connected fog nodes, which, in turn, are passed through the attention layer to be used for the Q-network. Unlike the models in the literature, the proposed model considers the preservation of network topology in the TSC problem using GATs. Additionally, recognizing that until CAVs are deployed on public roads, CAV-related solutions proposed in the literature cannot be applied in the practice, this paper proposes an intelligent, scalable traffic control model that can be integrated into large, urban networks without using CAVs directly.

2. Background

2.1. Reinforcement Learning

In general, reinforcement learning (RL) utilizes feedback of decisions, observations, and rewards. Deep reinforcement learning (DRL) combines RL with deep learning, which allows for end-to-end training of multilayer models that can solve complex problems. This is particularly useful for sequential decision making such as in robotics, video games, and traffic operations [10,14,15,16,17,18].

Due to the data-driven nature of traffic operations, and fueled by advancements in sensor and communication protocols, RL and DRL have been increasingly utilized to address problems in the transportation engineering domain. These applications include vehicle routing, signal control, vehicle control, and traffic operations. Du et al. (2021) presented the GAQ-EBkSP method, a DRL-based framework that dynamically routes urban traffic [19]. In the domain of individual signal timing problems, Li et al. (2016) proposed a DRL-based method for traffic signal timing, utilizing deep neural networks as the backbone for reinforcement learning [20]. For direct vehicle control, Koh et al. (2020) presented DRL-based real-time navigation using deep Q-learning based simulation, which has significant potential benefits particularly in urban transit studies [21].

One of the most popular single-agent RL method is Q-learning. Q-learning is a model-free reinforcement learning approach that can be considered as asynchronous dynamic programming, where agents learn optimal policies in Markovian domains through solving sequential decision making [22]. This is achieved by estimating the optimal value, ${\mathit{Q}}^{*}\left(\mathit{s},\mathit{a}\right)=\underset{\mathit{\pi }}{\mathbf{max}}{\mathit{Q}}^{\mathit{\pi }}\left(\mathit{s},\mathit{a}\right)$, for each action $\mathit{a}$ during state $\mathit{s}$. Because most problems have large state and action spaces to learn all action values separately, a parametrized value function $\mathit{Q}\left(\mathit{s},\mathit{a};{\mathit{\theta }}_{\mathit{t}}\right)$ can be learned instead. Thus, the standard Q-learning update for the parameter from taking action ${\mathit{a}}_{\mathit{t}}$ in state ${\mathit{s}}_{\mathit{t}}$ with observed reward ${\mathit{r}}_{\mathit{t}+\mathit{1}}$ and the subsequently resulting state, ${\mathit{s}}_{\mathit{t}}$ is:

$${\mathit{\theta }}_{\mathit{t}+\mathit{1}}={\mathit{\theta }}_{\mathit{t}}+\mathit{\alpha }\left({\mathit{Y}}_{\mathit{t}}^{\mathit{Q}}-\mathit{Q}\left({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}};{\mathit{\theta }}_{\mathit{t}}\right)\right){\mathit{\nabla }}_{{\mathit{\theta }}_{\mathit{t}}}\mathit{Q}\left({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}};{\mathit{\theta }}_{\mathit{t}}\right)$$

where $\mathit{\alpha }$ is the learning rate, and the target ${\mathit{Y}}_{\mathit{t}}^{\mathit{Q}}$ is defined as:

$${\mathit{Y}}_{\mathit{t}}^{\mathit{Q}}\equiv {\mathit{r}}_{\mathit{t}+\mathbf{1}}+\gamma \underset{\mathit{a}}{\mathbf{max}}\mathit{Q}\left({\mathit{s}}_{\mathit{t}+\mathbf{1}},\mathit{a};{\mathit{\theta }}_{\mathit{t}}\right)$$

where the constant $\mathit{\gamma }\in \left[\mathbf{0},\mathbf{1}\right)$ is a discount factor that adjusts the relative weight between immediate and later rewards.

Q-learning in multiagent reinforcement learning (MARL) differs primarily in that MARL is based on Markov game instead of a Markov decision process (MDP) [23]. Similarly to MDPs, Markov games can be represented as a tuple $\left(\mathit{M}, \mathit{S}, {\mathit{A}}_{\mathbf{1},\mathbf{2},\dots ,\mathit{M}},{\mathit{r}}_{\mathbf{1},\mathbf{2},\dots ,\mathit{M}},\mathit{p}\right)$, where M is the number of agents, $\mathit{S}=\left\{{\mathit{s}}_{\mathit{1}},{\mathit{s}}_{\mathbf{2}},\dots ,{\mathit{s}}_{\mathit{m}}\right\}$ is the set of system states, ${\mathit{A}}_{\mathit{m}}$ is the action set of agent $\mathit{m}\in \left\{\mathbf{1},\mathbf{2},\dots ,\mathit{M}\right\}$, ${\mathit{r}}_{\mathit{m}}: \mathit{S}\times {\mathit{A}}_{\mathbf{1}}\times \dots \times {\mathit{A}}_{\mathit{M}}\times \mathit{S}\to ℝ$ is the reward function for agent $\mathit{m}$, and $\mathit{p}:\mathit{S}\times {\mathit{A}}_{\mathbf{1}}\times \dots \times {\mathit{A}}_{\mathit{N}}\to \mathbf{\mu }\left(\mathit{S}\right)$ is the transition function for moving from one state $\mathit{s}$ to another state ${\mathit{s}}^{\prime }$ given action ${\mathit{a}}_{\mathbf{1},\mathit{2},\dots \mathit{M}}$. Partially observable Markov games additionally require $\Omega$, the set of observations of the hidden states, and $\mathcal{O}:\mathit{S}\times \Omega \to {ℝ}_{\mathbf{\ge }\mathbf{0}}$, the observation probability distribution.

In MARL, each agent learns to choose its actions according to their respective strategies. At each time step, the system state transfer occurs by taking the joint action $\mathit{a}=\left({\mathit{a}}_{\mathbf{1}},\dots ,{\mathit{a}}_{\mathit{M}}\right)$ under the joint strategy $\mathbf{\pi }\triangleq \left({\mathbf{\pi }}_1,\dots ,{\mathbf{\pi }}_{\mathit{M}}\right)$, and each agent receives their immediate reward from the joint action. For each agent $\mathit{m}$ under joint policy $\mathbf{\pi }$ and initial state $\mathit{s}\left(\mathbf{0}\right)=\mathit{s}\in \mathit{S}$, the expected discounted reward is:

$${\mathit{V}}_{\mathit{m}}^{\mathbf{\pi }}\left(\mathit{s}\right)={\mathit{E}}^{\mathbf{\pi }}\left\{{\displaystyle \mathbf{\sum }}_{\mathit{t}=\mathbf{0}}^{\infty }{\mathbf{\gamma }}^{\mathit{t}}{\mathit{r}}_{\mathit{m}}\left(\mathit{t}+\mathbf{1}\right)|\mathit{s}\left(\mathbf{0}\right)=\mathit{s}\right\}$$

Additionally, the agent-specific average reward can be found as:

$${\mathit{J}}_{\mathit{m}}^{\mathbf{\pi }}\left(\mathit{s}\right)=\underset{\mathit{T}\to \infty }{\mathbf{lim}}\frac{\mathbf{1}}{\mathit{T}}{\mathit{E}}^{\mathbf{\pi }}\left\{{\displaystyle \mathbf{\sum }}_{\mathit{t}=\mathbf{0}}^{\mathit{T}}{\mathit{r}}_{\mathit{m}}\left(\mathit{t}+\mathbf{1}\right)|\mathit{s}\left(\mathbf{0}\right)=\mathit{s}\right\}$$

2.2. Graph Neural Networks

Graph neural networks (GNNs) are able to preserve acyclic and nonacyclic graph topology, which can enhance road network representation particularly in the context of scalable network traffic signal control [24,25,26,27,28,29,30]. Due to the graphical nature of transportation networks, for both vehicles as well as roadways, GNNs can serve as robust neural network architectures for deep reinforcement learning [26,27]. Deep reinforcement learning requires a robust neural network architecture that enables forward and backpropagation for purposes of model training [28]. Graph convolutional networks (GCNs) can serve as powerful neural networks that can address graph data for deep reinforcement learning [26,27]. The nodes of a GCN layer aggregates its own observed states and those of its neighbors into embeddings. Given different relational graphs, the message propagation is as follows [30]:

$${\mathit{h}}_{\mathit{i}}^{\mathit{l}+\mathbf{1}}=\mathbf{\varsigma }\left({\mathbf{\Sigma }}_{\mathit{m}\in {\mathcal{M}}_{\mathit{i}}}{\mathit{g}}_{\mathit{m}}\left({\mathit{h}}_{\mathit{i}}^{\mathit{l}},{\mathit{h}}_{\mathit{j}}^{\mathit{l}}\right)\right)$$

where ${\mathit{h}}_{\mathit{i}}^{\mathit{l}}\in {ℝ}^{{\mathit{d}}^{\left(\mathbf{l}\right)}}$ denotes the hidden state of node ${\mathit{v}}_{\mathit{i}}$ in the ${\mathit{l}}^{\mathit{t}\mathit{h}}$ layer of the neural network, ${\mathit{d}}^{\left(\mathit{l}\right)}$ is layer dimensions, ${\mathcal{M}}_{\mathit{i}}$ is the set of incoming messages, and ${\mathit{g}}_{\mathit{m}}\left(·\right)$ is the transformation for the message from the nodes.

In essence, these node embeddings can address problems caused by variable length inputs to perform various sequential learning tasks given graph data, and error terms can be used to backpropagate to perform the requisite gradient descent for parameter tuning purposes.

3. Methodology

DRL Model Architecture

The fog-based graphic RL (FG-RL) model for TSC presented in this paper uses a scalable and decentralized methodology. Due to the inherent complexity associated with large networks, centralized models may not converge. Therefore, this study utilizes a decentralized approach. The graphical structure of the network topology is preserved with traffic signals and intersections, along with their relative adjacencies. The fog arrangement determines the topology of the connected entities and the number of the connected intersections within its range. As such, the adjacency matrix containing the relative adjacencies and connectivity of intersections vary corresponding to the distribution of RSUs (and in turn, the fog nodes) in the network and the number of road intersections overseen by each RSU. In DRL architecture, each RSU is represented as a fog node, which serves as an agent that makes decisions to select traffic signal phases for each of the intersections it oversees, with an overall goal of congestion reduction.

The network topology and information attention are modeled using GAT. The fog node can oversee multiple intersections, some of which may have few or no queued vehicles. Therefore, it must learn to divert attention away from relatively uncongested intersections and focus more on congested intersections. However, a given intersection’s congestion levels can vary drastically between episodes or even across different time-steps in one episode. As a result, applying an attention model can facilitate the learning process under conditions when such variations exist.

Each fog node $\mathit{i}$ produces node embeddings that encode node features ${\mathit{h}}_{\mathit{i}}$. The state is a tuple of $\mathit{N}\times \mathit{F}$ node feature matrix ${\mathit{X}}_{\mathit{t}}$ and an $\mathit{N}\times \mathit{N}$ adjacency matrix ${\mathit{A}}_{\mathit{t}}$, where $\mathit{N}$ is the total number of nodes, and $\mathit{F}$ is the number of features in each node. The feature matrix considers the states consistent with those in the literature [7,14], namely, (i) the cumulative delay of the first vehicle on each incoming lane at an intersection, and (ii) the total number of approaching vehicles on each incoming lane.

The network architecture is shown in Figure 1. At each time-step $\mathit{t}$, the node feature matrix ${\mathit{X}}_{\mathit{t}}$ is fed as the input into a fully connected encoder denoted $\mathbf{\phi }$ that generates node embeddings ${\mathit{H}}_{\mathit{t}}$ in $\mathit{d}$ dimensional embedding space $\mathcal{H}\mathbf{\in }{ℝ}^{\mathit{N}\times \mathit{d}}$

$${\mathit{H}}_{\mathit{t}}=\mathbf{\phi }\left({\mathit{X}}_{\mathit{t}}\right)\mathbf{\in }\mathcal{H}$$

The node embeddings then are passed through the graph convolution with attention mechanism.

Figure 1. GAT Model Architecture.

Figure 1. GAT Model Architecture.

The adjacency matrix is weighted using the following attention mechanism:

$$H_t^{\prime }=GAT\left(H_t, A_t\right)=\alpha H_{t}W+b$$

where ${\alpha }_{ij}$ are coefficients computed by the attention mechanism defined in the literature [31]:

$${\alpha }_{ij}=\frac{\exp \left(LeakyReLU\left({\mathit{a}}^T[\mathit{W}{h}_i|\left|\mathit{W}{h}_j\right]\right)\right)}{{\mathsf{\Sigma }}_{k\in \left({\mathcal{N}}_i\right)}\exp \left(LeakyReLU\left(a^T[\mathit{W}{h}_i|\left|\mathit{W}{h}_k\right]\right)\right)}$$

The output of the GAT layer is then used as inputs to the Q network to obtain the Q values. Further, experience relay and soft target update are utilized to enhance learning [11,31], and the model is trained on randomly sampled batches from a replay buffer. Thus, the architecture can be summarized as follows:

• FCN Encoder $\phi$: Dense (32) + Dense (32)
• GAT Layer $GAT$: GATConv (32)
• Q Network: Dense (32) + Dense (32) + Dense (64) + Dense (32)
• Output Layer: Dense (5)

4. Case Study

The case study utilized the Simulation of Urban MObility (SUMO) for traffic simulation [32], an open-source simulator that enables detailed tracking of vehicle and traffic light parameters. Sumo is a highly portable, microscopic, and continuous traffic simulation package designed to handle large networks. For an initial proof of concept, a small 6-node network is considered (Figure 2). The traffic signal parameters are defined as the pre-timed phases, with the RL agents selecting the appropriate phase for each intersection. Further, the lane-change parameters for the vehicles follow the LC2013 model, provided in SUMO. The vehicles are defined for car-following behavior but are permitted to change lanes if needed.

4.1. Network Descriptions

A small grid network was used for numerical experimentation, as shown in Figure 2. Two network settings are considered for numerical experimentation. The first setting utilizes a “smart cities” approach, where each intersection is connected via a central controller in a cloud environment. This setting is a fully observable Markov decision process or MDP. It must be noted that this is an ideal setting that has no constraints, meaning that all entities are assumed to be connected. While this can be achieved easily in simulation, it will need many connectivity facilitating infrastructure units to ensure that the entire network is connected. This could be problematic particularly at large networks.

The second setting utilizes the proposed fog-node approach, where intersections are grouped together by a small number of connectivity-facilitating infrastructure such as RSUs or drones. To assess the effects of multiple facilitating infrastructure with varying numbers and arrangements of connected intersections, this numerical example was conducted with two and three fog nodes, detailed in Figure 3. As previously stated, the two benefits of segmenting the whole network into smaller fog nodes are the improved scalability and the possibility of reducing the infrastructure (number of RSUs and/or drones) required to facilitate the intelligent TSC models.

Each westbound and eastbound road segment entering signalized intersections is a two-lane arterial comprised of a through lane and a left-turn lane. Each northbound and southbound road segment consists of a single through lane. Vehicles enter each outer road segments (10 total) at a flow rate of 2200 vehicles/hour. The vehicle origins and flows are randomly distributed.

Figure 3. Fog node deployment arrangements with connected intersections (0–3, 1–4, 2–5) (left); (0–1–2, 3–4–5) (middle); and (0–1–3, 2–4–5) (right).

Figure 3. Fog node deployment arrangements with connected intersections (0–3, 1–4, 2–5) (left); (0–1–2, 3–4–5) (middle); and (0–1–3, 2–4–5) (right).

4.2. Markov Decision Process Settings

Action Space

Each fog node controls the traffic signals in its range. As shown in Figure 3, the fog nodes control specific intersections under their “jurisdiction”. The fog node arrangements are as follows, with the top left intersection denoted as 0, and subsequent intersections are numbered from left to right and top to bottom.

Arrangement 1: intersections 0–3, 1–4, and 2–5 share information and control.

Arrangement 2: intersections 0–1–2 and 3–4–5 share information and control.

Arrangement 3: intersections 0–1–3 and 2–4–5 share information and control.

Fully-Observable: all intersections share information and control.

In the (0–3, 1–4, 2–5) arrangement, there are a total of three fog nodes, sharing data across only connected intersections. In the remaining two arrangements, there are only two fog nodes. Each signal can have one of five pre-determined phases, as is consistent with most literature and the practice [7,14]: east–west straight, east–west left-turn, three straight and left-turn phases for east, west, and north–south.

State Space

The local state observed within each fog node is defined as follows:

$$s_{k,t}=\left\{wai{t}_{k,t}\left[lane\right], wav{e}_{k,t}\left[lane\right]\right\}$$

As stated previously, $wai{t}_{k,t}\left[lane\right]$ denotes the cumulative delay of the first vehicle for a given lane in an intersection, and $wav{e}_{k,t}\left[lane\right]$ denotes the total number of approaching vehicles on each incoming lane.

Rewards

The reward function consists of two main penalties:

$$\begin{gathered} r_1=wai{t}_{k,t}\left[lane\right] \\ {r}_2=wav{e}_{k,t}\left[lane\right] \end{gathered}$$

The total reward is the negative weighted sum of the two penalties:

$$r=-{\mathsf{\Sigma }}_{lane}\left({\sigma }_1{r}_1+{\sigma }_2{r}_2\right)$$

where ${\sigma }_1,{\sigma }_2$ are used to scale the two penalties. This numerical example used ${\sigma }_1=1$ and ${\sigma }_2=0.30$.

5. Results

Figure 4 presents a comparison of the training results using 2 fog nodes versus a fully observable system.

For each setting, the model was trained using a soft target update set at $1\times {10}^{-3}$ and a learning rate of $1\times {10}^{-5}$. Each model was trained for a total of 100,000 time steps, with 20,000 time steps for warm-up. Given these training parameters, the training results for the fully observable “smart cities” setting and the fog-node setting are shown in Figure 4. It can be seen that despite the lack of information sharing between fog nodes, the training performance is similar to the fully observable model with the exception of Arrangement 2 (0–1–2, 3–4–5).

However, despite strong training performance, the use of fog nodes results in higher average intersection delay, as shown in Figure 5. Over a 1000 time-step policy replay with randomized seed, the fully observable model exhibits the best performance, with 50.37 s of average intersection delay. On the other hand, the worst performance is obtained from an actuator-based, static traffic light system with an average intersection delay of 243.72 s. Among the fog node models, the models with only two fog nodes performed worse than the model with three fog nodes. The (0–1–2, 3–4–5) model outperformed the (0–1–3, 2–4–5) model by about 30 s (118.25 s vs. 148.26 s), and the (0–3, 1–4, 2–5) model performed the closest to the ideal fully observable scenario with an average intersection delay of 74.23 s.

The primary shortcoming of a fully observable model for traffic signal control problems is that fully observable TSC models are difficult to scale well due to the curse of dimensionality (dramatic increase in complexity as the number of connected nodes increases). These results indicate that the use of separately controlled fog nodes allows for comparable training performance while being more scalable, albeit at the cost of some performance. Based on the discrepancies in average intersection delays across fog node arrangements, there is an opportunity to achieve pareto-optimality, particularly in large networks.

Figure 5. Intersection Average Delay.

Figure 5. Intersection Average Delay.

6. Conclusions

In order to create a more easily scalable and intelligent traffic signal control (TSC) model that can be applied to large networks, this paper proposed the use of graph attention networks (GATs) and fog-node architecture. The added benefit of segmenting large networks into smaller fog-nodes includes the possibility of reducing the number of smart infrastructure units required to facilitate the intelligent TSC models. Multiagent reinforcement learning based models for TSC typically can be affected by the curse of dimensionality. The proposed model addresses scalability in two ways: (i) graph attention that only utilizes relevant node features and neighbor node features to reduce the input complexity, and (ii) fog-nodes that break up the large network into manageable sizes. Preliminary findings show that the proposed model shows promising results that can be scaled into larger networks.

However, their performance in reducing average intersection delay may be relatively inferior compared to a fully observable model. As such, ongoing work on various fog node deployment arrangements and their performance, are expected to provide additional insights on the tradeoff between scalability and performance using the proposed GAT and fog-node architecture. Another promising research direction is to create a simplified or averaged performance within each fog node to reduce the data size and complexity, thereby allowing fog nodes to exchange data between each other to make decisions based on the performance of other fogs.