Leveraging Multi-Agent Reinforcement Learning for Digital Transformation in Supply Chain Inventory Optimization

Abstract

In today’s volatile supply chain (SC) environment, competition has shifted beyond individual companies to the entire SC ecosystem. Reducing overall SC costs is crucial for success and benefits all participants. One effective approach to achieve this is through digital transformation, enhancing SC coordination via information sharing, and establishing decision policies among entities. However, the risk of unauthorized leakage of sensitive information poses a significant challenge. We aim to propose a Privacy-preserving Multi-agent Reinforcement Learning (PMaRL) method to enhance SC visibility, coordination, and performance during inventory management while effectively mitigating the risk of information leakage by leveraging machine learning techniques. The SC inventory policies are optimized using multi-agent reinforcement learning with additional SC connectivity information to improve training performance. The simulation-based evaluation results illustrate that the PMaRL method surpasses traditional optimization methods in achieving cost performance comparable to full visibility methods, all while preserving privacy. This research addresses the dual objectives of information security and cost reduction in SC inventory management, aligning with the broader trend of digital transformation.

1. Introduction

In recent years, there has been increasing research regarding supply chain (SC) coordination and information sharing aimed at improving overall SC operation through digital processes [1,2,3]. Digital transformation empowers SCs to utilize data-driven decision making in order to increase coordination and visibility. It has proven beneficial in enhancing operational profitability, service quality, and mitigating demand uncertainties for individual firms as well as the SC ecosystem [3,4,5]. During the COVID-19 pandemic, SC uncertainties rose, making the SC even more fragile [6]. Furthermore, with the fierce and various global competition and the increasing complexity of SC networks, coordination and visibility play an increasingly crucial role in reducing the overall cost and uncertainties to all entities in the SC [7,8,9]. However, achieving effective coordination and information sharing is challenging. Firms usually seek to improve their performance by centralized coordination and decision making within complex SC networks [1].

Data-driven coordination in SCs can be categorized into two main types: full information sharing and partial information sharing [10]. Full information sharing involves gathering data from various firms and centralizing them with a trusted party. A decision model is then developed on a central server using historical data to make decisions for all firms using the real-time data [10]. Since decisions are made for all firms using real-time data, it requires coordination of information at every step. On the other hand, in partial information sharing, for example, each SC firm receives the demand forecast of its downstream firms and aims to meet this demand as fully as possible, utilizing its inventory policy model to generate its own demand prediction. However, the error in demand prediction inevitably accumulates over time. Moreover, although the above-mentioned approaches to collaborating and sharing information offer advantages to firms, they also expose firms to potential risks that the shared information may be leaked. The information leakage may lead to financial loss, reputational setback, and negative brand image [11,12]. Generally speaking, current approaches improve SC visibility, but they come with heightened concerns regarding information security. As a result, protecting the privacy of shared information is a major obstacle to SC coordination and impedes the digital transformation of SC optimization.

The development of multi-agent reinforcement learning (MaRL) presents a promising data-driven decision-making solution for improving SC coordination and mitigating the risks of information leakage. MaRL achieves great success in solving various problems. It involves optimizing the behavior of multiple agents within a shared environment [13]. Data can be gathered from the SC’s insight to train the MaRL system for decision making. Each agent in MaRL represents a firm in the SC, allowing each firm to make its own decisions. A key feature of MaRL that helps to preserve data privacy is decentralized decision making [14]. Once the decision model has been trained, each learning agent can make decisions independently. This eliminates the need to share information at every decision step, thereby reducing the risk of information leakage.

We present a Privacy-preserving Multi-agent Reinforcement Learning (PMaRL) approach which balances privacy concerns and SC performance. In line with the broader trend of digital transformation, PMaRL trains in a semi-private manner, requiring firms to share information only during the training. This includes shared embedding data, connectivity details, and training loss among firms. Based on this shared information, each firm can train its own policy and critic networks without disclosing any raw information. The policy networks are used to make decisions based on the current state and the critic networks assess the value of decisions made by the policy networks and guide its parameter updates. Once the policy and critic networks have been trained for all firms, each firm can then make decisions independently using its local state.

In addition, PMaRL incorporates topology information into the models to guide agent training to focus on useful shared information. As a result, PMaRL outperforms the MaRL methods that do not utilize network information in terms of training time, convergence speed, and final optimization objective.

The contributions of our work are as follows:

• Our proposed method only requires sharing limited information during the training stage, overcoming the limitations of existing approaches that require full information visibility across SC.
• The proposed method incorporates SC network topology to optimize agent training by classifying SC firms into distinct groups and leveraging their relationship to enhance the training process.
• Simulation-based evaluations are used to demonstrate the superior performance of PMaRL compared to current optimization methods on practical scenarios of real-world SC networks.

The remaining sections are organized as follows: Section 2 presents related works on information sharing, inventory optimization approaches, and MaRL. Section 3 describes the problem statement of SC optimization. Section 4 presents the details of the PMaRL method. Section 5 evaluates the performance of PMaRL against the existing methods using SC simulation. Section 6 concludes our paper and discusses limitations and future work.

2. Literature Review

2.1. Information Sharing

Multi-stage inventory optimization is a well-studied problem since it is fundamental to studying bullwhip effects, operation cost reduction, and firm cooperation in SCs [15,16]. In SC optimization, information sharing has been viewed as a significant strategy to counter the bullwhip effect and reduce inventory cost [17]. For instance, downstream firms share their projected demand quantity to minimize the disturbances in the SC. This collaborative approach allows all firms to benefit from increased revenues, more agile demand planning, and reduced SC risk [18]. Highlighting the importance of sharing forecasting information within the SC, studies have demonstrated that such sharing results in lower demand prediction errors [19]. Furthermore, information sharing also helps to smooth out the demand curve [17,20].

However, sharing information among different firms introduces the risk of sensitive information leakage, which is a significant challenge for SCs [21,22]. This issue is compounded by the need to balance data privacy with effective collaboration. During the training process, it is crucial to determine what information can be shared without potentially exposing sensitive data. Additionally, it is important to assess whether the distributed decision-making process can increase the benefits of individual firm autonomy while reducing the likelihood of information leakage and potential negative consequences.

2.2. Inventory Optimization Approach

Existing approaches to SC inventory optimization are varied, including stochastic programming, heuristic search, and reinforcement learning (RL) [23,24]. SC inventory optimization involves setting inventory levels to minimize costs and maximize overall efficiency. SC networks can be classified into single-chain and multi-stage networks. A single-chain network is a linear network; whereas a multi-stage SC consists of multiple tiers or levels of SC entities. SC inventory optimization is confronted by challenges such as uncertain demand, complex SC networks, and complicated interactions among SC entities.

To tackle the single-chain inventory problem, Barlas and Gunduz [17] proposed both optimal and heuristic approaches. For multi-stage SC optimization problems, Rong et al. [25] utilized recursive optimization and decomposition aggregation heuristics. In practice, multi-stage SC networks are complex [26] and finding optimal solutions for these complex networks is challenging due to the curse of dimensionality [8,16,23]. Given uncertain demand and complex multi-stage networks, only the optimal solution for the single-chain network can be computed in polynomial time [27,28]. Achieving optimal policy for multi-stage SCs using stochastic programming remains difficult [29]. Existing approaches in such cases are NP-hard, with computational time increasing exponentially with model complexity [28].

2.3. Reinforcement Learning

In an increasingly unstable and complex SC environment, managing SC inventory continues to be challenging in many areas, such as decision making in continuous and discrete processes, the increasing amount of process data, and the intricate external SC environment [7]. Advances in computational resources and machine learning algorithms offer new potential for dynamic, data-driven decision making [30]. Specifically, RL approaches provide end-to-end data-driven solutions for managing SC inventory.

RL primarily focuses on optimizing the actions of either a single agent or multiple agents, typically within simulated environments. In single-agent RL, an agent learns an action policy by exploring and interacting with the environment. In contrast, MaRL involves agents not only interacting with the environment but also developing strategies for competition or cooperation with other agents. This distinct feature of MaRL sets it apart from single-agent RL, as it necessitates that agents account for the dynamics of the multi-agent system and adapt their policies accordingly.

In single-agent RL, Barat et al. [13] leveraged the actor–critic approach in an SC replenishment scenario, focusing on minimizing wastage and backorder quantities within a firm. Additionally, a deep Q-network was introduced to optimize inventory policies, specifically in the context of the beer game [31]. While single-agent RL approaches excel at optimizing the cost for individual entities within the SC, they achieve this independently. However, when each entity optimizes its operation without cooperation, this can lead to a “prisoner’s dilemma” situation [32], potentially degrading overall SC performance. The main drawback of single-agent RL is its limited focus on optimizing the performance of a single entity, often neglecting the broader impact on the entire SC.

In MaRL, studies by Dehaybe et al. [33], Kotecha and Chanona [34], and Nurkasanah [35] applied deep RL approaches to optimize overall SC performance in inventory management and replenishment strategies. These approaches require raw data, such as inventory level and order quantity, from SC entities to train a shared model, which is then used by each SC entity to make informed decisions. While these approaches help SC entities avoid the “prisoner’s dilemma”, they necessitate the sharing of original raw data and even decision models among entities. The primary goal of MaRL is to find optimized policies for multi-agents within a system, but it does not account for data privacy concerns. Although MaRL demonstrates exceptional performance in SC optimization, the issue of privacy remains a significant concern.

3. Research Objective

In this section, we describe the problem settings for multi-stage SC optimization. In many instances, SC optimization occurs under the conditions of incomplete information. For example, some firms may not be willing to share their data or business objectives. Additionally, customer demand is often unpredictable and subject to fluctuations. As a result, obtaining complete information during the optimization process is typically impractical.

The objective of SC optimization is to minimize the inventory costs across the entire SC. A multi-stage SC has the following characteristics [32]:

• Cooperation with minor competition;
• Incomplete information;
• Prisoners’ dilemmas.

In a multi-stage SC, the optimal strategy for firms is to coordinate with one another to avoid competition. When firms act individually, such as by ordering excessive quantities to reduce short-term cost, it ultimately results in higher overall SC costs and increasing long-term costs for individual firms. Moreover, in a multi-stage SC, incomplete information scenarios are common, where firms can only access their own data. The inventory management decisions of each firm also affect others, leading to “prisoner’s dilemma” situations. In these cases, firms must consider the potential actions of others and cooperate accordingly.

Figure 1 shows an instance of a multi-stage SC where each stage consists of a single entity. The SC network consists of several entities, namely, the retailer, wholesaler, distributor, manufacturer, and supplier. Each of these entities is represented by an agent. In a multi-stage SC, upstream agents (e.g., wholesaler) receive stochastic orders from downstream agents (e.g., retailer). In response, they fulfill orders by sending shipments to the downstream agents.

To simulate real-world scenarios, each agent in the network experiences varying order and shipping lead times. Furthermore, each agent within the SC has its own distinct holding and backorder costs, reflecting the unique cost considerations for each agent.

In the multi-stage SC, $C_i^t$ is the SC cost at time t for agent i:

$$C_i^t=c_i^{b}B{O}_i^t+c_i^{h}I{L}_i^t,$$

(1)

where $c_b^i$ and $c_h^i$ are the backorder and holding cost coefficients for agent i, respectively. $B{O}_i^t$ and $I{L}_i^t$ are the backorder and inventory levels for agent i at time t, respectively.

Then, the long-run system-wide cost Z in the following function that we aim to minimize is

$$Z\left(\pi \right)=\sum _{t=1}^T\sum _{i=1}^n{C}_i^t,$$

(2)

where $\pi$ is the set of order policies for all agents and T indicates the maximum number of time steps of the system.

The objective of the SC optimization problem is to find a policy set ${\pi }^{*}$ that minimizes the expected systemic SC cost Z:

$${\pi }^{*}=\underset{\pi \in \Pi }{arg\; min}\mathbb{E}\left[Z\left(\pi \right)\right],$$

(3)

where $\Pi$ is all possible order policy sets. In the next section, we elaborate on how the PMaRL can be used to optimize the SC costs.

4. Methods

In this section, we explore the optimization of the overall inventory cost in a multi-stage SC. To tackle this challenge, we introduce a PMaRL model that incorporates network topology information. The model is trained in a centralized manner to enable effective coordination among agents, but during the decision-making phase it operates in a decentralized manner to ensure privacy. Each agent has its own critic network, which includes an encoder, decoder, and the shared attention structure. During training, agents exchange the embedding information and corresponding loss value. While some raw data could potentially be reconstructed through reverse engineering, this approach still offers a viable solution for preserving data privacy.

4.1. Fundamentals

A multi-stage SC can be considered as a special case of a Markov decision process, defined by a tuple $(S,A,P_A,R_A)$, where S is a set of states; $A=\{A_1,\cdots ,A_n\}$, $A_i$ is the action set of agent i; $P_A$ is the probability distribution of all possible next states given state S and action set A: $S\times A_1\times \cdots \times A_n\to P_A\left(S\right)$; $R_A=\{r_1,\cdots ,r_n\}$, $r_i$ is the expected immediate reward of agent i: $S\times A_i\to r_i$; and n is the total number of agents.

Furthermore, each agent i has its own observation, $o_i$, representing its observable partial information from the global state $s\in S$; and its own policy, ${\pi }_i:o_i\to P\left(A_i\right)$, corresponding to the action probability distribution under the agent’s observation. Agents optimize policy by maximizing their expected returns:

$$J_i\left({\pi }_i\right)=E_{a_i\sim {\pi }_i,\cdots ,a_n\sim {\pi }_n}[\sum _{t=0}^{\infty }{\gamma }^t{r}_i^t(s^t,a_i^t,\cdots ,a_n^t)],$$

(4)

where $a_i\in A_i$ and $\gamma \in [0,1]$ is the discount factor that determines the extent to which the policy prioritizes immediate rewards over long-term benefits.

The gradient of agent policy networks can be estimated using the policy gradient technique [36] as follows:

$${▽}_{\theta }J\left({\pi }_{\theta }\right)={▽}_{\theta }\mathit{log}\left({\pi }_{\theta }\left(a_t\right|s_t)\right)\sum _{t^{\prime }=t}^{\infty }{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}(s_{t^{\prime }},a_{t^{\prime }}),$$

(5)

where $\theta$ represents the learnable parameter. In practice, especially in complex scenarios, it is hard to fully exploit and explore the entire environment to obtain the precise value of the term ${\sum }_{t^{\prime }=t}^{\infty }{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}(s_{t^{\prime }},a_{t^{\prime }})$ as the expected returns in the policy gradient estimator can vary greatly in different training episodes. To address this issue, actor–critic methods [37] are used to augment the original policy gradient approach. These estimate the expected returns using the Q-value function:

$$Q(s_t,a_t)=\mathbb{E}\left[\sum _{t^{\prime }=t}^{\infty }{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}(s_{t^{\prime }},a_{t^{\prime }})\right].$$

(6)

The Q-value function can be learned by a temporal-difference learning technique that minimizes the regression loss of the past Q-value and the reward:

$${\mathcal{L}}_Q{\displaystyle ={\mathbb{E}}_{s,a,r,s^{\prime }\sim D}\left[{(Q(s,a)-y)}^2\right],}$$

(7)

$$y{\displaystyle =r(s,a)+\gamma {\mathbb{E}}_{a^{\prime }\sim \pi \left(s^{\prime }\right)}\left[Q(s^{\prime },a^{\prime })\right],}$$

(8)

where D is a replay memory storing past experiences.

To encourage the RL agents to explore the environment and avoid premature convergence at a local optimal policy, Haarnoja et al. [38] proposes the soft actor–critic approach that introduces the entropy term $log\left(\pi \right(a\left|s\right))$ and incorporates it into the policy gradient:

$${▽}_{\theta }J\left({\pi }_{\theta }\right)=\mathbb{E}\left[{▽}_{\theta }\mathit{log}\left({\pi }_{\theta }\left(a\right|s)\right)(Q(s,a)-\alpha \mathit{log}\left({\pi }_{\theta }\left(a\right|s)\right)-b\left(s\right))\right],$$

(9)

where $b\left(s\right)$ is the baseline of the Q-value function that measures the value of the current state s, and $\alpha$ is a temperature parameter. Accordingly, the loss function of the Q-value is revised by adding the entropy term to Equation (8):

$${\displaystyle y=r(s,a)+\gamma {\mathbb{E}}_{a^{\prime }\sim \pi \left(s^{\prime }\right)}\left[Q(s^{\prime },a^{\prime })-\alpha log\left(\pi \left(a^{\prime }\right|s^{\prime })\right)\right].}$$

(10)

The actor–critic methods involve updating the policy network and critic network using Equations (7) and (9) iteratively. The policy network makes decisions based on the current state and the critic network estimates the expected return of the policy network using the Q-value function.

4.2. Policy Networks

In the PMaRL method, all agents train in a centralized manner and make decisions based on their own local information. Each agent only observes local information during decision making and learns to cooperate during the training process. Each agent has its own policy network and the critic network is partially shared among all the agents. To learn agents’ policies, we propose an approach using MaRL and an SC network topology and we provide further details on this approach below.

State variables: In a multi-stage SC, agents are required to make decisions at every time step. The agent i interacts with other agents and the environment by observing state $o_i^t$, taking action $a_i^t$, observing new states $o_i^{t+1}$, and obtaining reward $r_i^t$. The local state of agent i, $s_i^t$, is a tuple $(B{O}_i^t,I{L}_i^t,O{O}_i^t,A{S}_i^t,A{O}_i^t)$, where $B{O}_i^t$, $I{L}_i^t$, $O{O}_i^t$, $A{S}_i^t$, and $A{O}_i^t$ are the backorder level, inventory level, on-order items, arrival shipment, and arrival order at time t of agent i, respectively. The observation $o_i^t$ of the agent i at time step t contains its local states during the last m periods: ($s_i^{t-m},s_i^{t-m+1},\dots ,s_i^t$). m is set to 5 in our experiments [14].

Action space: In the actual scenario, each agent can order an arbitrary quantity, $[0,\infty )$, to its upstream agent(s). In MaRL, however, the policy networks output with finite size. Hence, the order quantity needs to be constrained to a finite action space, $\mathcal{A}$, which represents all possible actions an RL agent can take in the environment. To increase the robustness and stability of the policy network, we define that each RL agent sends order quantity $d+a_i$ to its upstream agent(s), where d is the total demand quantity from its downstream agent(s) [31] and $a_i$ is the output of agent i’s policy network. To regulate the demand fluctuation of the downstream agent(s), the policy network uses $a_i$ to control how much to order based on d. After the total order quantity is determined by $d+a_i$, it is then equally divided by the number of upstream agents and sent to them accordingly.

Reward function: At each time step t, agent i observes its own state $o_i^t$ and selects an action $a_i^t$. The environment then provides a reward $r_i^t$ and generates a new state $s_i^{t+1}$. To minimize the overall SC cost, $r_i^t$ is measured based on the agent’s cost at time step t, which includes the sum of inventory holding and backorder costs. However, due to order and shipment lead times in a multi-stage SC, the reward is not immediately observed after taking action $a_i^t$. Moreover, the reward $r_i^t$ reflects not only a single action but also the combined effect of joint actions from previous periods. As a result, decomposing $r_i^t$ to evaluate the corresponding single action is challenging. So, to estimate action rewards, we define the local reward $r_i^t$ of agent i at time step t as follows:

$$r_i^t=-{\displaystyle \frac{1}{\Delta t}}\sum _{t^{\prime }=t+l{t}_i}^{t^{\prime }+\Delta t}{\gamma }^{t^{\prime }-t}·c_r·C_i^{t^{\prime }},$$

(11)

where $\Delta t$ represents a time window used to estimate the rewards of a given action within the future, $l{t}_i$ is the order lead time of agent i, $c_r$ is the cost coefficient, and $C_i^t$ is defined by Equation (1). Since the action $a_i^t$ impacts the agent’s rewards only after the products have arrived and, with the arrival time being at least $t+l{t}_i$, the local reward $r_i^t$ for agent i can be considered as the average of future costs within this time window. This allows agent i to approximate the reward for its action.

4.3. Critic Networks

Iqbal and Sha [14] proposed a Multi-Actor-Attention-Critic (MAAC) framework, which demonstrated exceptional performance in general MaRL scenarios. However, in SC networks, agents may be connected to varying numbers of upstream and downstream agents. Instead of adopting a broad approach like MAAC, where attention is evenly distributed among all agents, each agent should focus more on its immediate neighbors and adopt a decision-making policy that enhances cooperation with adjacent agents.

To address this issue, we incorporate SC network topological information into our proposed PMaRL framework. An overview of a critic model is depicted in Figure 2. Generally, each agent has its own encoder and decoder layers, while the embedding and attention structure are shared across all the agents. The encoder layer generates state embedding by mapping the observation o into a lower-dimensional representation. By sharing these state embeddings, each agent obtains the data encapsulating the current states and actions of the entire SC. Using the topological information, the shared attention structure extracts the hidden relationships among agents from the agents’ state embeddings, capturing the contributions of neighboring agents. Finally, the decoder maps these contributions and state embeddings into the Q-values to assess the performance of the agent’s action.

Initially, each agent’s observation $o_i^t$ and action $a_i^t$ are fed into the agent-specific encoder $f_i$, which is a multi-perceptron (MLP) network. This process generates the state embedding $e_i^t$ at each time step t, as described by the following equation:

$$e_i^t=f_i(o_i^t,a_i^t).$$

(12)

To model the relationship among agents in critic networks, we use an adjacency matrix M, which is an $n\times n$-dimensional matrix representing the network topology, where n is the number of agents. The SC networks can be considered as a directed graph. Each element $M_{i,j}\in [-1,0,1]$ captures the relation between agent i and agent j. A value of “−1” indicates that agent i is the predecessor (upstream) of agent j, “0” means agent j is not the neighbor of agent i, and “1” signifies agent i is the successor (downstream) of agent j. This matrix enables each agent to focus on the information from its neighboring agents during training. For each agent i, the list of connected neighbors $n{b}_i$ can be derived from the adjacency matrix.

To promote cooperation among agents, we use multiple attention heads for each agent in the shared attention structure [39], shown in Figure 3. Each head has its own set of parameters ($W_q$, $W_k$, $W_v$). It takes the embeddings of agent i and its neighbors as input to generate the neighbor contribution $X_i$ for agent i:

$$X_i=\sum _{j\in n{b}_i}{\beta }_j{W}_v{e}_j,$$

(13)

$${\beta }_j\propto exp\left(e_j^T{W}_k^T{W}_q{e}_i\right)$$

(14)

where ${\beta }_j$ is the attention weight calculated based on the embedding similarity between agent i and agent j using a bilinear mapping (i.e., scaled dot product operation). Each head captures the latent relationships extracted from both the neighbors’ embeddings and the self-embedding. These latent relationships from all heads are then concatenated to form the final neighbor contribution $X_i$ for agent i.

Once the attention structure generates $X_i$, the decoder g, which is an MLP network, converts $X_i$ to the expected Q-value $Q_i$:

$$Q_i=g_i(e_i,X_i).$$

(15)

In the SC network, agents may differ in the number of input/output edges, and therefore in their neighbor contributions. To classify the connectivity of agents, we define the number of incoming edges as the total number of downstream agents, and the number of outgoing edges as the total number of upstream agents. Agents with the same number of incoming and outgoing edges are grouped together based on the adjacency matrix, denoted as $G\left(i\right)$. $\left|G\right(i\left)\right|$ is a function used to determine the size of group $G\left(i\right)$. Agents within the same group are expected to have similar structures and inventory policies.

To train the critic model, we create a replay memory that stores historical transition data ($s_t$, $a_t$, $s_{t+1}$, $r_t$) to ensure training stability. At each training epoch, the loss function is defined as follows:

$$L_{\theta }=\sum _{i=1}^N{\sigma }_{G\left(i\right)}\left[{(Q_i(o_i,a_i)-y_i)}^2\right],$$

(16)

$${\sigma }_{G\left(i\right)}={\displaystyle \frac{1+log\left(\left|G\right(i\left)\right|\right)}{\left|G\right(i\left)\right|}}$$

(17)

where $y_i$ is defined in Equation (10), $1+log\left(\right|G\left(i\right)\left|\right)$ represents the log-scaled group weight for the group $G\left(i\right)$, and ${\sigma }_{G\left(i\right)}$ denotes the weight assigned to each agent within the group $G\left(i\right)$. This formula incorporates the log-scaled group weight into Equation (10) to balance the influence across groups of varying sizes and to encourage agents from different groups to explore.

Agents with the same number of incoming and outgoing edges exhibit similar functionalities. To account for this, we assign log-scaled weights based on group sizes. In contrast, treating every agent equally, as in the MAAC approach [14], can lead to the critic model disproportionately emphasizing the loss of agents in larger groups. This can lead to bias toward optimizing the loss of these agents, thereby discouraging exploration by agents in small groups. For example, if an SC network consists of 100 retailers and one wholesaler, the MAAC approach might cause the critic model to focus primarily on the retailers due to their similar structure and functionality. As a result, the wholesaler’s critic model may adopt the retailers’ policy to minimize the cumulative cost, limiting the scope of exploration. In such cases, the MAAC approach often leads to a local minimum and inadequate exploration. Unfortunately, real SC networks frequently have a large number of agents within the same group [26]. The MAAC approach may struggle to balance the exploration and exploitation, leading to suboptimal SC performance, as we demonstrate in Section 5.

In PMaRL, we tackle this issue by incorporating log-scaled group weights, allowing the critic model to fairly assess the actions of agents across different groups. The log-scaled group weight for group $G\left(i\right)$ is calculated as $1+log\left(\right|G\left(i\right)\left|\right)$. The weights assigned to agents within the same group, ${\sigma }_{G\left(i\right)}$, are distributed equally according to these log-scaled group weights. Compared to the MAAC approach, this method ensures that the agents in small groups have a greater impact on the loss function, enabling the critic models to better understand and address the imbalance in the SC network topology.

Subsequently, we apply the loss function to all agents and update their policy network gradients using Equation (9). The baseline $b\left(s\right)$ is set to $\mathbb{E}\left[Q_i(o_i,(a_i,a_{\setminus i}))\right]$ to estimate the value of the current state. Each agent samples the actions $a_{\setminus i}$ of other agents from their current policies to estimate its own gradient. Essentially, Equation (9) compares the value of the current action with the average action values of other agents to evaluate whether the current action contributes to the increased rewards.

5. Experiments and Results

In this section, we present a multi-stage SC simulation as an evaluation testbed. The experiments evaluate SC performance in terms of cost for various approaches, as well as the convergence of different MaRL methods. The simulation experiments were conducted on a platform equipped with an 11th Gen Intel Core i9 11900KF CPU, 64 GB memory, and NVIDIA RTX 3090 GPU.

5.1. Simulation Setup

To evaluate the performance of various inventory management approaches across different SC networks, we developed an SC simulation model as a test bed for assessing total SC costs. In this simulation model, each agent represents an entity within the SC network and is connected according to the network’s topology. At every time step of the simulation, each agent follows its respective inventory management policy to determine an order quantity, which is then sent to its upstream partner(s). The upstream agent fulfills the order based on its current inventory levels and initiates a shipment. If the order quantity exceeds the available inventory, the unfulfilled portion is recorded as a backorder, which incurs additional backorder costs. Upon receiving a new shipment, agents prioritize fulfilling any outstanding backorders. The total SC costs are calculated as outlined in Equation (2).

In our experiments, we construct four distinct SC networks based on the beer game (BG) [40], and three real SC networks (R1, R2, and R3) [26], which are illustrated in Figure 4;

Table 1. SC network characteristics.

SC Network	No. Stages	No. Nodes	No. Edges	Network Complexity
Beer Game (BG)	4	4	3	1.5
Real Network 1 (R1)	5	17	18	2.82
Real Network 2 (R2)	4	22	39	1.91
Real Network 3 (R3)	5	27	31	2.27

summarizes their characteristics [41]. The columns labeled stages, nodes, edges, and network complexity correspond to the number of stages, nodes, and edges in each SC network. Network complexity represents the uncertainty inherent in dynamic logistics within the SC systems [41].

As shown in Table 1, the real SC networks are selected to show the SC performance at different levels of network complexity. For instance, R1 and R3 exhibit higher network complexity with numerous linkages (see Figure 4a,c). In contrast, R2 shows lower network complexity, featuring a simple and repetitive connection topology among wholesaler agents (see Figure 4b). The objective is to evaluate the SC costs of the PMaRL model across different network complexity levels. For each network topology, we adjust the shipment lead time from 1 to 3 days. Increasing the lead time results in higher inventory and backorder costs, thereby allowing us to assess the total cost under various scenarios.

In the simulation, the customer demand for agent i at each time step is modeled as $\mathcal{P}\left({\lambda }_i\right)$, where $\mathcal{P}$ denotes the Poisson distribution, and ${\lambda }_i$ is the mean of the Poisson distribution. The The Poisson distribution is chosen because it is commonly used for demand pattern generation [28]. To ensure a fair comparison, the total average demand generated across all retailers in an SC is fixed at 32, i.e., $\sum {\lambda }_i=32$ [42]. The action spaces and the demand values are defined in

Table 2. SC network parameters.

SC Network	Retailers	Retailer Demand	Action Space $\mathcal{A}$
BG	1	$d_1^t\sim \mathcal{P}\left(32\right)$	{−8, 24}
R1	4	$d_1^t=d_2^t=d_3^t=d_4^t\sim \mathcal{P}\left(8\right)$	{−4, 10}
R2	9	$d_1^t=d_2^t=\dots =d_9^t\sim \mathcal{P}(32/9)$	{−3, 6}
R3	8	$d_1^t=d_2^t=\dots =d_8^t\sim \mathcal{P}\left(4\right)$	{−3, 6}

:

The number of retailers varies by network: there are 1, 4, 9, and 8 retailers in BG, R1, R2, and R3, respectively, based on the SC network topology in the dataset. The action space $\mathcal{A}$ is influenced by the Poisson distribution, with lower Poisson means leading to smaller action spaces [31].

In the SC simulation, the duration is set to 50 time steps, each representing one day. The first 5 days serve as a warm-up period. The total cost Z (see Equation (2)) is calculated by summing the backorders and holding the costs of all agents in the SC after the warm-up period. To ensure the robustness of the experimental results, the total costs are averaged over 100 simulation runs.

5.2. SC Inventory Management Settings

We compare our proposed PMaRL method with other SC inventory management methods, including both traditional and MaRL approaches, across different SC networks, as shown in Table 1.

For traditional methods, the first comparison is the base-stock (BS) policy where a firm places a replenishment order whenever inventory falls below a predetermined base-stock level [43]. The reason for choosing the BS method is that it is straightforward to implement and widely used in SC inventory optimization scenarios [3]. This base-stock level is calculated using the formula $\overline{D}\ast LT+Z_{ser}\ast std\left(D\right)\ast \sqrt{LT}$, where D is the set of retailer demands, $LT$ is the order lead time, $Z_{ser}$ is the standard score for a $95\%$ service level, and $std\left(D\right)$ is the standard deviation of demand.

The second traditional method is the decomposition–aggregation (DA) heuristic, which approximates the optimal base-stock level by breaking down the SC network into simpler serial SCs [25]. The key advantage of using the DA method is that it does not require the details of the cost function and achieves high SC cost reduction in complex SC networks [44]. DA calculates the optimal base-stock levels for these serial SCs and then aggregates the results to derive a heuristic base-stock level. Similarly, the service level in DA is also set to $95\%$. The key distinction between these methods is that the BS approach uses local information to determine the base-stock level, while DA uses the information from all entities in the network to approximate the optimal level.

For the MaRL-based methods, we also compare two methods. The first is the MAAC approach [39], which allows agents to exchange their attention embeddings for better coordination. The second is our proposed PMaRL method. The last method is a variation of the PMaRL approach with full visibility (FV). The differences between the compared MaRL approaches are given in

Table 3. Differences between MaRL methods.

Method	Critic Network	Policy Network
MAAC	Without SC network topology	Local Information
PMaRL	With SC network topology	Local Information
FV	With SC network topology	Global Information

. In the critic network, both the PMaRL and FV models utilize the SC network topology, whereas the MAAC does not. In the policy network, both MAAC and PMaRL rely only on local information as input, while the FV model requires information from all agents.

5.3. Results—Comparison with Traditional Methods

Figure 5 presents a comparison of the normalized costs across different methods for various lead times, with costs normalized against those of the PMaRL approach. In the figure, the green, orange, and blue bars represent the normalized costs for the BS, DA, and PMaRL approaches, respectively.

The data in Figure 5 indicate that as the network complexity increases, the PMaRL approach consistently outperforms other methods, achieving greater cost savings in the SC. Specifically, the PMaRL approach reduces inventory costs by a factor of 1.4 to 1.6 compared to other methods in high-network-complexity scenarios. The PMaRL approach surpasses the BS method in all cases and shows notable inventory cost savings in high-network-complexity scenarios. The DA approach outperforms BS due to its increased visibility.

The PMaRL approach achieves comparable performance to DA in low-network-complexity scenarios and significantly better performance in high-network-complexity scenarios.

Overall, the experiment results consistently demonstrate that the PMaRL approach effectively reduces inventory costs, underscoring its effectiveness and versatility in handling complicated SC systems. Its ability to achieve cost reductions that benefit all entities highlights the practical value and applicability of the PMaRL approach in real-world SC scenarios.

5.4. Results—Comparison Among Different Multi-Agent-Based RL Methods

Figure 6 demonstrates the normalized cost for different MaRL methods under various lead times, with costs normalized against those of the FV approach. In the figure, the red, purple, and green bars represent the MAAC, FV, and PMaRL methods, respectively. The comparison with the FV approach highlights the differences in performance when full visibility is available versus when it is restricted, whereas the comparison with the MAAC approach emphasizes the advantages of the PMaRL method, which incorporates network topology and logarithmic group weight scaling.

In general, the PMaRL approach outperforms the MAAC approach as the number of nodes in the SC network increases. This is because in larger networks with numerous agents that have similar structures and functions, the MAAC approach tends to focus on minimizing the losses of these agents. This potentially leads to local optima and limits further exploration. By incorporating grouping based on network topology, the PMaRL approach strikes a more effective balance of exploration and exploitation, preventing premature convergence to suboptimal solutions.

When comparing the PMaRL approach to the FV approach, we observe comparable performance across all scenarios. This demonstrates that the PMaRL approach is capable of effective cooperation even in cases where information sharing is restricted or where privacy concerns exist. This capability is crucial for maintaining data privacy while still achieving performance levels on par with full visibility models.

5.5. Results—Training Time and Convergence

Figure 7 compares the average training time per episode among the PMaRL, FV, and MAAC approaches. The variations between different training episodes are negligible. The PMaRL approach is slightly faster than the FV approach. This small difference arises because the PMaRL approach only requires local information as input in its policy network, while the FV approach processes global information. As the number of nodes in the SC network increases, the PMaRL approach demonstrates faster training time per episode, leveraging network topology to focus agents’ attention on their neighbors. This helps learning to focus on relevant information, thereby reducing training time. In contrast, the MAAC approach requires agents to compute attention across all other agents, leading to distractions from less relevant information and slower training time.

Figure 8 shows the normalized inventory cost curves during the training process for R3, with the cost normalized against the corresponding BS approach costs. The green, red, and purple lines represent the inventory costs of the PMaRL, MAAC, and FV approaches, respectively. Although the MAAC approach converges faster, the PMaRL approach achieves lower overall inventory costs, further demonstrating its ability to effectively balance exploration and exploitation during training. Compared to the FV approach, the PMaRL approach yields similar inventory costs but converges more quickly.

5.6. Discussion

Overall, the PMaRL approach consistently outperforms traditional methods in complex SC networks. This success is due to the adaptability of the MaRL model, which generates effective decisions across various scenarios.

Traditional methods typically excel in simple SC networks, especially in the BG network, but struggle with large and complex ones [33]. In contrast, the PMaRL approach utilizes network topology information and explores various inventory policies to establish a robust strategy for the entire SC network. The MaRL approach enables policy models of agents to learn collaboratively during training, enhancing coordination in distributed decision making.

The PMaRL approach is particularly effective for optimizing complex SC networks because it leverages network topology information and does not require prior knowledge for decision making. In comparison, the MAAC approach performs well only in simple SC network topologies. The topology of an SC network has a major impact on the agent’s actions. As each agent’s demand originates exclusively from its downstream, the behaviors of the agent are significantly influenced by its neighbors. Without incorporating network topology information, the MAAC approach fails to fully explore these relationships, leading to higher inventory costs [45]. The PMaRL approach, on the other hand, incorporates a shared attention structure that utilizes network topology and incorporates logarithmic scale group weights in the loss function, making it more adept at handling complex SC network scenarios. Consequently, as the number of nodes in the SC network increases, the PMaRL approach consistently outperforms the MAAC approach.

The PMaRL approach delivers similar performance with faster convergence compared to the FV approach. This indicates that agents can effectively cooperate while making decisions based on local information. The FV approach, however, requires full visibility of information from all agents, which poses privacy concerns in real-world SC scenarios. The main contribution of the PMaRL approach is to address these privacy concerns by sharing only partial information during the training process, allowing agents to make independent decisions during actual operation.

6. Conclusions and Future Works

This paper introduces a novel data-driven decision-making approach, PMaRL, using a MaRL model that leverages network topology to optimize inventory costs across the entire SC. PMaRL trains in a semi-private manner, requiring firms to share information only during the training while being able to make decisions independently using just its local states. Our SC inventory optimization experiments show that the PMaRL approach outperforms traditional methods and MAAC while achieving performance comparable to the FV method. The main contributions of the PMaRL approach are as follows: (1) Enhanced SC coordination and improved training time; (2) Decentralized decision making to reduce the risk of information leakage.

However, some data are still required to be shared during the training, e.g., the model losses. Moving forward, our future research will focus on enhancing privacy protection for firms by integrating federated learning (FL) into our model. This approach will allow firms to train their models locally without sharing raw data. By combining FL with MaRL, we aim to reduce the risk of data leakage while simultaneously improving performance through better SC coordination among firms.

Other than that, further research will also focus on the issue of SC resilience. Especially after the COVID-19 pandemic, the highly uncertain environment has accelerated the demand for building resilient SCs [46]. It is a difficult task for SC agents to be required to make decisions well in normal scenarios and be capable of recognizing an emergency scenario and swiftly recovering from unexpected disruptions. A possible solution may be to use a transfer learning technique. When the demand patterns change, well-trained decision models can quickly adapt with minimal additional data. As a result, SC agents can adapt to disruptions.