Comparing Reinforcement Learning Methods for Real-Time Optimization of a Chemical Process

Abstract

One popular method for optimizing systems, referred to as ANN-PSO, uses an artificial neural network (ANN) to approximate the system and an optimization method like particle swarm optimization (PSO) to select inputs. However, with reinforcement learning developments, it is important to compare ANN-PSO to newer algorithms, like Proximal Policy Optimization (PPO). To investigate ANN-PSO’s and PPO’s performance and applicability, we compare their methodologies, apply them on steady-state economic optimization of a chemical process, and compare their results to a conventional first principles modeling with nonlinear programming (FP-NLP). Our results show that ANN-PSO and PPO achieve profits nearly as high as FP-NLP, but PPO achieves slightly higher profits compared to ANN-PSO. We also find PPO has the fastest computational times, 10 and 10,000 times faster than FP-NLP and ANN-PSO, respectively. However, PPO requires more training data than ANN-PSO to converge to an optimal policy. This case study suggests PPO has better performance as it achieves higher profits and faster online computational times. ANN-PSO shows better applicability with its capability to train on historical operational data and higher training efficiency.

1. Introduction

1.1. Motivation

Many chemical processes, like coal combustion and bioreactors, are complex, and thus deriving appropriate models and optimizing outputs is difficult [1,2,3]. Machine learning has shown success in optimizing complex systems such as scheduling electricity prices to manage demand and maximize power grid performance [4,5,6]. This motivates exploration of other machine learning techniques like reinforcement learning (RL) on model-free optimization [7].

RL research has seen many breakthroughs in recent years with new algorithms capable of defeating most humans in difficult games [8,9,10,11]. These algorithms are not specifically designed to play games, but to learn and accomplish general tasks. A real-world example can be seen in OpenAI’s algorithm which learned how to control a robotic hand to solve a Rubick’s cube under disturbances [12].

1.2. Literature Review

One popular RL algorithm in process systems, ANN-PSO, trains an artificial neural network (ANN) to model the system and uses a global optimization method, such as particle swarm optimization (PSO), to select optimal inputs. ANN-PSO is an off-policy algorithm, meaning it can train using data previously gathered under normal operating conditions. In a real case study, ANN-PSO is shown to be effective in minimizing NOx production and high-temperature alarms of a power plant using ANN-PSO to control 68 dampers in a closed-loop operation [2]. This demonstrates that ANN-PSO is effective in learning and controlling high-dimensional systems. There have been other power plant successes with yield maximization, efficiency maximization, and multiobjective optimization [13,14,15,16]. Bhattacharya et al. demonstrates that in a real validation test, ANN-PSO increases $ϵ$-polylysine production in marine bacteria by predicting optimal medium composition [17]. Khajeh et al. shows ANN-PSO achieves high extraction molybdenum extraction from water using silver nanoparticles by determining optimal volumes of the extractant and complex, pH, and adsorption time [18]. There have been other successes for optimization in pharmaceuticals, extraction, and HVAC [19,20,21,22,23,24,25,26]. While this shows ANN-PSO is widely used to optimize process inputs, newer algorithms may offer further improvements.

Actor–critic algorithms are a newer class of RL algorithms which train an actor ANN to select optimal actions given a set of observations and a critic ANN to estimate the value of the set of observations. Proximal policy optimization (PPO) is an on-policy stochastic actor–critic algorithm, on-policy meaning the algorithm must gather the training data, and stochastic meaning the actions taken are sampled from a distribution. Over a 90-day real case study, Filipe et al. demonstrated the efficacy of PPO on minimizing power usage and high level alarms of a wastewater treatment plant [27]. For combined heat and power systems economic dispatch, Zhou et al. found PPO performs as well as conventional methods, but handles more scenarios without recalculation, has better time flexibility, and requires less input information [28]. For an integrated electrical and heating system with wind energy, Zhang demonstrates PPO can optimize under power, demand, and spot electricity price uncertainties to minimize operational costs [29]. Actor–critic algorithms have also been used in optimizing costs for power systems, lot scheduling, hybrid vehicles, autonomous vehicles, brine injection, and drones [30,31,32,33,34,35,36].

1.3. Contributions

While process systems RL applications is an active area of research, there are few papers comparing the algorithms in their applicability and performance in process systems. To investigate if the widely used ANN-PSO can be replaced by newer actor–critic methods, this paper presents a novel comparison between two algorithms—ANN-PSO and PPO—by comparing the methods of the algorithms and evaluating these algorithms on a case study of a stochastic steady-state chemical optimization problem. To explore both algorithms’ potential to achieve an optimal policy, ANN-PSO and PPO are also compared to two benchmark control strategies: nonlinear programming on a first principles model and maximum production. The algorithms’ performance and applicability are compared based on implementation feasibility, system optimization, training efficiency, and online computational time. Furthermore, insight into the behavior of these algorithms is explored through parity plots of the agents’ predicted and actual profits, and sensitivity analysis of the agents’ actions. Below is a summary of the novel contributions of this work.

• This work represents a first-of-its-kind comparison study between PPO and other methods for real-time optimization. Specifically, comparisons are made to maximum production operation (no optimization), optimization using an ML model (artificial neural network) and particle swarm optimization (ANN-PSO), and optimization using a first principles model and gradient-based non-linear programming.
• Our results demonstrate that PPO increases profitability by 16% compared to no optimization. It also outperforms ANN-PSO by 0.6% and comes remarkably close to matching the performance of the FP-NLP method, getting within 99.9% of FP-NLP profits.
• Though more time must be invested into training the system, PPO reduces online computational times, resulting in 10 and 10,000 times faster computation compared to FP-NLP and ANN-PSO, respectively.
• ANN-PSO has higher training efficiency compared to PPO as ANN-PSO converges with ≈${10}^5$ training examples while PPO converges to an optimal policy with ≈${10}^6$ training examples.
• Parity plots suggest ANN-PSO’s lower profits are due to PSO exploiting errors in the ANN, causing ANN-PSO to consistently overpredict profit and select suboptimal actions.
• Comparing PPO and ANN-PSO, PPO has better performance as shown by its higher profits and faster computational times. ANN-PSO has better applicability as shown by its higher training efficiency and capability to train on historical operational data.

The following are the sections and the material they cover. Section 2 introduces the optimization methods and their theory. Section 3 presents the case study of optimizing steady state operation of a continuously stirred tank reactor. Section 4 describes how the optimization methods are implemented and evaluated. Section 5 presents and analyzes the results from the case study. Section 6 summarizes our findings and suggests future work.

2. Materials and Methods

2.1. Markov Decision Process

An optimization problem is considered where an agent interacts with an environment which is assumed to be fully observable. This problem can be formulated as a Markov Decision Process (MDP) where the environment is described by a set of possible states $S\in {\mathbb{R}}^n$, possible actions $A\in {\mathbb{R}}^m$, a distribution of initial states $p\left(s_0\right)$, a reward distribution function $R(s_t,a_t)$ given state $s_t$ and action $a_t$, a transitional probability $p\left(s_{t+1}\right|s_t,a_t)$, and a future reward discount factor $\gamma$. An episode begins with the environment sampling an initial state $s_0\in S$ with probability $p\left(s_0\right)$. The agent then selects action $a\in A$, and receives the reward sampled from the environment $r_t\in R$ with probability $R(s,a)$. The environment then samples from $s_{t+1}\in S$ with probability $p\left(s_{t+1}\right|s_t,a_t)$. The agent selects another action and this process continues until the environment terminates. This process is shown in Figure 1.

The two algorithms of interest—ANN-PSO and PPO—and a benchmark algorithm operate as the agent in the MDP process to select actions. All three algorithms are briefly discussed in Section 2.2, Section 2.3 and Section 2.4 and are summarized in Figure 2.

2.2. Artificial Neural Network with Particle Swarm Optimization (ANN-PSO)

For ANN-PSO, an artificial neural network is trained to approximate the environment reward function by mapping states and actions to an expected reward. This is done by minimizing the mean squared error between predicted and actual reward on collected state–action pair data and is shown in Figure 2 (ANN-PSO offline preparation section). Once the ANN accurately predicts rewards from states and actions, particle swarm optimization (PSO) is used on the ANN to find actions that maximize the expected reward for a given state and is shown in Figure 2 (ANN-PSO online computation section). ANN-PSO is an off-policy algorithm meaning the algorithm can learn from data collected from other policies. ANN-PSO’s weaknesses include overfitting to noise in stochastic environments and being computationally expensive online since PSO performs many iterations to find optimal actions for each state. Below is a brief explanation of ANN-PSO which uses Mnih et al.’s q-learning formulation to train the ANN and Kennedy and Eberhart PSO algorithm [37,38] to search for optimal actions.

To maximize the reward, a state–action value function, $Q^{\pi }(s,a)$, is defined in Equation (1). This equation states that, given an initial state, the policy’s value of a state–action pair is the expected cumulative reward of policy $\pi$.

$$Q^{\pi }(s,a)=\mathbb{E}\left[\sum _{t\ge 0}{\gamma }^t{r}_t|s_0=s,a_0=a,\pi \right]$$

(1)

$\pi$ is a policy mapping states to actions.

An optimal state-action value function $Q^{*}(s,a)$ satisfies the Bellman equation which is shown in Equation (2).

$$Q^{*}(s_t,a_t)={\mathbb{E}}_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}\left[r_t+\gamma \underset{a_{t+1}}{max}{Q}^{*}(s_{t+1},a_{t+1})|s_t,a_t\right]$$

(2)

With Equation (2), $Q^{*}$ can be estimated with a function approximator such as a linear function or a nonlinear function through iteration. In this paper, an ANN is used to estimate $Q^{*}$. Training is done by discretizing each dimension of the state S and action sets A into $\mathcal{N}$ spaces. Every combination of the discretized states and actions are passed to the system, which returns the reward. The neural network is then trained to estimate the reward given a state–action pair. For stochastic environments, the ANN may overfit to noise which causes a plant–model mismatch.

To find the maximum value state–action pair for the estimated $Q^{*}$, PSO is used. PSO is a derivative free optimizer which uses multiple particles, collectively called the swarm, to search the solution space to find an optimal action. The particles are initially given random positions and velocities and then are guided by each particle individual best and the swarm best positions to find the optimal solution of function f. The algorithm is shown in Figure 2 (ANN-PSO online computation section).

For ANN-PSO, the actions are the positions and the function to be maximized is the reward returned by the ANN. For every new state, PSO searches for optimal actions. This process is computationally expensive since there are many particles and iterations that must be performed for every new state encountered.

2.3. Proximal Policy Optimization (PPO)

While ANN-PSO finds the optimal action by learning values of every state–action pair and searching for the optimal action, Proximal Policy Optimization [39] learns an optimal policy to sample actions, given states. PPO is an example of an actor–critic algorithm where a policy network is trained to maximize the cumulative reward, while a critic is trained to determine how much better the new policy is and push the probabilities of those actions up or down. PPO is an on-policy algorithm meaning the algorithm learns from data collected based on the algorithm policy, and thus the algorithm requires interaction with the environment.

The PPO algorithm for training is shown in Figure 2 (PPO offline preparation section). While online, the mean action is selected from ${\pi }_{\theta }\left(a_t\right|s_t)$ and is shown in Figure 2 (PPO online computational section).

For PPO, the policy, denoted as $\pi \left(a\right|s)$, is an action probability distribution function, given a state. $\pi \left(a\right|s)$ is approximated using a neural network ${\pi }_{\theta }\left(a\right|s)$ with parameters $\theta$ optimized to maximize a utility function $U\left(\theta \right)$. $U\left(\theta \right)$ is shown in Equation (3) and represents the policy expected reward from time t to time $t+t_f$ starting from state $s_t$.

$$U\left(\theta \right)={\int }_{\tau }r\left(\tau \right)p\left(\tau \right|\theta )d\tau$$

(3)

$\tau$ is all possible trajectories from state $s_t$, $r\left(\tau \right)$ is the discounted reward of $\tau$, and $p\left(\tau \right|\theta )$ is the probability of $\tau$ given parameters $\theta$. The probability distribution is typically a normal distribution.

Instead of directly maximizing Equations (3) and (4), a surrogate objective with the same gradient is maximized so constraints can be applied to limit update step sizes.

$$U^{*}\left(\theta \right)={\mathbb{E}}_{\tau \sim p\left(\tau \right|{\theta }_{old})}\left[r\left(\tau \right)\frac{p\left(\tau \right|\theta )}{p\left(\tau \right|{\theta }_{old})}\right]$$

(4)

However, using $r\left(\tau \right)$ in $U\left(\theta \right)$ poses a problem as the agent increases probabilities of all trajectories with positive reward which leads to high variance and low sample efficiency. Thus, $r\left(\tau \right)$ is replaced with a generalized advantage estimation ${\widehat{A}}_{\tau }$, which is shown in Equations (5a) and (5b).

$$\begin{array}{c}\hfill {\widehat{A}}_{\tau }={\delta }_t+\left(\gamma \lambda \right){\delta }_{t+1}+...+{\left(\gamma \lambda \right)}^{T-t+1}{\delta }_{T-1}\end{array}$$

(5a)

$$\begin{array}{c}\hfill {\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)\end{array}$$

(5b)

$\lambda$ is a factor for trade-off of bias vs. variance, and $V\left(s_t\right)$ is the expected episode cumulative discounted reward of state $s_t$ given ${\pi }_{\theta }\left(a\right|s)$. $V\left(s_t\right)$ is approximated with a neural network $V_{\theta }\left(s_t\right)$ and, in this paper, shares parameters with ${\pi }_{\theta }\left(a\right|s)$.

${\widehat{A}}_{\tau }$ provides a baseline by comparing action $a_t$ received reward to the previous policy estimated value of state $s_t$ and pushes the agent to increase probabilities of trajectories that have higher rewards than the current policy. Combining Equations (4) and (5a) yields Equations (6a) and (6b):

$$\begin{array}{c}\hfill U^{*}\left(\theta \right)={\mathbb{E}}_{\tau \sim p\left(\tau \right|{\theta }_{old})}[{\widehat{A}}_{\tau }{\eta }_{\tau }\left(\theta \right)]\end{array}$$

(6a)

$$\begin{array}{c}\hfill {\eta }_{\tau }\left(\theta \right)=\frac{p\left(\tau \right|\theta )}{p\left(\tau \right|{\theta }_{old})}\end{array}$$

(6b)

To prevent the agent from making excessively large updates, the surrogate objective is clipped as shown in Equation (7).

$$L^{CLIP}\left(\theta \right)=\widehat{\mathbb{E}}[min({\eta }_{\tau }\left(\theta \right){\widehat{A}}_{\tau },\mathrm{clip}({\eta }_{\tau }\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_{\tau })]$$

(7)

$ϵ$ is a hyperparameter to limit the policy updates.

This limits how much the policy changes the probabilities of trajectories compared to the old policy since ${\eta }_{\tau }\left(\theta \right)$ denotes trajectory probability ratios between the new and old policies. As the policy and value function neural networks share parameters in this paper, the objective function incorporates the value function error. An entropy bonus is also added to encourage the agent to explore and avoid premature convergence to a suboptimal solution. The final objective is shown in Equations (8a) and (8b).

$$\begin{array}{c}\hfill L^{CLIP+VF+S}\left(\theta \right)=\widehat{\mathbb{E}}[L^{CLIP}\left(\theta \right)-c_1{L}^{VF}\left(\theta \right)+c_{2}S\left[{\pi }_{\theta }\right]\left(s_t\right)]\end{array}$$

(8a)

$$\begin{array}{c}\hfill L^{VF}={(V_{\theta }\left(s_t\right)-V_t^{targ})}^2\end{array}$$

(8b)

$c_1$ and $c_2$ are hyperparameters, $S\left[{\pi }_{\theta }\right]\left(s_t\right)$ is an entropy bonus function, and $V_t^{targ}$ is the observed cumulative discounted reward from time t to the end of the episode.

2.4. First Principles with Nonlinear Programming (FP-NLP)

Maximizing the reward by using nonlinear programming (NLP) on a first principles model (FP-NLP) is a reliable and well-accepted approach [40,41,42]. Thus, FP-NLP is used as a benchmark to evaluate the performance of the two previous methods. Using FP-NLP requires thorough understanding of the system governing equations, such as fundamental equations for energy and mass balances, to formulate the first principles model. This first principles model can then be formulated into an optimization problem and solved using NLP algorithms such as an interior-point method for NLP (IPOPT). While this method does not require learning like ANN-PSO and PPO, the requirement for an accurate first principles model makes this method difficult to implement in systems where accurate models are difficult to develop. Furthermore, FP-NLP requires an iterative line search, as shown in Figure 2 (FP-NLP online computation section), which can lead to long computational times.

3. Case Study

These methods were evaluated by having them perform economic optimization of a steady state continuously stirred tank reactor (CSTR) model as shown in Figure 3. The values for the model constants are listed at the end of this section in

Table 1. Model simulation parameters.

Parameter	Value	Unit
c	1.2	${\mathrm{m}}^{2.5}·{\min }^{-1}$
h	8	$\mathrm{m}$
$\mathcal{A}$	0.23	${\mathrm{m}}^3·{\mathrm{kmol}}^{-1}·{\min }^{-1}$
$E_a$	5000	$\mathrm{kJ}·{\mathrm{kmol}}^{-1}$
R	8.314	$\mathrm{kJ}·{\mathrm{kmol}}^{-1}·\mathrm{K}-1$
$\rho$	780	$\mathrm{kg}·{\mathrm{m}}^{-3}$
$c_p$	3.25	$\mathrm{kJ}·{\mathrm{kg}}^{-1}·{\mathrm{K}}^{-1}$
$C_{A,0}$	15	$\mathrm{kmol}·{\mathrm{m}}^{-3}$
$C_{B,0}$	15	$\mathrm{kmol}·{\mathrm{m}}^{-3}$
$T_0$	500	$\mathrm{K}$
$T_{ub}$	800	$\mathrm{K}$
V	100.53	${\mathrm{m}}^3$
$\Delta H_{rxn}$	156	$\mathrm{kJ}·{\mathrm{kmol}}^{-1}$
$P_{A,low}$	2	$\$·{\mathrm{m}}^{-3}$
$P_{A,high}$	20	$\$·{\mathrm{m}}^{-3}$
$P_{B,low}$	5	$\$·{\mathrm{m}}^{-3}$
$P_{B,high}$	50	$\$·{\mathrm{m}}^{-3}$
$P_{C,low}$	2	$\$·{\mathrm{kmol}}^{-1}$
$P_{C,high}$	20	$\$·{\mathrm{kmol}}^{-1}$
$P_{q,low}$	0	$\$·{\mathrm{kJ}}^{-1}$
$P_{q,high}$	$2\times {10}^{-5}$	$\$·{\mathrm{kJ}}^{-1}$
$\sigma$	0.1
N	120

. The CSTR is assumed to be well mixed. A mass balance with species A and B flowing in at rates of ${\dot{v}}_A$ and ${\dot{v}}_B$, respectively, is shown in Equation (9).

$${\dot{v}}_A+{\dot{v}}_B=\dot{v}=c\sqrt{h}$$

(9)

c is a valve constant and h is the liquid height in the tank.

Inside the CSTR, an elementary, liquid phase, endothermic, irreversible reaction occurs with Equation (10).

$$A+B\to C$$

(10)

The rate $-r_A$ is given by Equation (11).

$$-r_A=k{C}_A{C}_B$$

(11)

$C_A$ and $C_B$ are the concentrations of A and B in the reactor, and k is related to temperature with Arrhenius equation as shown in Equation (12).

$$k=\mathcal{A}{e}^{\frac{-E_a}{RT}}$$

(12)

$\mathcal{A}$ is the Arrhenius pre-exponential factor, $E_a$ is the activation energy of the reaction, R is the universal gas constant, and T is the temperature of the reactor.

As the system is at steady state, the mole and energy balance equations for the system are written as Equations (13)–(16).

$$\begin{array}{cc}\hfill \frac{d{C}_A}{dt}=0& =(\dot{v}-{\dot{v}}_B)C_{A,0}-\dot{v}{C}_A-VAexp\left(\frac{-E_a}{RT}\right)C_A{C}_B\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \frac{d{C}_B}{dt}=0& ={\dot{v}}_B{C}_{B,0}-\dot{v}{C}_B-VAexp\left(\frac{-E_a}{RT}\right)C_A{C}_B\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \frac{d{C}_C}{dt}=0& =-\dot{v}{C}_C+VAexp\left(\frac{-E_a}{RT}\right)C_A{C}_B\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill \frac{dE}{dt}=0& =\dot{v}\rho c_p(T_0-T)-V\Delta H_{rxn}Aexp\left(\frac{-E_a}{RT}\right)C_A{C}_B+q\hfill \end{array}$$

(16)

A and B are assumed to have similar densities and heat capacities of $\rho$ and $c_p$, respectively, which remain constant over the operating temperature range. A and B enter with concentrations $C_{A,0}$ and $C_{B,0}$, respectively, $T_0$ is the feed temperature of both species, V is the volume of the reactor, $\Delta H_{rxn}$ is the heat of reaction, and q is the heat input.

Solving for $C_B$ using Equation (13) yields Equation (17).

$$C_B=\frac{(\dot{v}-{\dot{v}}_B)C_{A,0}-\dot{v}{C}_A}{VAexp\left(\frac{-E_a}{RT}\right)C_A}$$

(17)

Plugging this back into Equation (14) yields Equations (18a)–(18d).

$$\begin{array}{cc}\hfill a_0& =\frac{-(\dot{v}-{\dot{v}}_B)C_{A,0}\dot{v}}{VAexp\left(\frac{-E_a}{RT}\right)}\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill a_1& ={\dot{v}}_B{C}_{B,0}+\frac{{\dot{v}}^2}{VAexp\left(\frac{-E_a}{RT}\right)}-(\dot{v}-{\dot{v}}_B)C_{A,0}\hfill \end{array}$$

(18b)

$$\begin{array}{cc}\hfill a_2& =\dot{v}\hfill \end{array}$$

(18c)

$$\begin{array}{cc}\hfill C_A& =\frac{-a_1+\sqrt{a_1^2-4a_0{a}_2}}{2a_2}\hfill \end{array}$$

(18d)

Thus, $C_C$ and q can be written as Equations (19) and (20).

$$\begin{array}{cc}\hfill C_C& =\frac{VAexp\left(\frac{-E_a}{RT}\right)C_A{C}_B}{\dot{v}}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill q& =V\Delta H_{rxn}Aexp\left(\frac{-E_a}{RT}\right)C_A{C}_B-\dot{v}\rho C_p(T_0-T)\hfill \end{array}$$

(20)

The cost per time ($CPT$) function is defined as Equation (21).

$$CPT=(\dot{v}-{\dot{v}}_B)P_A+{\dot{v}}_B{P}_B+q{P}_q-\dot{v}{C}_C{P}_C$$

(21)

$P_i$ is the price of commodity i and is sampled from a uniform distribution ranging from $P_{i,low}$ to $P_{i,high}$.

The agents can manipulate ${\dot{v}}_B$ within the range of $[0,\dot{v}]$ and the temperature set point (Temp SP) within $[T_0,T_{ub}]$ where $T_{ub}$ is the upper temperature limit for steady state operation of the CSTR. The goal is to minimize the cost so the optimization problem can be formulated as Equation (22a) subject to Equations (17), (18a)–(18d),(19), (20), (21), (22a) and (22b).

$$\begin{array}{cc}\hfill \underset{{\dot{v}}_B,T}{min}& CPT\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{Equations}.\left(17\right),\left(18\mathit{a}\right)-\left(18\mathit{d}\right),\left(19\right),\left(20\right),\left(21\right)\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill & T_0\le T\le T_{ub}\hfill \end{array}$$

(22b)

$$\begin{array}{cc}\hfill & 0\le {\dot{v}}_B\le \dot{v}\hfill \end{array}$$

(22c)

For the MDP formulation, the possible states are all combinations of prices, and the possible actions are all combinations of ${\dot{v}}_B$ and T. $p\left(s_0\right)$ is a uniform distribution over the ranges of all four prices, and the reward function is $-CPT$ in Equation (21).

To make the model more realistic, feed concentrations and temperatures were stochastic and sampled from a normal distribution with a given mean and a relative standard deviation of $\sigma$. Actions were also perturbed by sampling from a normal distribution with a mean of the chosen action and a relative standard deviation of $\sigma$. For each set of prices, the agent selects one action and the simulation is repeated N times. The average reward is then calculated and returned.

4. Evaluation

Simulations were run on an Intel(R) Core(TM) i7-6500U CPU with 2.50 GHz clock rate and 8 GB of RAM.

4.1. ANN-PSO Implementation

The neural network was trained to map prices with actions to profits. The prices, $P_A$, $P_B$,$P_C$, and $P_q$, and actions ${\dot{v}}_B$ and T were each discretized into $\mathcal{N}$ spaces. $\mathcal{N}$ ranges from 3 to 14 to test the performance of ANN-PSO given various amounts of training data. All combinations of the prices and actions (${\mathcal{N}}^6$) are fed through the reactor simulation and the profit is returned. The ANNs were configured and trained using Keras [43] with an Adam optimizer. The network trained on the minimum-maximum scaled data and k-fold validation was used to evaluate the networks. The network architecture was optimized by varying number of layers, activation functions, regularization coefficients, and learning rate. The network trained with $N=13$ discretization points performed best and was used for algorithm comparisons. PSO was done using Pyswarms [44]. The hyperparameters can be found in

Table A1. ANN-PSO hyperparameters.

Hyperparameter	Value
Adam learning rate	$1\times {10}^{-4}$
Adam exponential decay rate for the first-moment estimates	0.9
Adam exponential decay rate for the second-moment estimates	0.999
Adam small number to prevent any division by zero	${10}^{-7}$
ANN layers	1
ANN nodes	64
ANN activation function	ReLU
ANN L1 norm regularization coefficient	${10}^{-4}$
ANN Loss function	Mean squared error
PSO cognitive parameter	$1\times {10}^{-3}$
PSO social parameter	$1\times {10}^{-3}$
PSO inertia parameter	0.9
PSO swarm size	100
PSO lower velocity bound	$1\times {10}^{-8}$
PSO upper velocity bound	1
PSO iterations	100

.

4.2. PPO Implementation

PPO training was done using Stable Baselines, which is an RL python package [45]. The PPO agents were trained for ${14}^6\approx 7.5\times {10}^6$ time steps and agents were saved periodically to investigate performance improvement over training time. The agents were trained in a normalized vectorized environment and the network used was 2 fully connected layers of 64 units followed by a long short-term memory layer of 256 units. This was then split into the value and policy functions. Hyperparameters, such as the entropy coefficient, value function coefficient, activation functions, and number of epochs were varied to optimize the agent performance. The final network hyperparameters as well as the particle swarm hyperparameters are listed in

Table A2. PPO hyperparameters.

Hyperparameter	Value
Policy Value Shared Network layers	2 Fully connected layers + 1 LSTM layer
Policy Value Shared Network structure	[64,64,128]
$\gamma$	0
n_steps $\left(t_f\right)$	100
ent_coef ($c_2$)	0.01
learning_rate	$2.5\times {10}^{-3}$
vf_coef ($c_1$)	0.5
max_grad_norm	0.5
lam ($\lambda$)	0.95
nminibatches	4
noptepochs	4
cliprange ($ϵ$)	0.2

. For evaluation, action means from the action probability distributions were used.

4.3. Benchmark Algorithms

For FP-NLP, the case model study was coded into GEKKO, an optimization suite [46]. IPOPT was then used as the NLP solver to minimize the cost. Another strategy evaluated was maximizing the production of product C regardless of the prices. The actions for maximizing production were $T=T_{ub}$ and ${\dot{v}}_B=\frac{\dot{v}}{2}$.

4.4. Testing and Metrics

To evaluate the agents’ overall performance in profitability, sample efficiency and computational time, ANN-PSO agents, PPO agents, FP-NLP, and maximum production chose actions for the same 1000 random price combinations. The random prices are not necessarily examples the agents have seen before, but they are within the range of prices the agents have encountered. The received profits from the stochastic system, predicted profits, and computational times were saved and compared between algorithms. To examine the agent actions and their effects on profits, a sensitivity analysis was performed by holding the prices of B and C at their average values of 27.5 $/${\mathrm{m}}^3$ and 11 $/kmol, respectively. Prices for A and q were discretized into 500 points and ranged from 2 to 20 $/${\mathrm{m}}^3$ and 0 to $2\times {10}^{-5}$ $/kJ, respectively. The agents’ selected actions for each combination of A and q and the simulation returned profit were recorded. For the sensitivity analysis, the simulation was made deterministic by setting all random variables to their mean.

5. Results

5.1. PPO Achieves Higher Profits, but ANN-PSO Has Better Training Efficiency

The methods’ average profits for the 1000 random price evaluation are shown in Figure 4. FP-NLP has the highest average profit at $139.28/min and is followed by PPO best profit of $139.20/min. ANN-PSO best profit is $138.37/min and max production has a mean profit of $120.23/min. This shows both methods achieve a higher profit than the maximum production strategy with ANN-PSO and PPO increase the average profit by 15% and 16%, respectively. PPO is also shown to perform as well as FP-NLP with PPO achieving 99.9% the profit of FP-NLP.

Figure 4 also shows ANN-PSO has better training efficiency than PPO, as ANN-PSO profit overtakes the max production benchmark with $\approx 4.1\times {10}^3$ data points and plateaus with ≈$1.2\times {10}^5$ data points as compared to PPO, which overtakes the max production benchmark with roughly four times the amount of data at ≈$1.6\times {10}^4$ data points and plateaus at ≈$2.6\times {10}^5$ data points. PPO’s lower training efficiency is due to needing to learn both the policy and the value of the state, while ANN-PSO only needs to learn the value of the state–action pairs. Furthermore, ANN-PSO exploration is determined by the fed data while PPO explores based on the stochastic policy. This allows for easier implementation of ANN-PSO since normal operation data—such as when PID controllers or expert personnel controls the process—can be used as training data.

More training examples likely would not improve performance of either method as ANN-PSO and PPO’s average profit starts oscillating near ${10}^7$ training examples.

While ANN-PSO training efficiency is appealing, PPO still converges to policy with higher profits. This can be explained by examining the neural networks’ prediction accuracy, shown by parity and residual plots in Figure 5. Figure 5a shows that ANN-PSO neural network accurately predicts the agent profits, as seen by the high $R^2$ value of 0.998. However, if the residual plot is examined, the prediction error has a mean of almost $3/min, indicating the prediction network is overestimating the profit. This is due to the PSO algorithm exploiting small errors in the neural network that arise from overfitting to stochastic data. Figure 5b shows that PPO value network has a similarly high $R^2$ value. The residual plot shows that PPO’s profit prediction errors are evenly distributed around 0 with a mean of −0.01. PPO does not overpredict the profit due to the update clipping, stochastic policy, and online training. The update clipping both prevents the critic from overfitting to noise and the actor from exploiting the critic prediction errors. The stochastic policy helps to further prevent critic error exploitation, and the online training allows the PPO agent to explore actions with high predicted profit and update the critic with the observed reward.

Figure 6 shows the computational times of the algorithms to return the optimal action. ANN-PSO has the longest average computational time of 41 s. PPO and FP-NLP have relatively fast computational times of 0.002 s and 0.02 s, respectively. This highlights the fast computational time of PPO since the computation is one forward pass through the policy neural network. Both FP-NLP and ANN-PSO require iterations to find the optimal action, which results in longer computational times.

5.2. Sensitivity Analysis

The results of the sensitivity analysis are shown in Figure 7. For these pricing cases, the temperature set point selection is the main variable affecting price, while the flow of B can be held nearly constant. All three methods have similar profit contour plots, but ANN-PSO has a more jagged profile and overall darker contour indicating the agent had a lower profit compared to the other methods. This is caused by PSO exploiting the errors in the ANN, which is shown clearly in the ANN-PSO flow of B actions. While FP-NLP flow of B ranges from 1.63 to 1.68 m${}^3$/min, ANN-PSO ranges from 1.6 to 1.73 m${}^3$/min. Overall, the ANN-PSO contour plot appears random showing how PSO finds apparent optimal actions in areas the ANN overpredicts the reward. The erroneous flow of B may contribute to the suboptimal temperature set point selection as ANN-PSO typically selects lower temperatures compared to FP-NLP when $P_q$ is between 0.25 to 0.75 $/kJ. The temperature profile is also jagged, indicating the PSO is again exploiting the ANN model overprediction.

PPO on the other hand has smoother actions and the profiles appear to be very similar to FP-NLP actions. The main reason for the smooth action contour plots is because instead of optimizing a complicated ANN, PPO is mapping the state to actions with a continuous function. The main difference between FP-NLP and PPO actions is in the flow of B, but the difference between the actions is negligible. PPO is choosing actions based on the training with the stochastic environment, and FP-NLP is choosing actions based on a perfect deterministic environment. This causes PPO’s actions to be slightly suboptimal in the tested deterministic environment compared to FP-NLP.

6. Conclusions

In this study, we compared the performance and applicability of two RL algorithms—ANN-PSO and PPO—by exploring their methods and applying them on stochastic steady-state economic optimization of a CSTR with FP-NLP as a benchmark algorithm [4,5,6]. We evaluate the RL algorithms’ performance with their profitability and online computational times, and their applicability with their data requirements and training efficiencies. On the case study, PPO shows better performance as its average profits are higher compared to ANN-PSO and its online computational times are faster. ANN-PSO shows better applicability as the algorithm can train with normal operational data and converges to a policy with less training data. Both algorithms perform similarly to FP-NLP with ANN-PSO and PPO achieving 99.3% and 99.9% of FP-NLP’s average profit, respectively. PPO has the fastest online computational time, 10 and 10,000 times faster than FP-NLP and ANN-PSO, respectively. While these results are based on this case study, they may be useful guidelines for more complex applications.

Investigation of the ANN-PSO residual plots reveal PSO exploits overpredictions in the ANN, resulting in suboptimal actions. PPO does not have this issue as it has update clipping and a stochastic policy. The ANN exploitation is also seen in a similar algorithm named deep deterministic policy gradient. Some methods to combat this issue are to clip the artificial neural network and perturb actions to prevent ANN error exploitation [47].

As ANN-PSO has high training efficiency, researching methods to limit value overprediction to improve performance, such as training two ANNs and taking the minimum predicted profit of the two, would be beneficial. Other off-policy algorithms should also be investigated like twin-delayed deep deterministic policy gradients and V-trace to observe their performance on a chemical system after training on operational data. Other improvements to ANN-PSO include decreasing the computational time. Our previous work shows replacing PSO with an actor ANN can significantly reduce the online computational time and find an optimal policy [48]. As PPO and other on-policy algorithms requires interaction with an environment, they should be explored in their ability to augment FP-NLP. Another research topic is exploring how these algorithms perform with more complex systems or when the system changes such as multiple reactions, reversible reactions, or heat exchanger fouling.