Robust Statistic Estimation of Constrained Optimal Control Problems of Pollution Accumulation (Part I)

Abstract

In this paper, we study a constrained optimal control on pollution accumulation where the dynamic system was governed by a diffusion process that depends on unknown parameters, which need to be estimated. As the true values are unknown, we intended to determine (adaptive) policies that maximize a discounted reward criterion with constraints, that is, we used Lagrange multipliers to find optimal (adaptive) policies for the unconstrained version of the optimal control problem. In the present context, the dynamic system evolves as a diffusion process, and the cost function is to be minimized by another function (typically a constant), which plays the role of a constraint in the control model. We offer solutions to this problem using standard dynamic programming tools under the constrained discounted payoff criterion on an infinite horizon and the so-called principle of estimation and control. We used maximum likelihood estimators by means of a minimum least square error approximation in a pollution accumulation model to illustrate our results. One of the advantages of our approach compared to others is the intuition behind it: find optimal policies for an estimated version of the problem and let this estimation tend toward the real version of the problem. However, most risk analysts will not be as used to our methods as they are to, for instance, the model predictive control, MATLAB’s robust control toolbox, or the polynomial chaos expansion method, which have been used in the literature to address similar issues.

1. Introduction

This work studies the problem of the optimal control of pollution accumulation with an unknown parameter, which needs to be statistically estimated, in a constrained context. We aim to construct adaptive policies for the discounted reward criterion on an infinite horizon. We assumed that the stock of pollution is driven by an Itô’s diffusion and used the discounted criterion. We consider the presence of constraints on the reward function. We estimated the unknown parameter $\theta$ and followed the principle of estimation and control (PEC) to use standard dynamic programming tools to find optimal solutions.

We kept the presentation at the application level to the pollution accumulation problem while attempting to maximize a utility function. However, the theory is general enough to be exploited in other contexts. The goal of pollution accumulation models is to examine how some things are managed for society’s consumption. It is commonly acknowledged that this consumption produces two by-products: pollution and social welfare. The latter term refers to the distinction between the benefits and harms connected with pollution. The theory studied here enables the decision-maker to identify a consumption policy that maximizes anticipated social welfare for society, subject to a limitation that may reflect, for example, that some environmental clean-up expenditures will not surpass a certain number over time, while (for instance) the rate at which nature cleans itself is unknown. One of the features of the discounted optimality criterion used in this paper is that emphasis is placed on the utility of consumption for the present generations, which is mirrored by the value functions that we obtained. This characteristic renders the problem a rather flexible one and enables us to use standard dynamic programming tools.

We employ the PEC, which has roots in Kurano (1972) [1] and Mandl (1974) [2], to analyze the adaptive control problem with constraints. The goal of the PEC is to estimate parameter $\theta$, replace the unknown $\theta$ with its estimated value, and then solve the optimal control problem with constraints. Refer to [1,3,4], and the references therein for studies developing asymptotically optimal adaptive strategies. For instance, Kurano and Mandl introduce the idea of estimation and control when thinking about Markov decision models with constrained rewards and a finite-state space. Based on a consistent estimator for the unknown parameter that is uniformly optimal in the parameter, they demonstrate the existence of an optimal policy. Reference [4] works with discrete-time stochastic control systems.

In the late 1990s and early 2000s, the stochastic optimization issue with constraints was addressed, under the assumption that all the coefficients—the diffusion itself, the reward function, and the restrictions—are bounded (see, for example, [5,6,7,8]). Related publications include Borkar and Ghosh’s foundational publication on constrained optimum control under discounted and ergodic criteria, the work of Mendoza-Pérez, Jasso-Fuentes, Prieto-Rumeau, and Hernandez-Lerma (see [9,10]), and the paper of Jasso-Fuentes, Escobedo-Trujillo, and Mendoza-Pérez [11].

The adaptive control of a linear diffusion process (of the sort we use here) with regard to the discounted cost criterion is studied in [12]. In [13], an adaptive optimal control for continuous-time linear systems is investigated. The use of statistical tools is common when modeling control problems. For instance, Bayesian adaptive techniques for ergodic diffusions are considered in [14], and ref. [15] uses the method of maximum likelihood estimation of parameters to study a self-tuning scheme for a diffusion process with the long-term average cost criterion. The main idea is to estimate the unknown parameter online so that the most recent estimate is used to determine the true parameter when choosing each control (see [16]). The estimation of parameters for diffusion processes using discrete observations was studied in several papers, including [17,18,19,20,21] and the references therein.

The issue of pollution accumulation has previously been studied from the perspective of dynamic optimization; for instance, refs. [22,23] use a linear-quadratic model to explain this phenomenon, ref. [24] deals with the average payoff in a deterministic framework, refs. [25,26] extends the former’s approach to a stochastic context, ref. [27] uses a robust stochastic differential game to characterize the situation, ref. [28] studies the problem from the perspective of constrained stochastic optimization, and ref. [29] is a statistical survey of the effects of air pollution on public health. The main contribution of our paper is as follows: a statistical estimation procedure is used to find $\theta$; we construct adaptive policies that are almost certainly optimal for the constrained optimization problem under the discounted payoff on an infinite horizon. These adaptive policies were obtained by substituting the estimates into optimal stationary controls (or PEC); see [1,2]. In this sense, our findings resemble those presented in [30], for most risk analysts will not be as used to our methods as they are to, for instance, the model predictive control, the robust control toolbox or the polynomial chaos expansion method, which have been used in the literature to address similar issues.

Our work lies at the intersection of three important classes of optimal control problem. The first is the problem of controlling pollution accumulation, as presented in [22,29,30,31,32]; the second class deals with constrained optimal control problems (references [11,33,34] study this type of problem, considering that all parameters are known); the third class concerns the adaptive optimal control problems, as presented in [14,15,20,35,36]. Reference [30] belongs to the first and third classes. Our work is an extension of [11,22,28,31,32,36] to the adaptive constrained optimal control framework. Reference [22] builds a robust control to explore the normative maxim that, in the presence of uncertainty, society must guard against worst-case outcomes; [11] studies a constrained optimal control problem where all the parameters are known, while [28,32] study the same in the context of pollution accumulation; Refs. [31,36] study an unconstrained adaptive optimal control problem. In addition, a numerical example is given for demonstration purposes.

The rest of the paper is organized as follows. We present the theoretical preliminaries in the next Section. Then, we devote Section 3 to our main results and we illustrate them in Section 4. We provide our conclusions in Section 5. Please note that, for the sake of self-completeness of the presentation and acknowledgement of our sources, we have included references at the end of this article. However, we recognize that these might diverge the reader. We apologize for this inconvenience.

Throughout our work, we use the following notations. All $x=(x_1,x_2,\dots ,x_n)$ in ${\mathbb{R}}^n$ and matrices $A=\left(A_{k,p}\right)$ we denote by $|·|$ the usual Euclidean norm ${\left|x\right|}^2:={\sum }_k{x}_k^2$, and we let ${\left|A\right|}^2:=\mathrm{Tr}\left(A{A}^{\top }\right)={\sum }_{k,p}{A}_{k,p}^2$, where $A^{\top }$ and $\mathrm{Tr}(·)$, denote the transpose and the trace of a square matrix, respectively. Sometimes, we use the notation ${\partial }_i:=\frac{\partial }{\partial x_i}$, and ${\partial }_{ij}^2:=\frac{{\partial }^2}{\partial x_i\partial x_j}$.

2. Problem Statement

In our model, the stock of pollution is modeled as an n-dimensional controlled stochastic differential equation (SDE) of the form:

$$\mathrm{d}x\left(t\right)=b(x\left(t\right),u\left(t\right),\theta )\mathrm{d}t+\sigma \left(x\left(t\right)\right)\mathrm{d}W\left(t\right),x\left(0\right)=x_0,t\ge 0,$$

(1)

where $b:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to {\mathbb{R}}^n$ and $\sigma :{\mathbb{R}}^n\to {\mathbb{R}}^{n\times d}$ are given functions, and $W(·)$ is an ${\mathcal{F}}_t$-adapted d-dimensional Wiener process, such that $W\left(t\right)-W\left(s\right)$ and ${\mathcal{F}}_s$ are pairwise independent. The compact set U is assumed to be contained in a suitable metric space and is called the control set. Here, $u\left(t\right)$ represents the flow of consumption at time t; this is a stochastic process that takes values from U, which, in turn, is bounded to represent the consumption restrictions imposed by worldwide protocols. In this work, we assume that the pollution decay rate $\theta$ is an unknown parameter taking values from a compact set $\mathsf{\Theta }\subset {\mathbb{R}}^m$ called the parameter set. Assumption A1 in Appendix A ensures the existence and uniqueness of a strong solution to (1).

2.1. Control Policies and Stability Assumptions

Although we manage to illustrate our theoretical developments using the so-called stationary Markovian policies, we need to introduce the concept of randomized policies (also known as relaxed controls). To this end, we used the following nomenclature:

• $\mathcal{B}\left(B\right)$ is the Borel $\sigma$-algebra spawned by the Borel set B.
• $\mathcal{C}\left(\mathcal{O}\right)$ is, as usual, the space of all real-valued continuous functions on a bounded, open and connected subset $\mathcal{O}\subset {\mathbb{R}}^n$.
• ${\mathcal{C}}_b\left(\mathcal{O}\right)$ stands for the space of all real-valued continuous bounded functions f on the bounded, open and connected subset $\mathcal{O}\subset {\mathbb{R}}^n$.
• ${\mathcal{C}}^{\kappa }\left(\mathcal{O}\right)$ is the space of all real-valued continuous functions f on the bounded, open and connected subset $\mathcal{O}\subset {\mathbb{R}}^n$, with continuous derivatives up to order $\kappa \in \mathbb{N}$.
• ${\mathcal{L}}^p\left(\mathsf{\Omega }\right)$ is, as is customary, the Lebesgue space of functions g on $\mathsf{\Omega }$ such that ${\int }_{\mathsf{\Omega }}{\left|g\right|}^p\mathrm{d}\mu <\infty$, with $(\mathsf{\Omega },\mathcal{F},\mu )$ a suitable measure space, and $p\ge 1$.
• $\mathcal{P}\left(B\right)$ is the family of probability measures on B endowed with the topology of weak convergence.

Definition 1.

A randomized policy is a family $\pi :=\left({\pi }_t:t\ge 0\right)$ of stochastic kernels on $\mathcal{B}\left(U\right)\times {\mathbb{R}}^n$, satisfying:

(a) 

for each $t\ge 0$ and $x\in {\mathbb{R}}^n$, ${\pi }_t(·|x)\in \mathcal{P}\left(U\right)$, and for each $t\ge 0$ and $D\in \mathcal{B}\left(U\right)$, ${\pi }_t\left(D\right|·)$ is a Borel function on ${\mathbb{R}}^n$;

(b) 

for each $D\in \mathcal{B}\left(U\right)$ and $x\in {\mathbb{R}}^n$, the function ${\pi }_t\left(D\right|x)$ is Borel-measurable in $t\ge 0$.

A randomized policy is said to be stationary if there is a probability measure $\pi (·|x)\in \mathcal{P}(U)$, such that ${\pi }_t(·|x)=\pi (·|x)$ for all $t\ge 0$ and $x\in {\mathbb{R}}^n$. The set of randomized stationary policies is denoted by Π.

Let $\mathbb{F}$ be the family of measurable functions $f:{\mathbb{R}}^n\to U$. A strategy $u\left(t\right):=f\left(x\right(t\left)\right)$ for some $f\in \mathbb{F}$ is said to be a stationary Markov policy.

For each randomized stationary policy $\pi \in \mathsf{\Pi }$, we write the drift coefficient b defined in (1) as

$$b(x,\pi ,\theta ):={\int }_{U}b(x,u,\theta )\pi \left(\mathrm{d}u\right|x).$$

(2)

Note that $b(x,\pi ,\theta )$ inherits the same continuity and Lipschitz properties from b, given in Assumption A1.

Remark 1.

Under Assumption A1, for each policy $\pi \in \mathsf{\Pi }$ and $\theta \in \mathsf{\Theta }$, there exists a weak solution $x^{\pi ,\theta }(·)$ of (1) which is a Markov–Feller process in the probability space $\left(\mathsf{\Omega },\mathcal{F},{\mathbb{P}}^{\pi ,\theta }\right)$. See Theorem 2.2.6 in [37].

Topology of relaxed controls. We will need the limit and continuity concepts. For this reason, we topologized the set of randomized stationary policies $\mathsf{\Pi }$, as in [38]. This topology renders $\mathsf{\Pi }$ a compact metric space, and is determined by the following convergence criterion (see [37,38,39]).

Definition 2.

A sequence $\left({\pi }_m:m=1,2,\dots \right)$ in Π converges to $\pi \in \mathsf{\Pi }$, if

$${\int }_{{\mathbb{R}}^n}g\left(x\right)h(x,{\pi }_m)\mathrm{d}x\to {\int }_{{\mathbb{R}}^n}g\left(x\right)h(x,\pi )\mathrm{d}x.$$

for all $g\in {\mathcal{L}}^1\left({\mathbb{R}}^n\right)$, and $h\in {\mathcal{C}}_b({\mathbb{R}}^n\times U)$, where

$$h(x,{\pi }_m):={\int }_{U}h(x,u){\pi }_m\left(\mathrm{d}y\right|x)and h(x,\pi ):={\int }_{U}h(x,u)\pi \left(\mathrm{d}y\right|x).$$

We denote this type of convergence as ${\pi }_m\stackrel{W}{\to }\pi$.

For $u\in U$, $\theta \in \mathsf{\Theta }$ and $\nu (·,\theta )\in {\mathcal{C}}^2\left({\mathbb{R}}^n\right),$ let

$$\begin{array}{ccc}\hfill {\mathbb{L}}^{u,\theta }\nu (x,\theta )& :=& \sum _{i=1}^n{b}^i(x,u,\theta ){\partial }_i\nu (x,\theta )+\frac{1}{2}\sum _{i,j=1}^n{a}^{ij}\left(x\right){\partial }_{ij}^2\nu (x,\theta ),\hfill \end{array}$$

(3)

where $b^i$ is the i-th component of b, and $a^{ij}$ is the $(i,j)$-component of the matrix $a(·)$, defined in Assumption A1(d). Again, for each randomized stationary policy $\pi \in \mathsf{\Pi }$, we write the infinitesimal generator ${\mathbb{L}}^{u,\theta }$ defined in (3) as

$${\mathbb{L}}^{\pi ,\theta }\nu (x,\theta ):={\int }_U{\mathbb{L}}^{u,\theta }\nu (x,\theta )\pi \left(\mathrm{d}u\right|x).$$

Note that the application of Dynkin’s formula to the function $v(t,x):={\mathrm{e}}^{ct}w\left(x\right)$, and Assumption A2(b) yields

$${\mathbb{E}}_x^{\pi ,\theta }\left[w\left(x\left(t\right)\right)\right]\le {\mathrm{e}}^{-ct}w\left(x\right)+\frac{d}{c}\left(1-{\mathrm{e}}^{-ct}\right),\mathrm{for}\mathrm{all}\pi \in \mathsf{\Pi },\theta \in \mathsf{\Theta },x\in {\mathbb{R}}^n\mathrm{and}t\ge 0,$$

(4)

where ${\mathbb{E}}_x^{\pi ,\theta }[·]$ stands for the conditional expectation of ·, given that (1) starts at x, the controller uses the randomized stationary policy $\pi$, and the unknown parameter is fixed at $\theta$. That is, ${\mathbb{E}}_x^{\pi ,\theta }[·]$ is the expectation of · taken with respect to the probability measure ${\mathbb{P}}^{\pi ,\theta }$ when $x\left(t\right)$ starts at x.

2.2. Reward, Cost and Constraint Rates

We will consider that the reward and the cost rates, along with the constraints of our model, can be unbounded from above and below, but are dominated by the Lyapunov function w given in Assumption A2. Namely, they are in the Banach space of real-valued measurable functions on ${\mathbb{R}}^n$ with the finite w-norm, which is defined as follows.

Definition 3.

Let ${\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ denote the Banach space of real-valued measurable functions v on ${\mathbb{R}}^n$ with finite w-norm, which is defined as

$${∥v∥}_w:=\underset{x\in {\mathbb{R}}^n}{\sup }\frac{\left|v\right(x\left)\right|}{w\left(x\right)}<\infty .$$

Let $r,c:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ be measurable functions that will be identified as the social welfare (also called payoff or reward) rate and the cost rate, respectively, and let $\eta :{\mathbb{R}}^n\times \mathsf{\Theta }\to \mathbb{R}$ be another measurable function that will be referred to as the constraint rate. In the present context, this restriction stands for the fact that, in some situations, due to each country’s legal framework, the cost of cleaning the environment must not exceed a given quantity. Both functions are supposed to meet Assumption A3.

When the controller uses policy $\pi \in \mathsf{\Pi }$, we write the reward and cost rates in a similar way as (2); that is,

$$c(x,\pi ,\theta )={\int }_{U}c(x,u,\theta )\pi \left(\mathrm{d}u\right|x),r(x,\pi ,\theta ):={\int }_{U}r(x,u,\theta )\pi \left(\mathrm{d}u\right|x).$$

(5)

3. Main Results

3.1. Discounted Control with Constraints

In the sequence, we work in the space ${\mathcal{W}}^{2,p}\left({\mathbb{R}}^n\right)\cap {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$, where ${\mathcal{W}}^{\ell ,p}\left(\mathcal{O}\right)$ stands for the Sobolev space of real-valued measurable functions on the open and connected subset $\mathcal{O}\in {\mathbb{R}}^n$ whose generalized derivatives up to order $\ell \ge 0$ are in ${\mathcal{L}}^p\left(\mathcal{O}\right)$ for $p\ge 1$.

Definition 4.

Given the initial state x, a parameter value $\theta \in \mathsf{\Theta }$ and a discount rate $\alpha >0$, we define the total expected α-discounted reward and cost when the controller uses a policy π in Π, as

$$\begin{array}{ccc}\hfill V(x,\pi ,r,\theta )& :=& {\mathbb{E}}_x^{\pi ,\theta }\left[{\int }_0^{\infty }{\mathrm{e}}^{-\alpha t}r\left(x\left(t\right),\pi ,\theta \right)\mathrm{d}t\right]and\hfill \\ \hfill V(x,\pi ,c,\theta )& :=& {\mathbb{E}}_x^{\pi ,\theta }\left[{\int }_0^{\infty }{\mathrm{e}}^{-\alpha t}c\left(x\left(t\right),\pi ,\theta \right)\mathrm{d}t\right].\hfill \end{array}$$

Propositions 1–3 below state the properties of the functional $v\mapsto V(x,\pi ,v,\theta )$ for all $(x,\pi ,\theta )\in {\mathbb{R}}^n\times \mathsf{\Pi }\times \mathsf{\Theta }$. Under Assumption A3 and inequality (4), a direct computation yields the following result. This estabilishes that the expected $\alpha$-discounted reward are dominated by a Lyapunov function.

Proposition 1.

If Assumptions A1–A3 hold, the functions $V\left(·,\pi ,r,\theta \right)$ and $V\left(·,\pi ,c,\theta \right)$ belong to the space ${\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ for each π in Π; in fact, for each x in ${\mathbb{R}}^n$ and $\theta \in \mathsf{\Theta }$, we have

$$\underset{\pi \in \mathsf{\Pi }}{\sup }\left|V\left(x,\pi ,r,\theta \right)\right|+\underset{\pi \in \mathsf{\Pi }}{\sup }\left|V\left(x,\pi ,c,\theta \right)\right|\le 2M\left(\alpha \right)w\left(x\right)with M\left(\alpha \right):=M\frac{\alpha +d}{\alpha c}.$$

Here, c and d are as in Assumption A2, and M is the constant in Assumption A3(b).

The following result is an extension of Proposition 3.1.5 in [40] to the topology of the relaxed controls. This shows that both the expected discounted reward and the expected cost are solutions to the linear version of the dynamic programming partial differential equation. This can be regarded as a necessary condition for the optimality of the value function. Its proof mimics the steps of the original, replacing the control sets with those used here while keeping $\theta \in \mathsf{\Theta }$ fixed.

Proposition 2.

Let Assumptions A1–A3 hold, and let $v:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ be a measurable function satisfying Assumption A3. Then, for every $\pi \in \mathsf{\Pi }$, the associated expected α-discounted function $V(·,\pi ,v,\theta )$ is in ${\mathcal{W}}^{2,p}\left({\mathbb{R}}^n\right)\cap {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$, and is such that

$$\alpha V(x,\pi ,v,\theta )=V(x,\pi ,v,\theta )+{\mathbb{L}}^{\pi ,\theta }V(x,\pi ,v,\theta )for all x\in {\mathbb{R}}^n and \theta \in \mathsf{\Theta }.$$

(6)

Conversely, if some function $\phi (·,\theta )\in {\mathcal{W}}^{2,p}\left({\mathbb{R}}^n\right)\cap {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ verifies (6), then

$$\phi (x,\theta )=V(x,\pi ,v,\theta )for all x\in {\mathbb{R}}^n and \theta \in \mathsf{\Theta }.$$

(7)

Moreover, if the equality in (6) is replaced by “≤” or “≥”, then (7) holds, with the respective inequality.

Definition 5.

Let $\alpha >0$. The total expected α-discounted constraint when the controller uses a policy $\pi \in \mathsf{\Pi }$, given the initial state $x\in {\mathbb{R}}^n$ and $\theta \in \mathsf{\Theta }$, is defined by

$$\overline{\eta }(x,\pi ,\theta ):=\alpha {\mathbb{E}}_x^{\pi ,\theta }\left[{\int }_0^{\infty }{\mathrm{e}}^{-\alpha t}\eta \left(x\left(t\right),\theta \right)\mathrm{d}t\right].$$

Remark 2.

The function $\overline{\eta }(·,\pi ,\theta )$ belongs to the space ${\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ for each $\pi \in \mathsf{\Pi }$. Moreover, for each $x\in {\mathbb{R}}^n$, we have

$$\underset{\pi \in \mathsf{\Pi }}{\sup }\left|\overline{\eta }(x,\pi ,\theta )\right|\le {\parallel \eta \parallel }_w\frac{\alpha +d}{c}w\left(x\right).$$

For each $\alpha >0$, $\theta \in \mathsf{\Theta }$ and $x\in {\mathbb{R}}^n$, assume we are given a constraint function $\eta (·,\theta )$ satisfying Assumption A3(c). In this way, we define the set

$${\mathcal{F}}_{\theta }^x:=\left\{\pi \in \mathsf{\Pi }|V(x,\pi ,c,\theta )\le \overline{\eta }(x,\pi ,\theta )\right\}.$$

We assume that ${\mathcal{F}}_{\theta }^x$ is nonempty.

Definition 6 (The discounted problem with constraints (DPC)).

We say that a policy ${\pi }^{*}\in \mathsf{\Pi }$ is optimal for the DPC with initial state $x\in {\mathbb{R}}^n$, given that $\theta \in \mathsf{\Theta }$ is the true parameter value if ${\pi }^{*}\in {\mathcal{F}}_{\theta }^x$ and, in addition,

$$V^{*}\left(x,{\pi }^{*},r,\theta \right)=\underset{\pi \in {\mathcal{F}}_{\theta }^x}{\sup }V\left(x,\pi ,r,\theta \right).$$

In this case, $V^{*}(x,r,\theta ):=V(x,{\pi }^{*},r,\theta )$ is called the $\alpha$-discount optimal reward for the DPC.

3.2. Lagrange Multipliers Approach

We mimic the technique we used in [32] to transform the original DPC into an unconstrained problem. To this end, take $\lambda \le 0$ and consider the new reward rate

$$r^{\lambda }(x,u,\theta ):=r(x,u,\theta )+\lambda (c(x,u,\theta )-\alpha \eta (x,\theta )).$$

(8)

Using the same notation of (5), we can write (8) as

$$r^{\lambda }(x,\pi ,\theta ):=r(x,\pi ,\theta )+\lambda (c(x,\pi ,\theta )-\alpha \eta (x,\theta )),\pi \in \mathsf{\Pi },\theta \in \mathsf{\Theta }.$$

Observe that, for each $\alpha >0$ and $\lambda <0$, $r^{\lambda }(·,\pi ,\theta )$ is in ${\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ uniformly in $\pi \in \mathsf{\Pi }$ and $\theta \in \mathsf{\Theta }$. In fact,

$$\begin{array}{ccc}\hfill |r^{\lambda }(x,\pi ,\theta )|& \le & \left|r\right(x,\pi ,\theta \left)\right|+\left|\lambda \right|\left|c\right(x,\pi ,\theta \left)\right|+\left|\lambda \right|\left|\alpha \eta \right(x,\theta \left)\right|\hfill \\ \hfill & \le & Mw\left(x\right)+M\left|\lambda \right|w\left(x\right)+\left|\lambda \right|\left|\alpha \eta \right(x,\theta \left)\right|\hfill \\ \hfill & \le & {(M+M|\lambda |+|\lambda |\alpha \parallel \eta \parallel }_w)w\left(x\right)=N^{\lambda }w\left(x\right),\hfill \end{array}$$

(9)

where $N^{\lambda }:=M+M\left|\lambda \right|+{\left|\lambda \right|\alpha \parallel \eta \parallel }_w$, and M, as in Assumption A3(b).

For all $x\in {\mathbb{R}}^n$ and $\theta \in \mathsf{\Theta }$, define

$$V(x,\pi ,r^{\lambda },\theta ):={\mathbb{E}}_x^{\pi ,\theta }\left[{\int }_0^{\infty }{\mathrm{e}}^{-\alpha t}{r}^{\lambda }\left(x\left(t\right),\pi ,\theta \right)\mathrm{d}t\right].$$

The discounted unconstrained problem is defined as follows.

Definition 7 ($\lambda$-Discounted unconstrained problem ($\lambda$-DUP)).

A policy ${\pi }^{*}\in \mathsf{\Pi }$ for which

$$V(x,{\pi }^{*},r^{\lambda },\theta )=\underset{\pi \in \mathsf{\Pi }}{\sup }V(x,\pi ,r^{\lambda },\theta )=:V^{*}(x,r^{\lambda },\theta )for all x\in {\mathbb{R}}^n,$$

(10)

is called discounted optimal for the λ-DUP, and $V^{*}(·,r^{\lambda },\theta )$ is referred to as the optimal discounted reward for the λ-DUP.

Let $v:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ be a measurable function satisfying similar conditions as those given in Assumption A3. The following is called a verification result in the literature. It shows that $V^{*}(·,v,\theta )$ is the unique solution of the Hamilton–Jacobi–Bellman (HJB) Equation (11), and also proves the existence of stationary policies $f_{\theta }^{*}\in \mathbb{F}$. Observe that, by virtue of Definition 7, the functional to which it refers can be the optimal discounted reward for the $\lambda$-DUP. Its proof can be found in [11,15,41], considering $\theta$ as fixed.

Proposition 3.

Suppose that Assumptions A1–A3 hold. Then:

(i) 

The α-optimal discount reward $V^{*}(·,v,\theta )$ belongs to ${\mathcal{W}}^{2,p}\left({\mathbb{R}}^n\right)\cap {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ and verifies the discounted reward HJB equation; that is, for all $x\in {\mathbb{R}}^n$ and $\theta \in \mathsf{\Theta },$

$$\begin{array}{ccc}\hfill \alpha V^{*}(x,v,\theta )& =& \underset{u\in U}{\sup }\{r(x,u,\theta )+{\mathbb{L}}^{u,\theta }{V}^{*}(x,v,\theta )\}.\hfill \end{array}$$

(11)

Conversely, if a function ${\phi }_{\theta }\in {\mathcal{W}}^{2,p}\left({\mathbb{R}}^n\right)\cap {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$ verifies (11), then ${\phi }_{\theta }\left(x\right)=V^{*}(x,v,\theta )$ for all $x\in {\mathbb{R}}^n$.

(ii) 

There exists a stationary policy $f_{\theta }^{*}\in \mathbb{F}$ that maximizes the right-hand side of (11); that is,

$$\alpha V^{*}(x,v,\theta )=r(x,f_{\theta }^{*},\theta )+{\mathbb{L}}^{f_{\theta }^{*},\theta }{V}^{*}(x,v,\theta )for allx\in {\mathbb{R}}^n,$$

and $f_{\theta }^{*}$ is α-discount optimal.

Remark 3.

(a) 

Notice that $V(x,\pi ,r^{\lambda },\theta )=V(x,\pi ,r,\theta )+\lambda \left[V(x,\pi ,c,\theta )-\overline{\eta }(x,\pi ,\theta )\right].$

(b) 

By Definitions 4 and 5,

$$\begin{array}{ccc}\hfill V(x,\pi ,c,\theta )-\overline{\eta }(x,\pi ,\theta )& =& {\mathbb{E}}_x^{\pi ,\theta }\left[{\int }_0^{\infty }{\mathrm{e}}^{-\alpha t}[c(x\left(t\right),\pi ,\theta )-\alpha \eta (x\left(t\right),\theta )]\mathrm{d}t\right]\hfill \\ & =:& V(x,\pi ,c-\alpha \eta ,\theta ),\hfill \end{array}$$

(c) 

Given that the cost and constraint rates $c,\eta$ satisfy Assumption A3, we deduce

$$\begin{array}{ccc}\hfill \left|c\right(x,\pi ,\theta )-\alpha \eta (x,\theta \left)\right|& \le & \left|c\right(x,\pi ,\theta \left)\right|+\left|\alpha \eta \right(x,\theta \left)\right|\hfill \\ \hfill & \le & Mw\left(x\right)+\left|\alpha \eta \right(x,\theta \left)\right|\hfill \\ \hfill & \le & (M+\alpha \parallel \eta {\parallel }_w)w\left(x\right),\hfill \end{array}$$

Thus, $c(·,\pi ,\theta )-\alpha \eta (·,\theta )\in {\mathcal{B}}_w\left({\mathbb{R}}^n\right).$

(d) 

The function $c-\alpha \eta :{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ is locally Lipschitz on ${\mathbb{R}}^n.$ In fact,

$$\begin{array}{ccc}& & \left|c\right(x,u,\theta )-\alpha \eta (x,\theta )-c(y,u,\theta )+\alpha \eta (y,\theta \left)\right|\hfill \\ & \le & |c(x,u,\theta )-c(y,u,\theta )|+|\eta (x,\theta )-{\eta }_{(}y,\theta \left)\right|\hfill \\ & \le & [K\left(R\right)+\tilde{K}\left(R\right)]|x-y|.\hfill \end{array}$$

(12)

for each $R>0$, and for all $\left|x\right|,\left|y\right|\le R$. The last inequality in (12) is met since we assume that Assumption A3 holds.

(e) 

Parts  (c)   and  (d)   imply that the function $c-\alpha \eta :{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ satisfies Assumption A3. Thus, the rate $r^{\lambda }(·,\pi ,\theta )$ is Lipschitz-continuous and $r^{\lambda }(·,\pi ,\theta )\in {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$. Furthermore, by virtue of (9) and Proposition 1,

$$\underset{\pi \in \mathsf{\Pi }}{\sup }\left|V(x,\pi ,r^{\lambda },\theta )\right|\le M_{\alpha }^{\lambda }w\left(x\right),with M_{\alpha }^{\lambda }:=N^{\lambda }\frac{\alpha +d}{\alpha c},$$

implying that $V(·,\pi ,r^{\lambda },\theta )\in {\mathcal{B}}_w\left({\mathbb{R}}^n\right)$.

3.3. Convergence of Value Functions $V(x,\pi ,r,\theta )$ and $V(x,\pi ,r^{\lambda },\theta )$

Definition 8.

A sequence $\left({\theta }_m:m=1,\dots \right)$ of measurable functions ${\theta }_m:\mathsf{\Omega }\to \mathsf{\Theta }$ is said to be a sequence of uniformly strongly consistent (USC) estimators of $\theta \in \mathsf{\Theta }$ if, as $m\to \infty$,

$${\theta }_m\left(\omega \right)\to \theta {\mathbb{P}}^{\pi ,\theta }\text{-}a.s.for all \pi \in \mathsf{\Pi },$$

where ${\mathbb{P}}^{\pi ,\theta }$ is the probability measure referred to by Remark 1.

For ease of notation, we write ${\theta }_m:={\theta }_m\left(\omega \right)\in \mathsf{\Theta }$. Let $v:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ be a measurable function satisfying similar conditions to those given in Assumption A3.

Remark 4.

(a) 

If Assumptions A1–A3 hold, then by Proposition 3.4 in [11], the mappings $\pi \mapsto V(x,\pi ,v,\theta )$, $\pi \mapsto V(x,\pi ,c-\alpha \eta ,\theta )$ and $\pi \mapsto V(x,\pi ,r^{\lambda },\theta )$ are continuous on Π for each $x\in {\mathbb{R}}^n$ and $\theta \in \mathsf{\Theta }.$

(b) 

Let $\left({\theta }_m:m=1,\dots \right)$ be a sequence of USC estimators of $\theta \in \mathsf{\Theta }$. Then, using Theorem 4.5 in [36], for every measurable function $v:{\mathbb{R}}^n\times \mathsf{\Pi }\times \mathsf{\Theta }\to \mathbb{R}$ that satisfies the Assumptions A1–A3, the sequence $\left(V\left(x,\pi ,v,{\theta }_m\right):m=1,\dots \right)$ converges to $V\left(x,\pi ,v,\theta \right)$${\mathbb{P}}^{\pi ,\theta }$-a.s., for each $x\in {\mathbb{R}}^n$ and $\pi \in \mathsf{\Pi }$.

(c) 

Let $\left({\pi }_m:m=1,\dots \right)$ be a sequence in Π. Since Π is a compact set, there exists a subsequence $\left({\pi }_{m_k}:k=1,\dots \right)\subset \left({\pi }_m:m=1,\dots \right)$ such that ${\pi }_{m_k}\stackrel{W}{\to }\pi \in \mathsf{\Pi }$, thus, combining parts  (a)    and   (b), and using the following triangular inequality:

$$\begin{array}{ccc}& & |V(x,{\pi }_{m_k},v,{\theta }_{m_k})-V(x,\pi ,v,\theta )|\hfill \\ & \le & |V(x,{\pi }_{m_k},v,{\theta }_{m_k})-V(x,\pi ,v,{\theta }_{m_k})|+|V(x,\pi ,v,{\theta }_{m_k})-V(x,\pi ,v,\theta )|,\hfill \end{array}$$

we deduce that, for every measurable function $v:{\mathbb{R}}^n\times U\times \mathsf{\Theta }\to \mathbb{R}$ satisfying Assumption A3, we have that

$$\begin{array}{c}\hfill V(x,{\pi }_{m_k},v,{\theta }_{m_k})\to V(x,\pi ,v,\theta ){\mathbb{P}}^{\pi ,\theta }-a.s.ask\to \infty .\end{array}$$

(d) 

The optimal discount reward for the $\lambda$-DUP, $V^{*}(·,r^{\lambda },\theta )$ satisfies Proposition 3. In addition, Proposition 3(ii) ensures the existence of stationary policy $f_{\theta }^{\lambda }\in \mathbb{F}$.

(e) 

For each $\lambda \le 0$, $\theta \in \mathsf{\Theta }$ and $\alpha >0$, we denote

$${\mathsf{\Pi }}^{\lambda ,\theta }:=\left\{\pi \in \mathsf{\Pi }|\alpha V^{*}(x,r^{\lambda },\theta )=r^{\lambda }(x,\pi ,\theta )+{\mathbb{L}}^{\pi ,\theta }{V}^{*}(x,r^{\lambda },\theta )for all x\in {\mathbb{R}}^n\right\}.$$

(13)

Since $\mathbb{F}$ can be seen as an embedding of Π, Proposition 3(ii) ensures that the set ${\mathsf{\Pi }}^{\lambda ,\theta }$ is nonempty.

(f) 

Under the hypotheses of Proposition 3, Lemma 3.15 in [11] ensures that for each $\theta \in \mathsf{\Theta }$ fixed and any sequence $\left({\lambda }_m:m=1,\dots \right)$ in $(-\infty ,0]$ converging to some $\lambda \le 0$; if there exists a sequence $\left({\pi }^{{\lambda }_m,\theta }:m=1,\dots \right)\in {\mathsf{\Pi }}^{{\lambda }_m,\theta }$ for each $m\ge 1$, such that it converges to a policy $\pi \in \mathsf{\Pi }$, then $\pi \in {\mathsf{\Pi }}^{\lambda ,\theta }$. That is, π satisfies

$$\alpha V^{*}(x,r^{\lambda },\theta )=r^{\lambda }(x,\pi ,\theta )+{\mathbb{L}}^{\theta ,\pi }{V}^{*}(x,r^{\lambda },\theta )for all x\in {\mathbb{R}}^n.$$

(g) 

Lemma 3.16 in [11] ensures that the mapping $\lambda ⟼V_{\alpha }^{*}(x,r^{\lambda },\theta )$ is differentiable on $(-\infty ,0)$, for any $x\in {\mathbb{R}}^n$ and $\alpha >0$; in fact, for each $\lambda <0$ and $\theta \in \mathsf{\Theta }$

$$\frac{\partial V^{*}(x,r^{\lambda },\theta )}{\partial \lambda }=V(x,{\pi }^{\lambda },c,\theta )-\overline{\eta }(x,{\pi }^{\lambda },\theta ).$$

(14)

3.4. Estimation Methods for Our Application

Pedersen [42] describes the approximate maximum likelihood estimator in the following manner. The unknown parameter $\theta$ is estimated by means of some function $h_m:\mathsf{\Omega }\times \mathsf{\Theta }\to \mathbb{R},\mathrm{for} m=1,\dots$ that measures the likelihood of different values of $\theta$. If for each $\omega \in \mathsf{\Omega }$ fixed, the function $h_m(\omega ,\theta )$ has a unique maximum point ${\theta }_m\left(\omega \right)\in \mathsf{\Theta }$, then $\theta$ is estimated by ${\theta }_m\left(\omega \right)$.

Under the assumption that, for $m\in \mathbb{N}$ and $\theta \in \mathsf{\Theta }$, $h_m(·,\theta )$ is a measurable function of $\omega$ and is also continuously differentiable in $\theta$ for all ${\mathbb{P}}^{\pi ,\theta }$ and almost all $\omega \in \mathsf{\Omega }$, it is proven that the function $\theta \to h_m(\omega ,\theta )$ is continuous and has a unique maximum point ${\theta }_m\left(\omega \right)$ for each $\omega \in \mathsf{\Omega }$ fixed. The number $m\in \mathbb{N}$ is the index of a sequence of random experiments on the measurable space $(\mathsf{\Omega },\mathcal{F})$.

In our application, the outcomes of the random experiments will be represented by a sequence $X_T:=\left(x_{t_i}:0\le i\le m\right)$ of a trajectory $\{x^{u,\theta }\left(t\right):t\in [0,T]\}$ at times $0=t_0<t_1<t_m:=T$ on $(\mathsf{\Omega },\mathcal{F}):=\left(\mathcal{C}\right([0,\infty )),\mathcal{B}(\mathcal{C}\left(\right[0,\infty \left)\right))$ and the function $h_m$ will be called the least square function (LSE), i.e., $h_m(w,\theta ):=LSE(w,\theta ).$

In practice, $x^{u,\theta }\left(t\right)$ in (1) can only be observed in a finite horizon; for example, T. Actually, this is one of the hypotheses of the so-called model predictive control. However, at least from a theoretical point of view, our version of the PEC makes no such assumption, but still chooses T to be as large as practically possible (with regard to computer power, measurement instruments, computation time, etc.) so that we can define LSE as:

$$LSE(X_T,\theta ):=\sum _{i=1}^m{\left(x_{t_i}-x_{t_{i-1}}-b(x_{t_{i-1}},u_{t_{i-1}},\theta )(t_i-t_{i-1})\right)}^2.$$

(15)

with b as in (1). The LSE function generates the least square estimator, ${\theta }_{LSE}$,

$${\theta }_{LSE}\equiv {\theta }_{LSE}\left(X_T\right):=\arg \underset{\theta \in \mathsf{\Theta }}{\min }LSE(X_T,\theta ).$$

(16)

Consistency and asymptotic normality of ${\theta }_{LSE}$ are studied in [18,19,42]. Shoji [18] demonstrates that the optimization based on the LSE function is identical to the optimization based on the discrete approximate likelihood ratio function when a one-dimensional stochastic differential equation with a constant diffusion coefficient is taken into account:

$$\begin{array}{cc}\hfill & MLR(X_T,\theta ):=\sum _{i=1}^{m}b(x_{t_{i-1}},u_{t_{i-1}},\theta ){\left[\sigma \left(x_{t_{i-1}}\right)\sigma {\left(x_{t_{i-1}}\right)}^{\top }\right]}^{-1}(x_{t_i}-x_{t_{i-1}})\hfill \\ \hfill & -\frac{1}{2}\sum _{i=1}^m\left\{b{(x_{t_{i-1}},u_{t_{i-1}},\theta )}^{\top }{\left[\sigma \left(x_{t_{i-1}}\right)\sigma {\left(x_{t_{i-1}}\right)}^T\right]}^{-1}·b(x_{t_{i-1}},u_{t_{i-1}},\theta )(t_i-t_{i-1})\right\},\hfill \end{array}$$

with b and $\sigma$ as in (1). The MLR function generates the discrete approximate likelihood ratio estimator:

$${\theta }_{LR}\equiv {\theta }_{LR}\left(X_T\right):=\arg \underset{\theta \in \mathsf{\Theta }}{\max }MLR(X_T,\theta ).$$

We establish our main result considering Remarks 3 and 4.

Theorem 1.

Let ${\theta }_m$ be a sequence of USC estimators of $\theta \in \mathsf{\Theta }$, and let ${\lambda }_{x,{\theta }_m}^{*}<0$ be a critical point of $V^{*}(x,r^{\lambda },{\theta }_m)$. Assume that there is a sequence $\left({\pi }^{{\lambda }_{x,{\theta }_m}^{*}}\in {\mathsf{\Pi }}^{{\lambda }_{x,{\theta }_m}^{*}}:m=1,\dots \right)$ that converges to $\pi \in \mathsf{\Pi }$. Then, π is optimal for the DPC. Moreover, the equalities $V(x,\pi ,c,\theta )=\overline{\eta }(x,\pi ,\theta )$ and $V^{*}(x,r^{{\lambda }_x^{*}},\theta )=V^{*}(x,r,\theta )$ hold ${\mathbb{P}}^{\pi ,\theta }$-a.s.

Proof. 

By Remark 3(a), we get

$$\begin{array}{cc}\hfill & V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},r^{{\lambda }_{x,{\theta }_m}^{*}},{\theta }_m)\hfill \\ & =V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},r,{\theta }_m)+{\lambda }_{x,{\theta }_m}^{*}\left[V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c,{\theta }_m)-\overline{\eta }(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},{\theta }_m)\right],\hfill \end{array}$$

(17)

whereas the facts in Remark 4(a)–(c) ensure that (17) converges to

$$\begin{array}{ccc}\hfill V(x,\pi ,r^{{\lambda }_{x,\theta }},\theta )& =& V(x,\pi ,r,\theta )+{\lambda }_{x,\theta }\left[V(x,\pi ,c,\theta )-\overline{\eta }(x,\pi ,\theta )\right]\hfill \end{array}$$

(18)

${\mathbb{P}}^{\pi ,\theta }$-a.s., as $m\to \infty$.

On the other hand, note that

$$\begin{array}{cc}\hfill |V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c-\alpha \eta ,{\theta }_m)-V(x,\pi ,c-\alpha \eta ,\theta )|& \le \end{array}$$

$$\begin{array}{cc}\hfill |V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c-\alpha \eta ,{\theta }_m)-V(x,\pi ,c-\alpha \eta ,{\theta }_m)|& +\end{array}$$

(19)

$$\begin{array}{cc}\hfill |V(x,\pi ,c-\alpha \eta ,{\theta }_m)-V(x,\pi ,c-\alpha \eta ,\theta )|,& \end{array}$$

(20)

thus, by Remark 4(a) we can obtain that the term in (19) converges to zero as $m\to \infty$; whereas, using Remark 4(b), we can deduce that the term in (20) converges to zero ${\mathbb{P}}^{\pi ,\theta }$-a.s. So,

$$V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c-\alpha \eta ,{\theta }_m)\to V(x,\pi ,c-\alpha \eta ,\theta ){\mathbb{P}}^{\pi ,\theta }\text{-}\mathrm{a}.\mathrm{s}.$$

(21)

As ${\lambda }_{x,{\theta }_m}^{*}<0$ is a critical point of $V^{*}(x,r^{\lambda },{\theta }_m)$, we obtain, from (14), that for every ${\pi }^{{\lambda }_{x,{\theta }_m}^{*}}\in {\mathsf{\Pi }}^{{\lambda }_{x,{\theta }_m}^{*},{\theta }_m}$,

$$\begin{array}{cc}\hfill \frac{\partial V^{*}(x,r^{\lambda },{\theta }_m)}{\partial \lambda }{|}_{\lambda ={\lambda }_{x,{\theta }_m}^{*}}& =V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c,{\theta }_m)-\overline{\eta }(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},{\theta }_m)\hfill \\ \hfill & =V(x,{\pi }^{{\lambda }_{x,{\theta }_m}^{*}},c-\alpha \eta ,{\theta }_m)=0.\hfill \end{array}$$

(22)

Therefore, from (21) and (22), we obtain

$$V(x,\pi ,c-\alpha \eta ,\theta )=V(x,\pi ,c,\theta )-\overline{\eta }(x,\pi ,\theta )=0{\mathbb{P}}^{\pi ,\theta }\text{-}\mathrm{a}.\mathrm{s}.$$

(23)

This last result, along with (18) yields

$$\begin{array}{ccc}\hfill V(x,\pi ,r^{{\lambda }_x^{*}},\theta )& =& V(x,\pi ,r,\theta ){\mathbb{P}}^{\pi ,\theta }\text{-}\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(24)

Using Remark 4(g), $\pi$ is in ${\mathsf{\Pi }}^{{\lambda }_x^{*},\theta }$,

$$V^{*}(x,r^{{\lambda }_x^{*}},\theta )=\underset{\overline{\pi }\in \mathsf{\Pi }}{\sup }V(x,\overline{\pi },r^{{\lambda }_x^{*}},\theta )=V(x,\pi ,r^{{\lambda }_x^{*}},\theta ).$$

(25)

We have that, for all $\overline{\pi }\in {\mathcal{F}}_{\theta }^x$,

$$V(x,\overline{\pi },c,\theta )-\overline{\eta }(x,\overline{\pi },\theta )\le 0.$$

This implies that ${\lambda }_x^{*}[V(x,\overline{\pi },c,\theta )-\overline{\eta }(x,\overline{\pi },\theta )]\ge 0$, which, in turn, together with (10), (24), (25) and Remark 3(a), lead to

$$\begin{array}{ccc}\hfill V(x,\pi ,r,\theta )& =& V(x,\pi ,r^{{\lambda }_x^{*}},\theta )=V^{*}(x,r^{{\lambda }_x^{*}},\theta )\hfill \\ & \ge & V(x,\overline{\pi },r^{{\lambda }_x^{*}},\theta )\hfill \\ & =& V\left(x,\overline{\pi },r,\theta \right)+{\lambda }_x^{*}\left[V(x,\overline{\pi },c,\theta )-\overline{\eta }(x,\overline{\pi },\theta )\right]\hfill \\ & \ge & V\left(x,\overline{\pi },r,\theta \right)\mathrm{for} \mathrm{all}\overline{\pi }\in {\mathcal{F}}_{\theta }^x{\mathbb{P}}^{\pi ,\theta }\text{-}\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

Thus,

$$\begin{array}{ccc}\hfill V(x,\pi ,r,\theta )& \ge & \underset{\overline{\pi }\in {\mathcal{F}}_{\theta }^x}{\sup }V\left(x,\overline{\pi },r,\theta \right){\mathbb{P}}^{\pi ,\theta }-\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(26)

Finally, by (25) we obtain that $\pi \in {\mathcal{F}}_{\theta }^x$. Therefore, (26), along with (24) and (25) show that

$$\begin{array}{ccc}\hfill V^{*}(x,r^{{\lambda }_x^{*}},\theta )& =& V(x,\pi ,r,\theta )=\underset{\overline{\pi }\in {\mathcal{F}}_{\theta }^x}{\sup }V\left(x,\overline{\pi },r,\theta \right),\hfill \end{array}$$

in other words, $\pi$ is optimal for the DPC, and $V^{*}(x,r^{{\lambda }_x^{*}},\theta )$ coincides with the optimal reward for the DPC ${\mathbb{P}}^{\pi ,\theta }$-a.s. □

4. Numeric Illustration

To exemplify our results, we substitute (1) by

$$\mathrm{d}x\left(t\right)=\left[u\right(t)-\theta x(t\left)\right]\mathrm{d}t+\sigma \mathrm{d}W\left(t\right),x\left(0\right)=x>0,$$

(27)

where $0\le u\left(t\right)\le \gamma$. We assume that the reward and cost rates $r,c:[0,\infty )\times U\to \mathbb{R}$, as well as the constraint rate $\eta :[0,\infty )\times \mathsf{\Theta }\to \mathbb{R}$, are defined by

$$\begin{array}{ccc}\hfill r(x,u)& :=& \sqrt{u}-ax\forall (x,u)\in [0,\infty )\times U \mathrm{with} a\ge 0,\hfill \\ \hfill c(x,u)& =& c_{1}x+c_{2}u\forall (x,u)\in [0,\infty )\times U,\hfill \\ \hfill \eta (x,\theta )& :& =\frac{c_{1}x}{\alpha +\theta }+q,\mathrm{for} \mathrm{all} x>0,\hfill \end{array}$$

with $U:=[0,\gamma ]$, $c_1>0$, $c_2\in \mathbb{R}$ satisfying $c_1+\theta c_2>0$ and $\theta c_1+{(\alpha +\theta )}^2{c}_2>0,$ where q is a positive constant. Here, $r(x,u)$ represents the social welfare, where $\sqrt{u}$ and $ax$ represent the social utility of the consumption u and the social disutility of the pollution x, respectively.

Remark 5.

Assumptions A1–A3 given in this work hold for the controlled diffusion (27); see Lemma 5.2 in [11]. In fact, the Lyapunov function in Assumption A2 is taken as $w\left(x\right)=x^2+1$.

4.1. The $\lambda$-DUP

Lemma 5.3 in [11] ensures that, under the conditions imposed on the constants $c_1$, $c_2$, $\alpha$, $\theta$ and q given above, for every $x>0$ and $\lambda \le 0$, the optimal reward $V^{*}(x,r^{\lambda },\theta )$ in (10) with $r^{\lambda }(x,u)=\sqrt{u}-ax+\lambda \left[c_{1}x+c_{2}u-\frac{c_{1}x}{\alpha +\theta }-q\right]$, becomes

$$V^{*}(x,r^{\lambda },\theta )=\frac{[\lambda \theta c_1-(\alpha +\theta )a]x}{{(\alpha +\theta )}^2}+\frac{\sqrt{f_{\theta }^{\lambda }}-a_{\lambda ,\theta }{f}_{\alpha }^{\lambda }}{\alpha }-\lambda q,$$

where

$$a_{\lambda ,\theta }:=\frac{(\alpha +\theta )a-\lambda [\theta c_1+{(\alpha +\theta )}^2{c}_2]}{{(\alpha +\theta )}^2}>0,$$

and the discounted optimal policy for the $\lambda \text{-}DUP$ ($f_{\theta }^{\lambda }\in \mathbb{F}$), which maximizes the right-hand side of (11) for this example, is the constant function given by

$$f_{\theta }^{\lambda }=\left\{\begin{array}{cc}\frac{1}{4{\left(a_{\lambda ,\theta }\right)}^2}\hfill & \mathrm{if}\frac{1}{2\sqrt{\gamma }}<a_{\lambda ,\theta },\hfill \\ \hfill & \\ \gamma \hfill & \mathrm{if}\frac{1}{2\sqrt{\gamma }}\ge a_{\lambda ,\theta }.\hfill \end{array}\right.$$

(28)

4.2. The Dpc

Using Theorem 5.5 in [11], for a fixed point $z>0$ such that

$$q<\frac{\eta c_{1}z}{{(\alpha +\theta )}^2}+\frac{[\theta c_1+{(\alpha +\theta )}^2{c}_2]\gamma }{\alpha {(\alpha +\theta )}^2}.$$

If $\frac{1}{2\sqrt{\left(\frac{\alpha {(\alpha +\theta )}^{2}q-\alpha \theta c_{1}z}{\theta c_1+{(\alpha +\theta )}^2{c}_2}\right)}}>\frac{a}{\alpha +\theta }$, then the mapping $\lambda ⟼V^{*}(z,r^{\lambda },\theta )$ admits a critical point ${\lambda }_{z,\theta }^{*}\equiv {\lambda }_{z,\theta }^{*}(\alpha ,z)<0$ satisfying

$$a_{{\lambda }_{z,\theta }^{*},\theta }=\frac{(\alpha +\theta )a-{\lambda }_{z,\theta }^{*}[\theta c_1+{(\alpha +\theta )}^2{c}_2]}{{(\alpha +\theta )}^2}=\frac{1}{2\sqrt{\left(\frac{\alpha {(\alpha +\theta )}^{2}q-\alpha \theta c_{1}z}{\theta c_1+{(\alpha +\theta )}^2{c}_2}\right)}}.$$

Hence, every ${\pi }^{{\lambda }_{z,\theta }^{*}}\in {\mathsf{\Pi }}^{{\lambda }_{z,\theta }^{*}}$ is $\alpha$-optimal for the DPC and $V(z,{\pi }^{{\lambda }_{z,\theta }^{*}},c,\theta )=\overline{\eta }(z,{\pi }^{{\lambda }_{z,\theta }^{*}},\theta )$; in particular, the corresponding $\alpha$-optimal policy for the DPC is $f_{\theta }^{{\lambda }_{z,\theta }^{*}}\in \mathbb{F}\cap {\mathsf{\Pi }}^{{\lambda }_{z,\theta }^{*}}$

$$f_{\theta }^{{\lambda }_{z,\theta }^{*}}=\frac{\alpha {(\alpha +\theta )}^{2}q-\alpha \theta c_{1}z}{\theta c_1+{(\alpha +\theta )}^2{c}_2},$$

(29)

and the $\alpha$-optimal value for the DPC is given by

$$\begin{array}{cc}\hfill & V^{*}(z,r^{{\lambda }_{z,\theta }^{*}},\theta )=V^{*}(z,{\pi }^{{\lambda }_{z,\theta }^{*}},r,\theta )\hfill \\ \hfill & =-\frac{az}{\alpha +\theta }+\frac{1}{\alpha }\sqrt{\left(\frac{\alpha {(\alpha +\theta )}^{2}q-\alpha \theta c_{1}z}{\theta c_1+{(\alpha +\theta )}^2{c}_2}\right)}-\frac{a}{\alpha +\theta }\left[\frac{{(\alpha +\theta )}^{2}q-\theta c_{1}z}{\theta c_1+{(\alpha +\theta )}^2{c}_2}\right].\hfill \end{array}$$

(30)

4.3. Numerical Results for the Optimal Accumulation Problem

To implement the optimal controller (28), we estimate the unknown parameter with LSE (15) and (16). By replacing $b(x,u,\theta )=u\left(t\right)-\theta x\left(t\right)$ in (15), a direct computation yields:

$${\theta }_{LS{E}_m}=\frac{{\displaystyle \sum _{i=2}^{m-1}u{x}_i-\mathrm{d}{x}_i{x}_i}}{{\displaystyle \sum _{i=2}^{m-1}{x}_i^2}},$$

where $\mathrm{d}{x}_i:=\frac{1}{2}\frac{x_{i+1}-x_{i-1}}{t_{i+1}-t_i}.$

Assume that the true parameter value of the parameter is $\theta =2.5$, and take $u\left(t\right)=0.5$, $\alpha =0.2$, $T=2.8$, $c_1=100$, $c_2=10$, $\lambda =-0.1$, $q=60$, $a=1.25$, and $\gamma =3$. We next obtained discrete observations of the stochastic differential Equation (27) by simulating the equation using the Euler–Maruyama technique on $[0,2.8]$. Based on this information, we obtained $m=1200$ observations with different values of the diffusion coefficient $\sigma =0.0001,0.0007.$

4.3.1. Numerical Results for the $\lambda$-DUP

In

Table 1. RMSE and absolute error between the estimated processes and the real processes ($\theta =2.5$) with $\sigma =0.0001$.

			$\mathit{\sigma }=\mathbf{0.0001}$
$\mathit{m}$	${\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}$	$\mathit{RMSE}({\mathit{V}}_{{\mathit{\theta }}_{\mathit{m}}}^{*}-{\mathit{V}}_{\mathit{\theta }}^{*})$	$\mathit{RMSE}({\mathit{x}}_{{\mathit{\theta }}_{\mathit{m}}}^{*}-{\mathit{x}}_{\mathit{\theta }}^{*})$	${\mathit{f}}^{\mathit{\lambda },{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}$	$|{\mathit{f}}^{\mathit{\lambda },{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}-{\mathit{f}}^{\mathit{\lambda },\mathit{\theta }}|$
240	$2.499627$	$0.00204305$	0.000509042	$0.00410459$	$3.1947\times {10}^{-8}$
300	$2.499643$	$0.00161749$	0.00040062	$0.00410458$	$3.0571\times {10}^{-8}$
400	$2.499659$	$0.00117931$	0.000285924	$0.00410458$	$2.91879\times {10}^{-8}$
600	$2.499678$	$0.000670105$	0.000155536	$0.00410458$	$2.75402\times {10}^{-8}$
1200	$2.499682$	$0.000437844$	$9.76504\times {10}^{-5}$	$0.00410458$	$2.72459\times {10}^{-8}$
Real	$2.5$	0	0	$0.00410455$	0

and

Table 2. RMSE and absolute error between the estimated processes and the real processes ($\theta =2.5$) with $\sigma =0.0007$.

			$\mathit{\sigma }=\mathbf{0.0007}$
m	${\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}$	$\mathit{RMSE}({\mathit{V}}_{{\mathit{\theta }}_{\mathit{m}}}^{*}-{\mathit{V}}_{\mathit{\theta }}^{*})$	$\mathit{RMSE}({\mathit{x}}_{{\mathit{\theta }}_{\mathit{m}}}^{*}-{\mathit{x}}_{\mathit{\theta }}^{*})$	${\mathit{f}}^{\mathit{\lambda },{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}$	$|{\mathit{f}}^{\mathit{\lambda },{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}-{\mathit{f}}^{\mathit{\lambda },\mathit{\theta }}|$
240	$2.498748$	$0.00299057$	0.000718096	$0.00410466$	$1.06613\times {10}^{-7}$
300	$2.498791$	$0.00266943$	0.000629978	$0.00410466$	$1.03033\times {10}^{-7}$
400	$2.498873$	$0.00184702$	0.000423607	$0.00410465$	$9.6021\times {10}^{-8}$
600	$2.498762$	$0.00176898$	0.000437829	$0.00410466$	$1.05499\times {10}^{-7}$
1200	$2.498762$	$0.00136883$	0.00033381	$0.00410466$	$1.05454\times {10}^{-7}$
Real	$2.5$	0	$0.00410455$	0

, we denote the root mean square error (RMSE) between the predicted process $x^{{\theta }_{LS{E}_m},f_{{\theta }_{LS{E}_m}}^{\lambda }}\left(t\right)$ and the real process $x^{\theta ,f_{\theta }^{\lambda }}\left(t\right)$ by $RMSE(x_{{\theta }_m}^{*}-x_{\theta }^{*})$, and the RMSE between the predicted optimal discount cost $V_{{\theta }_{LS{E}_m}}^{*}\left(x\right)$ and the real optimal discount cost $V_{\theta }^{*}\left(x\right)$, by $RMSE(V_{{\theta }_m}^{*}-V_{\theta }^{*})$.

Table 1 and Table 2 take $m=240,300,400,600$ and 1200 and display the information on the different values of ${\theta }_{LS{E}_m}$. As can be seen, as m increases, the estimator approaches the true parameter value $\theta =2.5$, and the RMSEs between the predicted processes $x{\left(t\right)}^{{\theta }_{LS{E}_m},f_{{\theta }_{LS{E}_m}}^{\lambda }}$, $V_{{\theta }_{LS{E}_m}}^{*}\left(x\right)$, and the real processes $x{\left(t\right)}^{\theta ,f_{\theta }^{\lambda }}$, $V_{\theta }^{*}\left(x\right)$ decrease, thus implying a good fit (see Figure 1 and Figure 2). We can also see that, as the amount of data increases, the absolute error between the predicted optimal control $f_{{\theta }_{LS{E}_m}}\left(x^{{\theta }_{LS{E}_m}}\left(t\right)\right)$, and the real optimal control $f_{\theta }\left(\left(x^{\theta }\left(t\right)\right)\right)$ decreases. Therefore, the predicted optimal control approaches the true optimal control.

The diffusion process (27) with $\sigma =0.0001$ showed the best fit because, with 1200 data, ${\theta }_{LS{E}_m}=2.499682$ and its RMSE is $9.76504\times {10}^{-5}$, which suggests that the lower the noise in the measured data, the more accurate the least square estimator.

4.3.2. Numerical Results for the DPC

Table 3. Estimated processes and the real processes ($\theta =2.5$) with $\sigma =0.0001$.

m	${\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}}$	$\left|{\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}}-{\mathit{f}}_{\mathit{\theta }}^{{\mathit{\lambda }}_{\mathit{z},\mathit{\theta }}^{*}}\right|$	${\mathit{V}}^{*}\left(\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}},{\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}},\mathit{r}\right)$	$|{\mathit{V}}_{\mathit{z}}^{*}\left({\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}\right)-{\mathit{V}}_{\mathit{z}}^{*}\left(\mathit{\theta }\right)|$
240	$0.116042$	$3.11391\times {10}^{-5}$	$0.971567$	$0.000257628$
300	$0.116043$	$2.97955\times {10}^{-5}$	$0.971578$	$0.000246511$
400	$0.116045$	$2.84453\times {10}^{-5}$	$0.971589$	$0.00023534$
600	$0.116046$	$2.68369\times {10}^{-5}$	$0.971602$	$0.000222032$
1200	$0.116047$	$2.65498\times {10}^{-5}$	$0.971605$	$0.000219656$
Real	$0.116073$	0	$0.971824$	0

and

Table 4. Estimated processes and the real processes ($\theta =2.5$) with $\sigma =0.0007$.

m	${\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}}$	$\left|{\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}}-{\mathit{f}}_{\mathit{\theta }}^{{\mathit{\lambda }}_{\mathit{z},\mathit{\theta }}^{*}}\right|$	${\mathit{V}}^{*}\left(\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}},{\mathit{f}}_{{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{{\mathit{\lambda }}_{\mathit{z},{\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}}^{*}},\mathit{r}\right)$	$|{\mathit{V}}_{\mathit{z}}^{*}\left({\mathit{\theta }}_{{\mathit{LSE}}_{\mathit{m}}}\right)-{\mathit{V}}_{\mathit{z}}^{*}\left(\mathit{\theta }\right)|$
240	$0.115969$	0.000104368	$0.970961$	$0.000863615$
300	$0.115972$	0.000100842	$0.97099$	$0.00083443$
400	$0.115979$	$9.39409\times {10}^{-5}$	$0.971047$	$0.000777316$
600	$0.11597$	0.000103271	$0.97097$	$0.000854533$
1200	$0.11597$	0.000103226	$0.97097$	$0.000854166$
Real	$0.116073$	0	$0.971824$	0

show the predicted optimal controls $f_{{\theta }_{LS{E}_m}}^{{\lambda }_{z,{\theta }_{LS{E}_m}}^{*}}$ defined in (29), as well as the predicted $\alpha$-optimal rewards for the DPC $V^{*}\left(z,{\theta }_{LS{E}_m},f_{{\theta }_{LS{E}_m}}^{{\lambda }_{z,{\theta }_{LS{E}_m}}^{*}},r\right)$ given in (30) and denoted by $V_z^{*}\left({\theta }_{LS{E}_m}\right)$. As m increases, the estimator approaches the true parameter value $\theta =2.5$, and the predicted optimal controls and rewards converge to the real optimal control and reward $f_{\theta }^{{\lambda }_{z,\theta }^{*}}$, $V^{*}(z,\theta ,f_{\theta }^{{\lambda }_{z,\theta }^{*}},r),$ respectively, implying a good fit. Again, $\sigma =0.001$ showed the best fit, which suggests that the lower the noise in the measured data, the more accurate the LSE.

5. Concluding Remarks

This paper concerns controlled stochastic differential equations of the form (1), where the drift coefficient depends on an unknown parameter $\theta \in \mathsf{\Theta }$. Using a statistical estimation procedure to find $\theta$, we constructed adaptive policies, which are almost certainly optimal for the constrained optimization problem under the discounted payoff on an infinite horizon. To this end, we let $V_{\theta }^{*}(x,r^{\lambda }):={\sup }_{\pi \in \mathsf{\Pi }}V(x,\pi ,r^{\lambda },\theta )$, and $V_{\theta }^{*}(x,r):={\sup }_{\pi \in {\mathcal{F}}_{\theta }^x}V(x,\pi ,r,\theta )$ be the optimal discounted rewards for the DUP and the DPC, respectively. Our results are our own version of the PEC, and can be summarized as follows:

1.

For each m, there are optimal control policies ${\pi }_{{\theta }_m}$ for the ${\theta }_m$-DUP and ${\theta }_m$-DPC.

2.

For each initial state $x\in {\mathbb{R}}^n$, $V_{{\theta }_m}^{*}(x,r^{\lambda })\to V_{\theta }^{*}(x,r^{\lambda })$ and $V_{{\theta }_m}^{*}(x,r)\to V_{\theta }^{*}(x,r)$, and is almost certainly $m\to \infty$.

3.

For the DUP, there is a subsequence $(m_k:k=1,\dots )$ of $\left(m\right)$ and a policy ${\pi }_{\theta }^{*}\in {\mathsf{\Pi }}^{\lambda ,\theta }$, such that ${\pi }^{{\lambda }_{m_k},{\theta }_{m_k}}$ converges to ${\pi }_{\theta }^{*}$ in the topology of relaxed controls, and, moreover, ${\pi }_{\theta }^{*}$ is optimal for the $\theta$-DUP. Moreover, if ${\lambda }_{m_k}<0$ is a critical point of $V_{{\theta }_{m_k}}^{*}(x,r^{\lambda })$, then ${\pi }_{\theta }^{*}$ is optimal for the $\theta$-DCP.

Some of the techniques we use are standard in the context of dynamic programming, and our use of the discounted payoff criterion on the infinite horizon renders the problem a rather flexible one. This criterion emphasizes the weight of the rewards and costs on the present generations while tending to overlook their effects on future generations. One way to prevent this from happening is to use the ergodic criterion by means of the so-called vanishing technique, which has been used in, for instance [43]. To obtain some insight as to how this method would alter the value function and the overall results presented here, we invite the reader to let $\alpha \downarrow 0$.

There are many ways to obtain a sequence of USC estimators of the unknown parameter $\theta$ (see [18]). However, when implementing the approximation algorithms for $\theta$, one needs to check the type numerical approximation of the derivative $\mathrm{d}x\left(t\right)$ that is required. In our case, we replaced $\mathrm{d}x\left(t\right)$ with its central difference, instead of the backward difference, because our applications yield more accurate approximations.

The PEC requires knowledge and storage of the optimal policies ${\pi }_{\theta }$ for all values of $\theta$, which may require considerable off-line computation and considerable storage. Therefore, for optimal control problems with closed-form solutions (${\pi }_{\theta }^{*},V_{\theta }^{*}\left(x\right)$), such as, for example, LQ problems (linear systems with quadratic costs), the PEC works well. In this sense, our model resembles the model predictive control. However, the fact that the horizon has to be finite in the latter is a serious limitation that is surpassed by our proposal. In fact, the numeric illustration from Section 4 is another example of the distinction between our version of the PEC, and the polynomial chaos expansion method. While the latter aims to approximate the probability densities of finite-variance random variables, our goal is to obtain an optimal control while making estimations of the infinite-total variation processes (1). This is particularly true in the case of (27), regardless of how small the diffusion coefficient $\sigma$ is in our illustration. There, the focus point should be that the lower the noise in the measured data, the more accurate the LSE.

One of the downfalls of our method is that a closed-form solution of ${\pi }_{\theta },V_{\theta }^{*}\left(x\right)$ is virtually impossible for many optimal control problems with or without constraints. Another limitation is that, for each application, there is a large number of assumptions and constraints that need to be verified. We believe this deflection from the main problem could be eased by the inclusion of our method in (for instance) MATLAB’s robust control.

The second part of this project will approximate the adaptive original problem using a sequence of discrete-time adaptive optimal control problems of controlled Markov switching diffusions.