Optimal Inspection and Maintenance Policy: Integrating a Continuous-Time Markov Chain into a Homing Problem

Abstract

The state of a machine is modeled as a controlled continuous-time Markov chain. Moreover, the machine is being serviced at random times. The aim is to maximize the time until the machine must be repaired, while taking the maintenance costs into account. The dynamic programming equation satisfied by the value function is derived, enabling optimal decision-making regarding inspection rates, and special problems are solved explicitly. This approach minimizes direct maintenance costs along with potential failure expenses, establishing a robust methodology for determining inspection frequencies in reliability-centered maintenance. The results contribute to the advancement of maintenance strategies and provide explicit solutions for particular cases, offering ideas for application in reliability engineering.

1. Introduction

The preventive maintenance framework is an essential element in ensuring the reliability and efficiency of operating systems in the context of reliability engineering, maintenance, and availability optimization. Maintenance planning takes into account scheduled tasks to ensure the continuity of operations and prevent breakdowns by proactively managing potential failures before they occur. Optimizing these strategies may prove more difficult if degradations are considered stochastic processes, and if the inherent randomness of system operation is taken into account. A mathematical model commonly used to represent these processes is the continuous-time Markov chain, which effectively represents the probabilistic state transitions of the machine over the planned period.

A key element of maintenance optimization is the adjustment of costs associated with maintenance activities and potential system failures. An effective maintenance policy must take into account both the direct costs of performing maintenance tasks and the indirect costs associated with system downtime, lost production, and repairs due to failures. This optimal adjustment is vital in areas where high demands are placed on system reliability and availability for operational safety and process costs.

Optimal control of the frequency of inspections, a fundamental element of preventive maintenance, is essential to achieving this objective. Inspection frequency dictates how often a system’s condition is checked, which in turn affects the scheduling and need for maintenance interventions. More frequent inspections can lead to excessive maintenance costs, while less frequent inspections can increase the risk of failure and the resulting downtime. Determining the optimal inspection rate is therefore essential to minimize maintenance costs and improve system reliability.

Our research has focused on developing an optimal inspection and maintenance plan for a system by modeling the state of degradation as a continuous-time Markov chain. The basic concept is to dynamically regulate the inspection rate u, which specifies the frequency or intensity of preventive maintenance tasks. This inspection rate u serves as a control parameter that balances the trade-off between maintenance actions and the risk of system failure. A higher u inspection rate results in more inspection tasks, which may reduce failure rates but simultaneously increases direct inspection–maintenance costs. Conversely, a low u rate reduces immediate maintenance expenditure but may result in higher long-term costs due to increased failures and downtime. The problem is to identify the optimal rate $u^{*}$ that balances the conflicting factors mentioned. The authors’ objective is to apply dynamic programming to determine the optimal value of u, which minimizes the long-term costs associated with both inspection operations and potential system downtime caused by failures.

Dynamic programming is perfectly aligned with this problem, as it allows for the systematic evaluation of different maintenance policies under uncertainty. The dynamic nature of the problem, where the state of the system and the optimal maintenance plan selection are influenced by past states and actions, necessitates a model which can handle sequential decision-making under randomness. By employing dynamic programming in a continuous-time Markov chain framework, this study tends to propose an approach that defines the optimal inspection–maintenance rate and yields valuable insights into the optimal timing of inspection–maintenance activities, based on the state of the system. Such insights can assist in modeling robust inspection–maintenance planning adaptable to real-time alterations in system conditions and operational requirements.

This methodology develops a new type of homing problem applied in reliability theory, characterized by decision-making under randomness. Deriving the dynamic programming equation and applying it involves notable theoretical challenges, which this research seeks to tackle. These complexities include precisely modeling the degradation phenomena, properly incorporating the inspection–maintenance and downtime costs, and ensuring the computational tractability when addressing the dynamic programming equations for sophisticated systems. Resolving these challenges will facilitate the formulation of more advanced inspection–maintenance strategies that are not only cost effective but also adaptable to diverse operational settings and varying risk preferences.

Literature Review

A thorough review of recent developments in the optimal control of continuous-time Markov chains, particularly in the context of reliability theory, reveals that the subject has been widely studied in this field, especially in the context of inspection–maintenance planning. Previous works cover a wide range of models and strategies for establishing optimal testing, maintenance strategies to improve machine reliability, availability, and safety while reducing costs. These explorations point to ongoing efforts to improve optimal control methods in this field.

We now present a chronological list of important articles related to our paper.

• G. Parmigiani [1], ‘Optimal scheduling of fallible inspections’. The article provides an optimal control model for a manufacturing system, applying a continuous-time Markov chain for the optimal control of maintenance frequencies, in order to improve system reliability.
• E. K. Boukas and Z. K. Liu [2], ‘Production and maintenance control for manufacturing systems’. This article deals with maintenance control using a continuous-time Markov process. The model focuses on the optimal control of preventive and corrective maintenance rates to improve system reliability.
• C.-H. Wang and S. H. Sheu [3], ‘Determining the optimal production-maintenance policy with inspection errors: using a Markov chain’. The paper develops an optimal policy for inspection intervals and maintenance for a deteriorating production system using a continuous-time Markov chain model to minimize total cost, taking into account system reliability.
• H. Suryadi and L. G. Papageorgiou [4], ‘Optimal maintenance planning and crew allocation for multipurpose batch plants’. This article proposes a mathematical programming method for optimizing maintenance planning in plants, using continuous-time Markov chains to model maintenance in order to improve system reliability.
• W. Liying, F. Youtong, S. Liying and L. Baoyou [5], ‘On fault diagnosis and inspection policy for deteriorating system’. It identifies failure diagnosis and inspection policy, and optimizes the inspection cycle to maximize revenue, enabling imperfect repairs before component replacement. A Markov vector process and a numerical example are used for validation.
• H. R. Golmakani and F. Fattahipour [6], ‘Optimal replacement policy and inspection interval for condition-based maintenance’. This work uses a Markov process to provide an optimal replacement strategy and inspection frequency for condition-based maintenance. The model tends to optimize maintenance costs, with inspection intervals balancing out the costs of preventive and failure replacements, thus improving system reliability.
• F. Naderkhani and V. Makis [7], ‘Optimal condition-based maintenance policy for a partially observable system with two sampling intervals’. The study develops an optimal conditional maintenance strategy using a continuous-time hidden Markov process. It tunes inspection intervals according to aging states, in order to minimize maintenance costs and improve reliability.
• K. He [8], ‘Optimal Maintenance Planning in Novel Settings’. This thesis develops the method of optimal maintenance planning in healthcare systems, focusing on events with imperfect inspection intervals and unpunctual preventive maintenance with the use of stochastic processes.
• Q. Sun, Z.-S. Ye and N. Chen [9], ‘Optimal inspection and replacement policies for multi-unit systems subject to degradation’. It proposes optimal inspection and replacement strategies for aging systems, using the Markov decision process framework. It uses the Wiener process to model component degradation and finds inspection intervals and replacement choices to minimize total operational cost while maintaining system reliability.
• P. Cao, K. Yang and K. Liu [10], ‘Optimal selection and release problem in software testing process: A continuous-time stochastic control approach’. It proposes optimal selection for software testing using continuous-time stochastic control. It takes into account the trade-off between testing costs and availability time to minimize total costs. The proposed model uses dynamic programming to determine when to test, release, or reject software, taking into account software reliability.
• Q. Sun, Z.-S. Ye and X. Zhu [11], ‘Managing component degradation in series systems for balancing degradation through reallocation and maintenance’. The control of component aging in serial systems is examined through the optimization of preventive replacement and reallocation policy by deploying stochastic optimization and Markov chain.
• C. P. Andriotis, K. G. Papakonstantinou and E. N. Chatzi [12], ‘Value of structural health monitoring quantification in partially observable stochastic environments’. This paper focuses on the optimal life cycle control of aging systems in an uncertain environment. It uses partially observable Markov decision processes to find optimal intervention and monitoring strategies.
• S. Gan, N. Yousefi and D. W. Coit [13], ‘Optimal control-limit maintenance policy for a production system with multiple process states’. It develops a maintenance strategy for a production system with several processing states, incorporating machine age and spare parts control to optimize maintenance tasks. The method uses a discrete-time Markov decision process to identify optimal maintenance activities, with the aim of minimizing overall long-term costs.
• P. Vrignat, F. Kratz and M. Avila [14], ‘Sustainable manufacturing, maintenance policies, prognostics and health management: A literature review’. This work contains a review of sustainable manufacturing focusing on maintenance policies, prognostics, and health management systems (HMS). It discusses the incorporation of Industry 4.0 and e-maintenance to evolve proactive maintenance strategies that are consistent with sustainability goals.
• P. G. Morato, K. G. Papakonstantinou, C. P. Andriotis, J. S. Nielsen and P. Rigo [15], ‘Optimal inspection and maintenance planning for deteriorating structural components through dynamic Bayesian networks and Markov decision processes’. The paper introduces a scheme integrating dynamic Bayesian networks and partially observable Markov decision processes to explore optimal planning of structural deterioration inspection and maintenance. It highlights the limitations of the heuristic method for optimizing structural reliability and minimizing costs through stochastic optimization.
• M. Roux, Y.-P. Fang and A. Barros [16], ‘Maintenance planning under imperfect monitoring: an efficient POMDP model using interpolated value function’. The field is improved by introducing an efficient partially observable Markov decision process model for maintenance planning in the case of imprecise observations.
• M. Lefebvre and P. Pazhoheshfar [17], ‘An optimal control problem for the maintenance of a machine’. The study develops an optimal control problem for machine maintenance using a discrete-time stochastic process model. The methodology consists of solving a dynamic programming equation to determine whether maintenance activities should be performed at each unit of time, given that the final time will be a random variable.
• Y. Wang and Y. Li [18], ‘Replacement policy for a single-component machine with limited spares in a finite time horizon’. This study develops a maintenance scheduling scheme with a Markov decision process to explore the optimal control of a system with limited spare parts over a finite period of time.
• S. Nasersarraf, S. Asadzadeh and Y. Samimi [19], ‘Determining the optimal policy in condition-based maintenance for electrical panels’. It develops a state-based optimal maintenance policy for electrical panels in parallel systems, using a proportional hazards model to account for failure dependency between components. The model aims to minimize expected total costs while maintaining system reliability by determining optimal inspection intervals and preventive replacement strategies.
• W. Wang and X. Chen [20], ‘Piecewise deterministic Markov process for condition-based imperfect maintenance models’. This work proposes a condition-based maintenance model using a piecewise deterministic Markov process. It integrates corrective and imperfect maintenance. The paper implements optimal control theory to explore the optimal maintenance policy, with the aim of minimizing total system cost.
• G. Wang and Z. Zhu [21], ‘Optimal control of sampled-data systems based on an optimized stochastic sampling’. This work covers the optimal control of a system with sampled data and stochastic sampling. The paper presents several cost functions, including one for sampling frequency, and implements dynamic programming to obtain optimal controllers in finite and infinite time.

Now, continuous-time Markov chains and dynamic programming are increasingly applied beyond classical industrial contexts, supporting advanced strategies in reliability and optimization as shown by recent studies (see, for example, Liu et al. [22] and Mancuso et al. [23]) in the context of prognostics and health management (PHM).

Despite remarkable developments in the optimal control of continuous-time Markov chains applied in the field of maintenance optimization, there are still a number of gaps in this literature. Many existing models are tailored to certain applications, such as production systems or healthcare, with incomplete evaluation of the optimal inspection rate in a more generalized setting. In addition, the use of dynamic programming to extract continuous decision variables, such as testing intervals, has not yet been studied, particularly in the context of reliability theory homing problems.

This research aims to fill existing gaps by proposing an extended framework for defining the optimal preventive maintenance and inspection rate of a system using continuous-time Markov chains and dynamic programming. The presented model will provide a robust approach to the homing problem in reliability theory optimization, which incorporates advanced stochastic control practices to formulate optimal maintenance planning. In addition, by synthesizing the latest advances in inspection–maintenance planning optimization and dynamic programming, this study contributes to the broader goal of advancing reliability theory and the cost-effectiveness of engineering systems, proposing practical policies for industries, where component reliability is of great importance.

This research addresses the question of optimal inspection times for maintenance in a generalized framework by improving the applicability of continuous-time Markov chains in a more global context. Section 3 of the paper deals with inspection rates obtained by dynamic programming, allowing us to propose a versatile methodology for determining the optimal inspection rate in a system, without limiting its relevance to specialized domains. This methodology is supported by Equation (18), where the generalized formulation of the optimal inspection rate calculation is described, enhancing the flexibility of the model for various maintenance application contexts.

The gap is further bridged by the inclusion of dynamic programming to determine continuous decision variables, such as the inspection rate, in the maintenance and reliability domain and in the context of the homing problem. The results, particularly in Section 4 and Section 5, illustrate how the determination of the optimal inspection is supported by dynamic programming, especially when random final times are taken into account in the cost function—a novel implementation of the homing problem in the discussed field of application. By focusing on continuous decision variables, the model provided fills this gap by proposing a robust approach to calculating inspection frequencies that would balance the cost and reliability resulting from the system’s maintenance–operating states as highlighted by our results.

2. Mathematical Formulation of the Problem

Let $X\left(t\right)$ denote the state of a machine at time t. We assume that the stochastic process $\{X\left(t\right),t\ge t_0\}$ is a continuous-time Markov chain having the following states:

$$\begin{array}{cc}0:& \mathrm{The}\mathrm{machine}\mathrm{is}\mathrm{working};\hfill \\ 1:& \mathrm{The}\mathrm{machine}\mathrm{is}\mathrm{undergoing}\mathrm{routine}\mathrm{maintenance};\hfill \\ 2:& \mathrm{The}\mathrm{machine}\mathrm{is}\mathrm{undergoing}\mathrm{prolonged}\mathrm{maintenance};\hfill \\ 3:& \mathrm{The}\mathrm{machine}\mathrm{is}\mathrm{being}\mathrm{repaired}.\hfill \end{array}$$

The transition diagram of the Markov chain is shown in Figure 1. This figure provides a visual illustration of state transitions during the machine’s operational life cycle, which is modeled as a continuous-time Markov chain. The arrows indicate possible transitions between defined states, each with specified transition probabilities. The machine can take different paths throughout its operation in the “Working” state. It can enter the “Routine Maintenance” state (with transition probability $p_{0,1}$), or the “Prolonged Maintenance” state (with transition probability $p_{0,2}$), or the “Repair” state (with transition probability $p_{0,3}$). Each of these transitions is triggered by particular maintenance requirements or failure events. At the end of “Routine Maintenance” or “Prolonged Maintenance”, the machine switches to the “Working” state as indicated by the transitions $p_{1,0}=1$ and $p_{2,0}=1$, respectively. This indicates that once maintenance is complete, the machine is once again operating at full capacity (as good as new). Similarly, the transition $p_{3,0}=1$ indicates that after “Repair”, the machine returns to the “Working” state, thus regaining full functionality (as if it were new).

The time spent by the Markov chain in state i is an exponential random variable $T_i$ with parameter ${\nu }_i$, for $i=0,1,2,3$. Moreover, inspection is carried out at random times. We assume that the time elapsed between two inspection operations is an exponential random variable with parameter $u>0$. When the machine is serviced, there is a probability equal to p that no serious problem will be discovered (and to $1-p$ that at least one important problem will be detected).

Let $p_{i,j}$ be the probability that the Markov chain will move to state j when it leaves state i for $i\ne j\in \{0,1,2,3\}$. We can write (see, for instance, Lefebvre [24]) that $p_{j,0}=1$ for $j=1,2,3$ and

$$p_{0,1}={\displaystyle \frac{pu}{u+{\nu }_0}},p_{0,2}={\displaystyle \frac{(1-p)u}{u+{\nu }_0}}\mathrm{and}{p}_{0,3}={\displaystyle \frac{{\nu }_0}{u+{\nu }_0}}.$$

(1)

Next, we define the first-passage time

$$\tau \left(i\right)=inf\{t\ge t_0:X\left(t\right)=3\mathrm{or}t=t_1(>t_0)\mid X\left(t_0\right)=i\}$$

(2)

for $i=0,1,2$ (with $\tau \left(3\right)=t_0$).

We suppose that the optimizer can choose any value of the constant $u\in U$ at any time instant so that $u=u\left(t\right)$ is the control variable and $p_{i,j}$ becomes $p_{i,j}\left(t\right)$. We look for the value $u^{*}\left(t\right)\ge 0$ of $u\left(t\right)$ that minimizes the expected value of the cost criterion

$$J\left(i\right)={\int }_{t_0}^{\tau \left(i\right)}\left\{f\left[u\right(t\left)\right]-\lambda \right\}\mathrm{d}t+K{I}_F,$$

(3)

where $f\left(0\right)=0$, $f\left[u\right(t\left)\right]>0$ for $u\left(t\right)>0$, $\lambda >0$, $K>0$, and $I_F$ is the indicator function of the event F: the optimizer chooses $u\left(t\right)\equiv 0$. Therefore, the aim is to maximize the expected time until the machine needs to be repaired, while taking into account the control costs $f\left[u\right(t\left)\right]$.

• Remarks.

(i)

The final cost K is imposed if the optimizer decides not to perform any maintenance, to reflect the fact that when the machine needs to be repaired (state 3), repair costs should be higher.

(ii)

The above problem is a particular homing problem, in which the optimizer controls a stochastic process until a certain event occurs; see Whittle [25,26]. The authors have recently considered homing problems for queuing systems; see Lefebvre and Yaghoubi [27,28].

(iii)

To the best of our knowledge, this is the first time that a homing problem will be treated for a continuous-time Markov chain that is not a queuing model. Numerous papers have been published in the field of reliability, in which continuous-time Markov chains have been used as models. Moreover, as we have seen in the literature review, optimal control problems for these models have also been the subject of numerous publications. In our problem, rather than controlling a stochastic process until a final time that is either fixed or infinite, we stop controlling the process at the instant when the machine needs to be repaired (or a given amount of time has elapsed). In reality, this instant is indeed a random variable.

(iv)

To validate the formulated model, we apply a series of actions designed to assess its robustness and reliability. As a first step, a sensitivity analysis is carried out to examine the effect of varying key parameters on the model’s results. This analysis enables us to verify that the model remains reliable and produces consistent results under a particular range of hypothetical scenarios. In addition, we analyze the model’s response to several initial conditions and parameters in order to simulate different system behaviors. These verification steps demonstrate the usefulness of the model for maintenance decision-making.

Now, the value function is defined by

$$F(i,t_0)=\underset{\begin{array}{c}u\left(t\right)\\ t\in [t_0,\tau \left(i\right))\end{array}}{inf}E\left[J\left(i\right)\right]$$

(4)

for $i=0,1,2,3$. We can write that $F(3,t_0)=0$. Moreover, when the Markov chain is in state 1 or 2, we have $u\left(t\right)=0$. It follows that

$$F(i,t_0)=-\lambda E\left[T_i\right]+F(0,t_0)=-{\displaystyle \frac{\lambda }{{\nu }_i}}+F(0,t_0)\mathrm{for}i=1,2.$$

(5)

• Remark. Let us denote by $F_0(0,t_0)$ the expected value of $J\left(0\right)$ if the optimizer uses no control at all so that $u\left(t\right)\equiv 0$. Since $f\left(0\right)=0$, we can write that

  $$F_0(0,t_0)=-\lambda E\left[T_0\right]+K=-{\displaystyle \frac{\lambda }{{\nu }_0}}+K.$$

  (6)

Thus, the value function $F(0,t_0)$ must necessarily be smaller than or equal to $-{\displaystyle \frac{\lambda }{{\nu }_0}}+K$.

In the next section, the dynamic programming equation satisfied by the function $F(0,t_0)$ will be derived. Then, in Section 3, a particular case of the above problem will be solved explicitly. In Section 4, the case when the constant $t_1$ in the definition of $\tau \left(i\right)$ tends to infinity will be treated. We will conclude this paper with a few remarks in Section 5.

3. Dynamic Programming Equation

Assume that the machine is working at time $t_0$ so that $X\left(t_0\right)=0$. We have

$$P[T_0<\Delta t]=1-e^{-{\nu }_0\Delta t}={\nu }_0\Delta t+o(\Delta t).$$

(7)

It follows that we can write that

$$X(t_0+\Delta t)=\left\{\begin{array}{cc}1& \mathrm{with}\mathrm{probability}{\nu }_0\Delta t{p}_{0,1}\left(t_0\right)[+o(\Delta t)],\hfill \\ 2& \mathrm{with}\mathrm{probability}{\nu }_0\Delta t{p}_{0,2}\left(t_0\right),\hfill \\ 3& \mathrm{with}\mathrm{probability}{\nu }_0\Delta t{p}_{0,3}\left(t_0\right),\hfill \\ 0& \mathrm{with}\mathrm{probability}1-{\nu }_0\Delta t.\hfill \end{array}\right.$$

(8)

• Remark. The fact that the various probabilities in the above equation are all proportional to $\Delta t$ is essential in order to use dynamic programming.

Let

$$g\left[u\right(t\left)\right]:=f\left[u\right(t\left)\right]-\lambda .$$

(9)

We have

$$\begin{array}{ccc}\hfill F(0,t_0)& =& \underset{\begin{array}{c}u\left(t\right)\\ t\in [t_0,\tau \left(0\right))\end{array}}{inf}E\left[{\int }_{t_0}^{t_0+\Delta t}g\left[u\left(t\right)\right]\mathrm{d}t+{\int }_{t_0+\Delta t}^{\tau \left(0\right)}g\left[u\left(t\right)\right]\mathrm{d}t\right]\hfill \\ & =& \underset{\begin{array}{c}u\left(t\right)\\ t\in [t_0,\tau \left(0\right))\end{array}}{inf}E\left\{g\left[u\left(t_0\right)\right]\Delta t+{\int }_{t_0+\Delta t}^{\tau \left(0\right)}g\left[u\left(t\right)\right]\mathrm{d}t+o(\Delta t)\right\}.\hfill \end{array}$$

(10)

Moreover,

$$E\left[{\int }_{t_0+\Delta t}^{\tau \left(0\right)}g\left[u\left(t\right)\right]\mathrm{d}t\right]=E\left[E\left[{\int }_{t_0+\Delta t}^{\tau \left(0\right)}g\left[u\left(t\right)\right]\mathrm{d}t|X(t_0+\Delta t)\right]\right].$$

(11)

Hence, using Bellman’s principle of optimality, we can write that

$$\underset{\begin{array}{c}u\left(t\right)\\ t\in [t_0+\Delta t,\tau \left(0\right))\end{array}}{inf}E\left[{\int }_{t_0+\Delta t}^{\tau \left(0\right)}g\left[u\left(t\right)\right]\mathrm{d}t\right]=E\left[F\left[X(t_0+\Delta t),t_0+\Delta t\right]\right].$$

(12)

Furthermore, we deduce from Equation (8) that [since $X\left(t_0\right)=0$]

$$\begin{array}{ccc}\hfill E\left[F\left[X(t_0+\Delta t),t_0+\Delta t\right]\right]& =& \sum _{j=1}^{2}F(j,t_0+\Delta t){\nu }_0\Delta t{p}_{0,j}\left(t_0\right)\hfill \\ & & +F(0,t_0+\Delta t)(1-{\nu }_0\Delta t).\hfill \end{array}$$

(13)

Next, assuming that the function $F(0,t_0)$ is differentiable with respect to $t_0$, we deduce from Taylor’s theorem that

$$F(·,t_0+\Delta t)=F(·,t_0)+\Delta t{F}^{\prime }(·,t_0)+o(\Delta t).$$

(14)

It follows that

$$F(·,t_0+\Delta t)(1-{\nu }_0\Delta t)=(1-{\nu }_0\Delta t)F(·,t_0)+\Delta t{F}^{\prime }(·,t_0)+o(\Delta t).$$

(15)

Making use of the above results, we find that

$$\begin{array}{ccc}\hfill 0& =& \underset{u\left(t_0\right)}{inf}\{g\left[u\left(t_0\right)\right]\Delta t+{\nu }_0\Delta t\sum _{j=1}^2{p}_{0,j}\left(t_0\right)F(j,t_0)\hfill \\ & & -{\nu }_0\Delta tF(0,t_0)+\Delta t{F}^{\prime }(0,t_0)+o(\Delta t)\}.\hfill \end{array}$$

(16)

Furthermore, we deduce from Equation (5) that

$$\begin{array}{ccc}\hfill 0& =& \underset{u\left(t_0\right)}{inf}\{g\left[u\left(t_0\right)\right]\Delta t+{\nu }_0\Delta t\sum _{j=1}^2{p}_{0,j}\left(t_0\right)\left\{-{\displaystyle \frac{\lambda }{{\nu }_j}}+F(0,t_0)\right\}\hfill \\ & & -{\nu }_0\Delta tF(0,t_0)+\Delta t{F}^{\prime }(0,t_0)+o(\Delta t)\}.\hfill \end{array}$$

(17)

Finally, dividing each side of the preceding equation by $\Delta t$ and taking the limit as $\Delta t$ decreases to zero, we can state the following proposition.

Proposition 1.

The value function $F(0,t_0)$ satisfies the dynamic programming equation (DPE)

$$0=\underset{u\left(t_0\right)}{inf}\left\{g\left[u\left(t_0\right)\right]+{\nu }_0\sum _{j=1}^2{p}_{0,j}\left(t_0\right)\left[-{\displaystyle \frac{\lambda }{{\nu }_j}}+F(0,t_0)\right]-{\nu }_{0}F(0,t_0)+F^{\prime }(0,t_0)\right\}.$$

(18)

Moreover, we have the boundary condition $F(0,t_1)=0$ if $u\left(t_0\right)>0$.

Let $u_0:=u\left(t_0\right)$. We have

$$p_{0,1}\left(t_0\right)={\displaystyle \frac{p{u}_0}{u_0+{\nu }_0}}\mathrm{and}{p}_{0,2}\left(t_0\right)={\displaystyle \frac{(1-p)u_0}{u_0+{\nu }_0}}.$$

(19)

Furthermore, the function $f(·)$ is often chosen of the form

$$f\left[u\left(t\right)\right]={\displaystyle \frac{1}{2}}{q}_0{u}^2\left(t\right),$$

(20)

where $q_0$ is a positive constant. Then, if $u\left(t\right)$ is a continuous function of t, we can obtain the optimal control $u_0^{*}$ in terms of $F(0,t_0,t_1)$ by differentiating the expression between the curly brackets in Equation (18) with respect to $u_0$ and solving the resulting equation for $u_0$.

If we are able to find an explicit expression for $u_0^{*}$, we replace it into Equation (18) and we try to solve the resulting differential equation, subject to the boundary condition $F(0,t_1)=0$.

In the next section, we will consider the simplest possible case, namely, the one when $u\left(t\right)\equiv k_1$ or $k_2$.

4. Particular Case

Suppose that the set U contains only two values, denoted by $k_1$ and $k_2$. We deduce from Equation (18) that we must consider the following equation:

$$\begin{array}{ccc}\hfill 0& =& g\left(k_i\right)+{\nu }_0\left({\displaystyle \frac{p{k}_i}{k_i+{\nu }_0}}\right)\left(-{\displaystyle \frac{\lambda }{{\nu }_1}}+F(0,t_0)\right)\hfill \\ & & +{\nu }_0\left({\displaystyle \frac{(1-p)k_i}{k_i+{\nu }_0}}\right)\left(-{\displaystyle \frac{\lambda }{{\nu }_2}}+F(0,t_0)\right)-{\nu }_{0}F(0,t_0)+F^{\prime }(0,t_0)\hfill \end{array}$$

(21)

for $i=1,2$.

Let us choose the function $f\left[u\right(t\left)\right]$ defined in Equation (20) with $q_0=2$. Furthermore, we take $\lambda ={\nu }_i=1$ for $i=0,1,2,3$, $p=1/2$, $k_1=1$ and $k_2=0.9$. With these values, we find that we must solve the first-order linear differential equations (denoting the function $F(0,t_0)$ by $F_i\left(t_0\right)$ if $u_0=k_i$, for $i=1,2$)

$$1=-F_1\left(t_0\right)+2F_1^{\prime }\left(t_0\right)$$

(22)

and

$$0=639+1000F_2\left(t_0\right)+1900F_2^{\prime }\left(t_0\right).$$

(23)

The solutions to the above equations that satisfy the boundary condition $F_i\left(t_1\right)=0$ for $i=1,2$ are

$$F_1\left(t_0\right)=-1+e^{(t_0-t_1)/2}$$

(24)

and

$$F_2\left(t_0\right)={\displaystyle \frac{639}{1000}}\left\{1-e^{10(t_1-t_0)/19}\right\}.$$

(25)

The two functions are presented in Figure 2 in the case when $t_1=2$. We see that the optimal solution in this example is to choose

$$u_0=\left\{\begin{array}{cc}0.9& \mathrm{if}0\le t_0\le t^{*},\hfill \\ 1& \mathrm{if}{t}^{*}\le t_0\le 2,\hfill \end{array}\right.$$

(26)

where $t^{*}\simeq 1.2285$.

• Remark. The value of $u_0$ given in Equation (26) is actually the optimal solution if we assume that the optimizer must choose $u\left(t\right)$ equal to $k_1=1$ or $k_2=0.9$ for any $t\in [t_0,t_1)$. However, if we admit the possibility that we can choose $u\left(t\right)\equiv 0$, then this result is correct at time $t_0$ if and only if the corresponding value function is less than or equal to $K-1$ [see Equation (6)]. When $K=0$, we find that the optimizer should use no control at all for $t_0\in [0.21,2)$ (approximately).

In the next section, we will consider the case when $t_1$ tends to infinity.

5. The Time-Invariant Case

Once we have computed the value function in the preceding section, we can take the limit as $t_1$ tends to infinity. Actually, as can be seen in the example that we presented, the value function will then become a constant. This is true because when $t_1\to \infty$, the problem becomes time-invariant so that $F^{\prime }(0,t_0)=0$. It follows that the DPE given in Equation (18) reduces to

$$0=\underset{u\left(t_0\right)}{inf}\left\{g\left[u\left(t_0\right)\right]+{\nu }_0\sum _{j=1}^2{p}_{0,j}\left(t_0\right)\left[-{\displaystyle \frac{\lambda }{{\nu }_j}}+F(0,t_0)\right]-{\nu }_{0}F(0,t_0)\right\}.$$

(27)

We will now present a problem that can be solved explicitly. Assume that the function $f(·)$ in the cost function $J\left(i\right)$ defined in Equation (3) is such that

$$f\left(u_0\right)={\displaystyle \frac{c{u}_0^2}{{(u_0+{\nu }_0)}^2}},$$

(28)

where c is a positive constant. Notice that $f\left(0\right)=0$ and $f\left(u_0\right)>0$ if $u_0>0$, as required. Moreover, we find that $f^{\prime }\left(u_0\right)>0$, so that the maintenance costs increase with $u_0$, which is logical.

Next, assume that ${\nu }_1={\nu }_2$. Then, differentiating Equation (27) with respect to $u_0$, we obtain, after simplification, that

$$0={\displaystyle \frac{2c{u}_0}{u_0+{\nu }_0}}-{\displaystyle \frac{{\nu }_0}{{\nu }_1}}+{\nu }_{0}F(0,t_0).$$

(29)

It follows that the optimal control is given by

$$u_0^{*}={\displaystyle \frac{{\nu }_0^2\left[1-{\nu }_{1}F(0,t_0)\right]}{2c{\nu }_1-{\nu }_0\left[1-{\nu }_{1}F(0,t_0)\right]}},$$

(30)

which must be non-negative.

Let us take ${\nu }_0={\nu }_1=1$. Then, $u_0^{*}$ becomes

$$u_0^{*}={\displaystyle \frac{1-F(0,t_0)}{2c-1+F(0,t_0)}}.$$

(31)

Substituting this expression into Equation (27), we obtain, if $p=1/2$, that the value function $F(0,t_0)$ satisfies the algebraic equation

$$0={\displaystyle \frac{{[1-F(0,t_0)]}^2}{4c}}-\lambda -\lambda {\displaystyle \frac{[1-F(0,t_0)]}{2c}}-{\displaystyle \frac{[2c-1-F(0,t_0)]}{2c}}F(0,t_0).$$

(32)

We find that

$$3F(0,t_0)=2c-\lambda -\sqrt{4c^2+8c\lambda +{\lambda }^2+6\lambda -3},$$

(33)

from which we deduce that the optimal control is given by

$$u_0^{*}={\displaystyle \frac{-3+2c-\lambda -\sqrt{{\lambda }^2+(8c+6)\lambda +4c^2-3}}{3-8c+\lambda +\sqrt{{\lambda }^2+(8c+6)\lambda +4c^2-3}}}.$$

(34)

Now, it is clear that the optimal control should be $u_0^{*}=0$ if $K=0$ and $\lambda \le 1$. Let us choose $\lambda =10$. Then, we find that the constant c must be such that

$$c>{\displaystyle \frac{1}{5}}\left(12+\sqrt{139}\right)\simeq 4.76.$$

(35)

For instance, if we take $c=5$, then $F(0,t_0)=-\sqrt{657}/3\simeq -8.544$ and

$$u_0^{*}={\displaystyle \frac{-1-\sqrt{73}}{1-\sqrt{73}}}\simeq 20.93.$$

(36)

The optimal control is shown in Figure 3 for $c\in [5,10]$.

• Remark. If $u\left(t\right)\equiv 0$, we have $F_0(0,t_0)=K-10$. Therefore, the optimal control is indeed equal to (approximately) $20.93$ when $\lambda =10$ and $c=5$ if and only if $K>1.456$.

6. Conclusions

In this paper, an optimal control problem known as a homing problem for a continuous-time Markov chain used as a model for the state of a machine has been considered. The aim was to find the optimal maintenance rate in order to maximize the time until the machine needs to be repaired, while taking the maintenance costs into account.

The model could be generalized by defining other states. For instance, there could be a state that corresponds to the case when the machine is broken down and cannot be repaired. If the optimizer uses no control at all (that is, if the machine is never serviced), there could be a high probability that the chain will move from the state when the machine is working to that state.

We were able to obtain explicit and exact solutions to two particular problems. In general, numerical methods or simulations could be used to obtain the value function and the optimal control.