New Methods of Entropy-Robust Estimation for Randomized Models under Limited Data

Abstract

The paper presents a new approach to restoration characteristics randomized models under small amounts of input and output data. This approach proceeds from involving randomized static and dynamic models and estimating the probabilistic characteristics of their parameters. We consider static and dynamic models described by Volterra polynomials. The procedures of robust parametric and non-parametric estimation are constructed by exploiting the entropy concept based on the generalized informational Boltzmann’s and Fermi’s entropies.

1. Introduction

The problem of useful information retrieval (we comprehend it as parametric and nonparametric estimation based on real data) is a major one in modern science. Different scientific disciplines suggest numerous methods of solving this problem. Each method stems from certain hypotheses regarding the properties of data accumulated during the normal functioning of their source. Among advanced scientific disciplines in this field, we mention mathematical statistics [1–4], econometrics [5–7], financial mathematics [8,9], control theory [10,11] and others.

Methods developed within their frameworks rest upon two groups of fundamental hypotheses. The first one relates to models, whereas the other concerns data. Notably, models are supposed to have well-defined parameters (we call them deterministic). Parameter values appear unknown and unmeasurable directly.

The second group of hypotheses applies to data and plays an essential role. In fact, these hypotheses are stated in terms of the statistical properties of data arrays (e.g., a sufficient number of data arrays, a property of a sample from a universal set, normal probability density). In practice, it seems impossible to check such properties and assumptions in concrete problems.

The described situation happens in a class of problems, where the sizes of real data arrays are limited and data incorporate errors [5,6,12].

Consequently, the characteristics (parameters) of a model are estimated by a small amount of incompletely reliable data. We can treat them as random objects. In this case, the estimated characteristics of a model acquire the properties of random variables.

Therefore, one naturally arrives at the idea of considering model parameters as random quantities. This idea transforms the model with deterministic parameters to the model with random parameters. In the sequel, we adopt the term of a randomized model (RM). The characteristics of an RM include the probability density functions (pdfs) of the model parameters. Thus, one should estimate the pdfs of model parameters (not the estimates of their values) based on available data. Having such estimates at one’s disposal, one can apply an RM for:

• constructing = moment models (MMs), where appropriate moments of random parameters serve as the model parameters;

• generating an ensemble of random vectors (ERV) of RM “output” with a pdf estimate (by the Monte Carlo method) and performing the statistical processing of the ensemble to form the desired numerical characteristics (including moment ones).

Both directions of RM usage enlarge appreciably the application domains of such models (especially, the ones with a high level of uncertainties). However, a researcher still faces a certain problem. How could the probability density functions of parameters be estimated in randomized models?

This paper proposes involving the informational entropy maximization principle on sets defined by the “input-output” measurements of an RM. The proposal originates from a couple of considerations discussed and formalized below.

The first consideration is connected with the generalized notion of likelihood, viz., with transition from likelihood functions to likelihood functionals and, subsequently, to informational entropy functionals.

The second consideration is based on methodological interpretations of the notion of informational entropy as a measure of uncertainty. Entropy maximization guarantees the best solutions under the maximal uncertainty. This line of reasoning was pioneered in [13]. Informational entropy characterizes uncertainty caused by random parameters of an RM and by measurement noises. The last property of informational entropy ensures the best estimates for maximally uncertain noises (in units of entropy). Hence, the pdf estimates resulting from informational entropy maximization can be viewed as robust. This interpretation varies from the classic conception of robustness suggested in [14].

2. Randomized Models

2.1. Static Objects

Consider a static parameterized object with a measurable “input” described by a matrix, X, of sizes (s × n) and a measurable “output” characterized by a vector, y, of length s. Here, s indicates the amount of input observations, and n stands for the number of the object’s parameters.

The relationship between the “input” and “output” (including measurement errors) is defined by the randomized static model (RSM):

$$\mathbf{v}=\mathbf{F}[X+\eta ,\mathbf{a}]+\xi$$

(1)

We adopt the following notation: F is a given s-dimensional vector-function; adesignates a random n-dimensional vector of some parameters with independent components a_i, $i=\overline{1,n}$, possessing values in the ranges ${𝒜}_i=[a_i^{-},a_i^{+}]$, $i=\overline{1,n}$.

The linear modification of the RSM acquires the form:

$$\mathbf{v}=(X+\eta )\mathbf{a}+\xi$$

(2)

Measurements incorporate errors modeled by a matrix, η, of sizes (s × n) (“input” errors) and by a vector, ξ, of length s (“output” errors). The elements ${\eta }_{ji}(j=\overline{1,s},\hspace{0.17em}\hspace{0.17em}i=\overline{1,n})$ and the components ${\xi }_j\left(j=\overline{1,s}\right)$ represent independent random variables. Their values lie within the intervals ${ℰ}_{ji}=[{\eta }_{ji}^{-},{\eta }_{ji}^{+}]$ and ${\Xi }_j=[{\xi }_j^{-},{\xi }_j^{+}]$, j ∈ [1, s], respectively.

2.2. Dynamic Objects

Consider a discrete dynamic object with a finite “memory”, m. The object’s input, x[k], is measured precisely, while the output, y[k], is measured with an additive noise, ξ[k]. Here, k ∈ $𝒯$ = [m, m + s], and s corresponds to the number of measurements. Suppose that the process, ξ[k], is random with independent values.

The connection between the observed input and output (including output measurement errors) is defined by a randomized dynamic model (RDM). This model is described by the discrete functional Volterra polynomial of degree, R [15]:

$$v[k]=\sum _{h=1}^R\sum _{(n_1,\dots ,n_h)=0}^m\left(w^{(h)}\hspace{0.17em}[n_1,\dots ,n_h]\prod _{r=1}^{h}x[k-n_r]\right)+\xi [k]$$

(3)

Equality Equation (3) employs the weight functions (actually, impulse responses) w^(h)[n₁, . . . , n_h]. These functions are random with independent ordinates belonging to the intervals:

$${𝒲}^{(h)}=\left[{\beta }_{-}^{(h)}\text{exp}\left(-{\alpha }^{(h)}(n_1+\cdots +n_h)\right),{\beta }_{+}^{(h)}\text{exp}\left(-{\alpha }^{(h)}(n_1+\cdots +n_h)\right)\right]$$

(4)

where ${\beta }_{\pm }^{(h)}$, ${\alpha }_{\pm }^{(h)}$ correspond to given constants.

The discrete dynamic model has the nonlinear expression Equation (3). Nevertheless, it can be linearized by the lexicographic ordering of the variables {n₁, . . . , n_h}. Renumber the resulting sets from zero to t_h = (m + 1)^h − 1 according to a lexicographic rule. Introduce a local index, i^(h) ∈ [0, t_h], and a set, such that:

$$\{n_1,\dots ,n_h\}\to i^{(h)},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{i}^{(h)}\in [0,t_h]$$

(5)

According to the accepted numbering, adopt the following indexing of random parameters that correspond to the values of weight functions in Equation (3):

$$a_{i^{(h)}}^{(h)}\to w^{(h)}[n_1,\dots ,n_h],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{i}^{(h)}\in [0,t_h]$$

(6)

Construct the random vector:

$${\mathbf{a}}^{(h)}=\{a_0^{(h)},\dots ,a_{t_h}^{(h)}\}$$

Components of the vector, a^(h), belong to the ranges:

$${𝒜}^{(h)}(i^{(h)})=\left[a_{-}^{(h)}(i^{(h)}),a_{+}^{(h)}(i^{(h)})\right]$$

(7)

By virtue of Equation (4), we have:

$$\begin{array}{l}{a}_{-}^{(h)}(i^{(h)})={\beta }_{-}^{(h)}\text{exp}\left[-i^{(h)}{\alpha }_{-}^{(h)}\right]\\ a_{+}^{(h)}(i^{(h)})={\beta }_{+}^{(h)}\text{exp}\left[-i^{(h)}{\alpha }_{+}^{(h)}\right]\end{array}$$

(8)

Similarly to Equation (5), introduce the lexicographic ordering of the variables {k − n₁, . . . , k − n_h}, where k indicates a fixed parameter and indexes n₁, . . . , n_h possess values in the interval [0, m]. For each fixed k, renumber the resulting sets according to Equation (5):

$$\{k-n_1,\dots ,k-n_h\}\to (k,i^{(h)})$$

(9)

Consequently, for fixed k in formula Equation (3), the lexicographically ordered products of the variables, x[k − n_r], form the vector:

$${\mathbf{x}}^{(h)}[k]=\left\{x_{k,0}^{(h)},\dots ,x_{k,t_h}^{(h)}\right\}$$

(10)

where:

$${\mathbf{x}}^{(h)}[k]={\mathbf{x}}^{(h)}[m+k],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k\in [0,s]$$

Therefore, equality Equation (3) can be rewritten as:

$$v[k]=\sum _{h=1}^R\langle {\mathbf{a}}^{(h)},{\mathbf{x}}^{(h)}[k]\rangle +\xi [k],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}v[k]=v[m+k],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\xi [k]=\xi [m+k];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k\in [0,s]$$

(11)

Consider the interval $𝒯$ = [m, m + s], which corresponds to the measurements of the RDM input and output at instants m + k, k ∈ [0, s]. The observed output and noise are characterized by the vectors:

$$\mathbf{v}=\{v[0],\dots ,v[s]\},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\xi =\{\xi [0],\dots ,\xi [s]\}$$

(12)

Define the input measurement matrices, whose rows are formed from vector Equation (10):

$$X^{(h)}=[{\mathbf{x}}^{(h)}[k],\hspace{0.17em}\hspace{0.17em}k\in [0,s]],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}h\in [1,R]$$

Build the block matrix, X, of sizes [(s + 1) × u], where $u={\sum }_{h=1}^R(t_h+1)$:

$$X=[X^{(1)},\dots ,X^{(R)}]$$

Finally, construct the block random vector of model parameters (of length u):

$$\mathbf{a}=\{{\mathbf{a}}^{(1)},\dots ,{\mathbf{a}}^{(R)}\}$$

(13)

Thus, dynamic model Equation (3) belongs to the class of randomized models with the random parameters, a, and the output noise, ξ. It can be reduced to the linear form:

$$\mathbf{v}=X\mathbf{a}+\xi$$

(14)

Structurally, this expression is analogous to linear static data model Equation (2). However, the input matrix embraces the nonlinearity and dynamics of model Equation (14).

3. Probabilistic Characteristics of RMs

Numerical characteristics of RMs are understood as the probability density functions (alternatively, probability functions) of model parameters and noise components.

In the sequel, we believe that RMs whose random components are described by probability density functions belong to the class RM-PWQ. By analogy, RMs described by the probabilities of lying within appropriate intervals will form the class RM-pwq.

3.1. RM-PWQ

The parameters of an RDM from this class and the measurement noises represent continuous random variables. They take values in intervals, where there exist probability density functions (pdfs):

• -for the random parameters of an RSM:

  $$P(\mathbf{a})=\prod _{i=1}^n{p}_i(a_i),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{a}_i\in {𝒜}_i$$

  (15)

• - for the random parameters of an RDM:

  $$P(\mathbf{a})=\prod _{h=1}^R\prod _{i=1}^{t_h}{p}_i^{(h)}(a_i^{(h)})$$

  (16)

• - for the “input” measurement noises:

  $$W(\eta )=\prod _{j=1}^s\prod _{i=1}^n{w}_{ji}({\eta }_{ji}),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\eta }_{ji}\in {ℰ}_{ji}$$

  (17)

• - for the “output” measurement noises:

  $$Q(\xi )=\prod _{j=1}^s{q}_i({\xi }_j),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\xi }_j\in {\Xi }_j$$

  (18)

The above-mentioned pdfs of the random parameters and noises should be estimated using measurements of the RM “input” and “output” and a priori information on the pdfs. Formalization of such information involves the a priori pdfs of the parameters and noises, (P⁰(a), W⁰(η) and Q⁰(ξ)), as well as the classes of pdfs.

An RM generates an ensemble, $𝒱$, of the random vectors, v, Equations (2) and (14). Measurements make up the vector, y, with measured components. Therefore, the estimation of pdfs lies in the forming of the vectors of the appropriate numerical characteristics of this ensemble. We employ moments of random components of the vector v:

$${\mathbf{m}}^{(k)}=\{ℳ(v_1^{(k)}),\dots ,ℳ(v_s^{()})\}$$

(19)

where k denotes the order of the moment,

$$ℳ(v_j^{(k)})={\int }_{\mathbf{a}\in 𝒜,\eta \in ℰ,\xi \in {\Xi }_j}{\left(F_j[X+\eta ,\mathbf{a}]+{\xi }_j\right)}^{k}P(\mathbf{a})W(\eta )Q(\xi )d\mathbf{a}d\eta d\xi ,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}$$

(20)

This paper utilizes the first moments, viz., the average values of components from the vector, v. For static model RSM-PWQ Equation (1), we obtain:

$$ℳ(\mathbf{v})=\overline{\mathbf{v}}={\int }_{(\mathbf{a}\in 𝒜,\eta \in ℰ,\xi \in \xi )}(\mathbf{F}[X+\eta ,\mathbf{a}]+\xi )\hspace{0.17em}\hspace{0.17em}P(\mathbf{a})W(\eta )Q(\xi )d\mathbf{a}d\eta d\xi$$

(21)

For the dynamic model RM-PQ Equation (14), we similarly have:

$$ℳ(\mathbf{v})=\overline{\mathbf{v}}=X{\int }_{\mathbf{a}\in 𝒜}\mathbf{a}P(\mathbf{a})d\mathbf{a}+{\int }_{\xi \in \Xi }\xi Q(\xi )d\xi$$

(22)

3.2. RM-pwq

The parameters of an RM from this class and the measurement noises turn out continuous random variables. Their belonging to an appropriate interval is characterized by some probability.

The parameters a₁, . . . , a_n possess values within the intervals 𝒜₁, . . . , 𝒜_n with the probabilities p₁, . . . , p_n, respectively. Actually, the probabilities p_i ∈ [0, 1], $i=\overline{1,n}$. By analogy, the elements, η_ji, of the “input” measurement noise matrix take values from the intervals, $ℰ$_ji, with the probabilities, w_ji ∈ [0, 1], respectively. Finally, the components, ξ_j, of the “output” measurement noises lie inside the intervals, Ξ_j, with the probabilities, q_j ∈ [0, 1], respectively. Denote by $p_i^0$, $w_{ji}^0$, $q_j^0$ the a priori values of the listed probabilities ( $j=\overline{1,s}$, $i=\overline{1,n}$).

Just like models from the class RM-PWQ, RMs belonging to the class considered to reproduce an ensemble, $𝒱$, of the random vectors, v. This is done by generating the random parameters with the probabilities, p, the elements of the “input” measurement noise matrix with the probabilities, W, and the components of the “output” measurement noises with the probabilities, q.

As a numerical characteristic for the ensemble generated by an RM of this class, select the vector of the first quasi-moments (see [5]):

$$\mathbf{a}={\mathbf{a}}^{-}+L_{\mathbf{a}}\mathbf{p},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\eta ={\eta }^{-}+L_{\eta }\otimes W,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\xi ={\xi }^{-}+L_{\xi }\mathbf{q}$$

(23)

In the previous formula, ⊗ stands for element-wise multiplication,

$$\begin{array}{l}{L}_{\mathbf{a}}=\text{diag}\left[\left(a_i^{+}-a_i^{-}\right)|i=\overline{1,n}\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{L}_{\xi }=\text{diag}\left[\left({\xi }_j^{+}-{\xi }_j^{-}\right)|j=\overline{1,s}\right]\\ L_{\eta }=\left[\left({\eta }_{ji}^{+}-{\eta }_{ji}^{-}\right)|i=\overline{1,n},\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}\right]\end{array}$$

(24)

The transform Equation (23) can be interpreted as a substitution of the random parameters and noises by their “quasi-average” values.

By applying Equation (23) to Equation (1), we obtain the following expression for the first quasi-moment of the random vector, v, for RSM Equation (1):

$$\tilde{\mathbf{v}}=\mathbf{F}\left[X+({\eta }^{-}+L_{\eta }\otimes W),{\mathbf{a}}^{-}+L_{\mathbf{a}}\mathbf{p}\right]+{\xi }^{-}+L_{\xi }\mathbf{q}$$

(25)

In the case of RDMs, the first quasi-moment formula of the random vector, v, Equation (10), acquires the form:

$$\tilde{\mathbf{v}}=X\mathbf{a}+L_{\mathbf{ap}}+{\xi }^{-}+L_{\xi }\mathbf{q}$$

(26)

Concluding this section, we emphasize a relevant aspect. For the class RM-PWQ, it is necessary to estimate probability density functions. For the class RM-pwq, one has to estimate vectors characterizing probability distributions.

4. Principles of Entropy-Robust Estimation

We propose to introduce a likelihood functional for the estimation of probability density functions for RM-PWQ (RM-PQ). So long as the model parameters and measurement noises are independent, their joint distribution function takes the form:

$$\Phi (\mathbf{a},\eta ,\xi )=P(\mathbf{a})W(\eta )Q(\xi )$$

(27)

Suppose that we know the a priori density functions, P⁰(a), W⁰(η), Q⁰(ξ).

Following [4], specify the log-likelihood ratio (LLR) by:

$$\phi (\mathbf{a},\eta ,\xi )=\text{ln}\frac{P(\mathbf{a})}{P^0(\mathbf{a})}+\text{ln}\frac{W(\eta )}{W^0(\eta )}+\text{ln}\frac{Q(\xi )}{Q^0(\xi )}$$

(28)

Hence, the LLR represents a nonrandom function of random arguments.

The probabilities, p, W, q, of certain events (e.g., entering appropriate intervals by the parameters, “input” and “output” noises) being estimated, one should define the LLR by:

$$\phi (\mathbf{p},W,\mathbf{q})=\text{ln}\frac{\mathbf{p}}{{\mathbf{p}}^0}+\text{ln}\frac{W}{W^0}+\text{ln}\frac{\mathbf{q}}{{\mathbf{q}}^0}$$

(29)

where p⁰, W⁰ and q⁰ are a priori probabilities.

Now, introduce the likelihood functional (the generalized informational Boltzmann entropy functional), $ℒ$ (see Equation (29)), in the form:

$$\begin{array}{ll}\hfill & ℒ[P(\mathbf{a},W(\eta ),Q(\xi )]=-\mathscr{H}[P(\mathbf{a}),W(\eta ),Q(\xi )]=\hfill \\ =\hfill & {\int }_{\mathbf{a}\in 𝒜}P(\mathbf{a})\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{P(\mathbf{a})}{P^0(\mathbf{a})}d\mathbf{a}+{\int }_{\eta \in ℰ}W(\eta )\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{W(\eta )}{W^0(\eta )}d\eta +\hfill \\ +\hfill & {\int }_{\xi \in \Xi }Q(\xi )\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{Q(\xi )}{Q^0(\xi )}d\xi \hfill \end{array}$$

(30)

Obviously (see Equation (28)), the likelihood functional with the minus mark represents the generalized Boltzmann entropy functional. This function has numerous interpretations. In particular, it is treated as the “distance” between pdfs [16], as an uncertainty measure [12,13,17] or as a robustness measure [6] (the degree of invariance of the pdfs P(a), W(η), Q(ξ), with respect to observations). In this paper, we involve the last interpretation of the entropy functional to construct estimates of the pdfs.

Define the likelihood function (the generalized informational Boltzmann entropy function) by (see [18]):

$$\begin{array}{r}\hfill H(\mathbf{p},W,\mathbf{q})=-\sum _{i=1}^n{p}_i\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{p_i}{p_i^0}-\sum _{j=1}^s\sum _{i=1}^n{w}_{ji}\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{w_{ji}}{w_{ji}^0}-\\ \hfill -\sum _{j=1}^s{q}_j\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{q_j}{q_j^0}\end{array}$$

(31)

The estimation quality for pdfs or probability vector components is characterized by the maximal value of the generalized informational Boltzmann entropy functional (or function). In the sequel, we distinguish between two types of estimates.

• The ${𝒮}_{\mathit{PWQ}}^k$ -robust entropy estimate (the ${𝒮}_{PQ}^k$-robust entropy estimate) of the pdfs of the parameters and measurement noises results from solving the problem:

  $$\mathscr{H}[P(\mathbf{a},W(\eta ),Q(\xi )]\Rightarrow \text{max}$$

  (32)
  provided that:

  • the probability density functions belong to the class:

    $$𝒮=𝒫\cup 𝒲\cup 𝒬$$

    (33)
    where $𝒫$, $𝒲$ and $𝒬$ denote the classes of the pdfs of the parameters, “input” and “output” noises, respectively;

  • the balance condition holds true for the degree (1/k) of the k-th moment, v, and the measured vector, y. For RSM-PWQ, this condition becomes:

    $${\left(ℳ\left\{{\mathbf{F}}^{(k)}[(X+\eta ),\mathbf{a}]+{\xi }^{(k)}\right\}\right)}^{(1/k)}=\mathbf{y}$$

    (34)
    For RDM-PQ, the balance condition takes the form:
    $${\left(ℳ\{X\mathbf{a}+\xi \}\right)}^{(1/k)}=\mathbf{y}$$

    (35)

• The ${𝒮}_{\mathit{pwq}}^{\tilde{1}}$ -robust entropy estimate of the probability distributions (pds) of the model parameters and measurement noise components follow from solving the problem:

  $$H(\mathbf{p},W,\mathbf{q})\Rightarrow \text{max}$$

  (36)
  provided that:

  • the probability distributions belong to the class:

    $$𝕊=\mathbb{P}\cup 𝕎\cup \mathbb{Q}$$

    (37)
    where ℙ, 𝕎 and ℚ designate the classes of the pds of the parameters, “input” and “output” noises, respectively;

  • the balance condition holds true for the quasi-average vector, ṽ, and the measured vector, y:

    $$\begin{array}{r}\hfill \mathbf{F}\left[X+({\eta }^{-}+L_{\eta }W),({\mathbf{a}}^{-}+L_{\mathbf{a}}\mathbf{p})\right]+\\ \hfill +{\xi }^{-}+L_{\xi }\mathbf{q}=\mathbf{y}\end{array}$$

    (38)

5. Structural Properties of ${𝒮}_{PQ}^1$-Estimates

5.1. Power-Type RSM-PQ

Consider the subclass of power-type RSM-PQ with the following nonlinearities:

$$v(t)=\sum _{h=1}^R\sum _{i=1}^n{a}_i^h{x}_i^{(h)}(t)+\xi (t)$$

(39)

The vector of parameters, a, turns out to be random with independent components (a₁, . . . , a_n) possessing values in the intervals ${𝒜}_i=[a_i^{-},a_i^{+}]$, ( $i=\overline{1,n}$) with the pdfs p₁(a₁), . . . , p_n(a_n).

The “input” and “output” are observed at instants t₁, . . . , t_s. The measured “input” is characterized by the set of R matrices:

$$X^{(h)}=\left(\begin{array}{ccc}{x}_1^{(h)}(t_1)& \cdots & x_n^{(h)}(t_1)\\ \cdots & \cdots & \cdots \\ x_1^{(h)}(t_s)& \cdots & x_n^{(h)}(t_s)\end{array}\right)=\left(\begin{array}{ccc}{x}_{11}^{(h)}& \cdots & x_{1,n}^{(h)}\\ \cdots & \cdots & \cdots \\ x_{s,1}^{(h)}& \cdots & x_{s,n}^{(h)}\end{array}\right),\hspace{0.17em}\hspace{0.17em}h\in [1,R]$$

(40)

Additionally, the observed “output” of the RM is characterized by the random vector v = {v(t₁), . . . , v(t_s)}.

Therefore, using observation results, we rewrite the RSM-PQ as:

$$\mathbf{v}=\sum _{h=1}^R{X}^{(h)}{\mathbf{a}}^{(h)}+\xi$$

(41)

Here, ${\mathbf{a}}^{(h)}=\{a_1^{(h)},\dots ,a_n^h\}$, ξ = {ξ(t₁), . . . , ξ(t_s)} = {ξ₁, . . . , ξ_s} denotes the “output” measurement noise vector. It has independent components belonging to the intervals ${\Xi }_j=\left[{\xi }_j^{-},{\xi }_j^{+}\right]$, ( $j=\overline{1,s}$) with the pdfs q₁(ξ₁), . . . , q_s(ξ_s). Suppose that a priori information is absent, i.e., P⁰(a) = Q⁰(ξ) = const.

To proceed, we analyze the ${𝒮}_{PQ}^1$-robust entropy estimate of the pdfs P(a) = {p₁(a₁), . . . , p_n(a_n)} and Q(ξ) = {q₁(ξ₁), . . . , q_s(ξ_s)}. Moreover, we study some structural properties of the estimate.

Revert to problem Equations (32)–(34). Reexpress these as:

$$\begin{array}{lll}\mathscr{H}[P(\mathbf{a}),Q(\xi )]\hfill & =\hfill & -\sum _{i=1}^n{\int }_{a_i\in {𝒜}_i}{p}_i(a_i)\hspace{0.17em}\text{ln}\hspace{0.17em}{p}_i(a_i)d{a}_i-\hfill \\ \hfill & -\hfill & \sum _{j=1}^s{\int }_{{\xi }_j\in {\Xi }_j}{q}_i({\xi }_j)\hspace{0.17em}\text{ln}\hspace{0.17em}{q}_j({\xi }_j)d{\xi }_j.\Rightarrow \text{max}\hfill \end{array}$$

(42)

subject to the constraints:

$$\begin{array}{lll}{D}_i[p_i(a_i)]\hfill & =\hfill & 1-{\int }_{a_i\in {𝒜}_i}{p}_i(a_i)d{a}_i=0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=\overline{1,n}\hfill \\ T_j[q_j({\xi }_j)]\hfill & =\hfill & 1-{\int }_{{\xi }_j\in {\Xi }_j}{q}_j({\xi }_j)d{\xi }_j=0,\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}\hfill \end{array}$$

(43)

$$\begin{array}{lll}{\Phi }_j[p(\mathbf{a}),q(\xi )]\hfill & =\hfill & \sum _{h=1}^R\sum _{i=1}^n{x}_{ji}^{(h)}{\int }_{a_i\in {𝒜}_i}{a}_i^h{p}_i(a_i)d{a}_i+\hfill \\ \hfill & +\hfill & {\int }_{{\xi }_j\in {\Xi }_j}{\xi }_j{q}_j({\xi }_j)d{\xi }_j-y_j=0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}\hfill \end{array}$$

(44)

The structural properties of the ${𝒮}_{PQ}^1$-estimate are understood as a certain class of the robust family of probability density functions of the model parameters and noise. The procedure of ${𝒮}_{PQ}^1$-estimation consists in solving problem Equations (32)–(34) for RSM-PWQ (alternatively, problem Equations (32), (33) and (35) for RDM-PQ). Both problems belong to the class of functional entropy-linear programming problems with equality constraints. The standard approach to such variational problems (see [19]) naturally brings the following result. The entropy-optimal pdfs (in the class of continuously differentiable functions) of the parameters and noise components are defined by:

$$\begin{array}{l}{p}_i^{*}(a_i)~{\pi }_i\text{exp}\left(-\sum _{h=1}^R{\alpha }_{ih}{a}_i^h\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i\in [1,n]\\ q_j^{*}({\xi }_j)~{\kappa }_j\text{exp}(-{\theta }_j{\xi }_j),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=[1,s])\end{array}$$

(45)

where:

$${\alpha }_{ih}=\sum _{j=1}^s{\theta }_j{x}^{{(h)}_{ji}},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\pi }_i=\text{exp}(-1-{\gamma }_i);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\kappa }_j=\text{exp}(-1-{\omega }_j)$$

(46)

In the last formulas, the Lagrange multipliers, γ_i, ω_j, correspond to constraint Equation (43), while the ones of θ_j correspond to constraint Equation (44).

Figures 1 and 4 demonstrate some examples of estimating the pdfs of the parameters. For a linear RM, the ${𝒮}_{PQ}^1$-estimate always represents an exponential function (see Figure 1). The “input” and “output” measurements do not affect the structure of the estimate (yet, they change the shape of the functions).

For nonlinear RMs, the structure of the ${𝒮}_{PQ}^1$-estimates varies depending on the “input” and “output” measurements. For instance, Figures 2 and 4 show the estimates of the pdfs for a quadratic, quadratic-linear and cubic RM-PQ.

5.2. Power-Type RM-PQ

Address the linear form of RM-PQ Equation (26). In this case, the problem of ${𝒮}_{PQ}^1$-estimation of the pdfs for parameter Equation (16) and noise Equation (18) acquires the form:

$$\mathscr{H}[P(\mathbf{a}),Q(\xi )]=-{\int }_{\mathbf{a}\in 𝒜}P(\mathbf{a})\hspace{0.17em}\mathit{ln}\hspace{0.17em}P(\mathbf{a})d\mathbf{a}-{\int }_{\xi \in Xi}Q(\xi )\hspace{0.17em}\mathit{ln}\hspace{0.17em}Q(\xi )d\xi \Rightarrow \text{max}$$

(47)

subject to the constraints:

$$\begin{array}{l}D[P(\mathbf{a})]=1-{\int }_{\mathbf{a}\in 𝒜}P(\mathbf{a})d\mathbf{a}=0\\ T[Q(\xi )]=1-{\int }_{\xi \in \Xi }Q(\xi )d\xi =0\end{array}$$

(48)

$$\begin{array}{lll}\overline{\Phi }[P(\mathbf{a}),Q(\xi )]\hfill & =\hfill & X{\int }_{\mathbf{a}\in 𝒜}\mathbf{a}P(\mathbf{a})d\mathbf{a}+\hfill \\ \hfill & +\hfill & {\int }_{\xi \in \Xi }\xi Q(\xi )d\xi -\mathbf{y}=\mathbf{0}\hfill \end{array}$$

(49)

The vector, Φ̄[P(a), Q(ξ)], has the length of (s + 1) (the number of measurements).

Problem Equations (47)–(49) represent a modification of functional entropy-linear programming problem Equations (42)–(44), which involves the linear data model. To obtain the solution, again, adopt the standard technique for such variational problems [19]. This yields the following entropy-optimal pdfs of the parameters and noises:

$$\tilde{P}(\mathbf{a})=\text{exp}(-1-\gamma -\langle \theta ,X\mathbf{a}\rangle ),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tilde{Q}(\xi )=\text{exp}(-1-\omega -\theta \otimes \xi )$$

(50)

Here, γ, ω designate the Lagrange multipliers associated with constraint Equation (48). In addition, the vector, θ (whose length equals (s + 1)), comprises the Lagrange multipliers for constraint Equation (49).

Formula (50) indicates that the entropy-optimal pdfs of the parameters and noises belong to the exponential class, with the parameters representing the Lagrange multipliers. The latter quantities meet a system of equations generated by constraint Equations (48),(49).

6. Normalized and Interval ${𝒮}_{pq}^{\tilde{1}}$-Estimates for Linear RSM-pq

Consider the subclass of linear RSM-pq with accurate “input” measurements:

$$\tilde{\mathbf{v}}=X{L}_{\mathbf{a}}\mathbf{p}+L_{\xi }\mathbf{q}+\mathbf{K}({\mathbf{a}}^{-}{\xi }^{-})$$

(51)

where:

$$\mathbf{K}({\mathbf{a}}^{-},{\xi }^{-})=X{\mathbf{a}}^{-}+{\xi }^{-}$$

(52)

Denote by p⁰ and q⁰ the a priori probabilities of the parameters and noises, respectively.

Let us study the ${𝒮}_{pq}^{\tilde{1}}$-estimates for some classes of the vectors, p, q.

Problem 1. Normalized probabilities: the${𝒮}_{pq}^{\tilde{1}}$-estimate results from solving the problem:

$$H(\mathbf{p},\mathbf{q}=-\sum _{i=1}^n{p}_i\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{p_i}{p_i^0}--\sum _{j=1}^s{q}_j\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{q_j}{q_j^0}\Rightarrow \text{max}$$

(53)

subject to the probability normalization conditions:

$$\sum _{i=1}^n{p}_i=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sum _{j=1}^s{q}_j=1$$

(54)

and the balance conditions for the real observations, y_j, with the first quasi-moment of the output of RSM-pq:

$$\sum _{i=1}^n{x}_{ji}{L}_{\mathbf{a}}^i{p}_i+L_{\xi }^j{q}_j+K_j=y_j,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j\in [1,s],s<n$$

(55)

This problem belongs to the class of entropy-linear programming problems [20]. Its solution employs the necessary conditions of optimality in terms of the Lagrange function:

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}L(\mathbf{p},\mathbf{q},\lambda ,\epsilon ,\theta )=H(\mathbf{p},\mathbf{q})+\lambda \left(1-\sum _{i=1}^n{p}_i\right)\\ +\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\epsilon \left(1-\sum _{j=1}^s{q}_j\right)+\sum _{j=1}^s{\theta }_j\left[y_j-\sum _{i=1}^n{x}_{ji}{L}_{\mathbf{a}}^i{p}_i-L_{\xi }^j{q}_j-K_j\right]\end{array}$$

(56)

Consequently, we derive a system of equations determining the entropy-optimal probabilities, p^*, q^*:

$$\begin{array}{lll}0\le p_i^{*}(\theta )\hfill & =\hfill & \frac{p_i^0\text{exp}\left(-{\sum }_{j=1}^s{\theta }_j{x}_{ji}{L}_{\mathbf{a}}^i\right)}{{\sum }_{i=1}^n{p}_i^0\text{exp}\left(-{\sum }_{j=1}^s{\theta }_j{x}_{ji}{L}_{\mathbf{a}}^i\right)}\le 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=\overline{1,n}\hfill \\ 0\le q_j^{*}(\theta )\hfill & =\hfill & \frac{q_j^0\text{exp}\left(-{\theta }_j{L}_{\xi }^j\right)}{{\sum }_{j=1}^s{q}_j^0\text{exp}\left(-{\theta }_j{L}_{\xi }^j\right)}\le 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}\hfill \end{array}$$

(57)

and the corresponding Lagrange multipliers θ₁, . . . , θ_s:

$${\Phi }_j(\theta )\frac{1}{y_j-K_j}\sum _{i=1}^n{x}_{ji}{L}_{\mathbf{a}}^i{p}_i^{*}(\theta )+L_{\xi }^j{q}_j^{*}(\theta )=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}$$

(58)

Problem 2. The interval probabilities: the${𝒮}_{pq}^{\tilde{1}}$-estimate satisfies the following maximization problem for the generalized informational Fermi–Dirac entropy. The application of the generalized informational Fermi–Dirac entropy provides estimates lying in appropriate intervals [18]:

$$\begin{array}{lll}H(\mathbf{p},\mathbf{q})\hfill & =\hfill & -\sum _{i=1}^n\left(p_i\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{p_i}{{\phi }_i^0}+(1-p_i)\hspace{0.17em}\text{ln}\hspace{0.17em}(1-p_i)\right)-\hfill \\ \hfill & -\hfill & \sum _{j=1}^s\left(q_j\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{q_j}{{\varphi }_j^0}+(1-q_j)\hspace{0.17em}\text{ln}\hspace{0.17em}(1-q_j)\right)\Rightarrow \text{max}\hfill \end{array}$$

(59)

subject to the interval probability constraints:

$$0\le p_i\le 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=\overline{1,n};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0\le q_j\le 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j\in [1,s],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}s<n$$

(60)

and the balance conditions for the real observations, y_j, with the first quasi-moment of the output of RM-pq:

$$\sum _{i=1}^n{x}_{ji}{L}_{\mathbf{a}}^i{p}_i+L_{\xi }^j{q}_j+K_j=y_j,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}$$

(61)

where:

$${\phi }_i^0=\frac{p_i^0}{1-p_i^0},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\varphi }_j^0=\frac{q_j^0}{1-q_j^0}$$

(62)

Lagrange’s method of multipliers gives the following system of equations for the estimates of the probabilities, p, q:

$$\begin{array}{lll}0\le p_i^{*}(\theta )\hfill & =\hfill & \frac{p_i^0}{p^0+(1-p_i^0)\text{exp}\left({\sum }_{j=1}^s{\theta }_j{x}_{ji}{L}_{\mathbf{a}}^i\right)}\le 1,\hspace{0.17em}\hspace{0.17em}i=\overline{1,n}\hfill \\ 0\le q_j^{*}(\theta )\hfill & =\hfill & \frac{q_j^0}{q_j^0+(1-q_j^0)\text{exp}(-{\theta }_j{L}_{\xi }^j)}\le 1,\hspace{0.17em}\hspace{0.17em}j=\overline{1,s}\hfill \end{array}$$

(63)

and the Lagrange multipliers θ₁, . . . , θ_s:

$${\Phi }_j(\theta )=\frac{1}{y_j-K_j}\sum _{i=1}^n{\theta }_j{x}_{ji}{p}_i^{*}(\theta )+L_{\xi }^j{q}_j^{*}(\theta )=1,j=\overline{1,s}$$

(64)

So long as the estimates of p and q are expressed analytically through the Lagrange multipliers, θ, one should only solve the system of Equation (64). For this, it is possible to use the following multiplicative algorithm [20] for the exponential Lagrange multipliers z_j = exp(−θ_j), j ∈ [1, s]:

$$z_j^{k+1}=z_j^k{\Phi }_j(z^k),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em},z_j^0>0,\hspace{0.17em}\hspace{0.17em}j\in [1,s]$$

(65)

Under certain conditions, the estimates generated by Problems 1–2 differ in the values of some entropy (e.g., generalized Boltzmann entropy Equation (53)). Let us introduce the following notation:

• the entropy:

  $$H(\mathbf{p},\mathbf{q})=H(\mathbf{g}),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathbf{g}=(\mathbf{p},\mathbf{q})\in R_{+}^{n+s}$$

  (66)

• the estimate, ${\mathbf{g}}_1^{*}$, obtained for normalized probabilities (Problem 1), and the estimate, ${\mathbf{g}}_2^{*}$, obtained for interval probabilities (Problem 2);

• the absolute maximum of the entropy ĝ = arg max H(g);

• the set:

  $$\Pi =\left\{\mathbf{g}:\mathbf{0}\le \mathbf{g}\le \mathbf{1}\right\}\supset \tilde{\Pi }=\{\mathbf{g}:\langle \mathbf{p},\mathbf{1}\rangle ,\hspace{0.17em}\hspace{0.17em}\langle \mathbf{q},\mathbf{1}\rangle \}$$

  (67)

Theorem Assume that$\widehat{\mathit{\text{g}}}\in \left(R_{+}^{(n+s)}\backslash \tilde{\Pi }\right)$.

Then:

$$H({\mathbf{g}}_1^{*})\le H({\mathbf{g}}_2^{*})$$

and the equality takes place iff${\mathit{g}}_1^{*}=\widehat{\mathit{g}}$.

Proof. Entropy Equation (53) represents a strictly concave function with a unique absolute maximum at the point, ĝ. The value of the entropy at an arbitrary point, g, depends on the distance to the absolute maximum point. Denote by $\varrho \left(\widehat{\mathbf{g}},{\mathbf{g}}_1^{*}\right)$ and $\varrho \left(\widehat{\mathbf{g}},{\mathbf{g}}_2^{*}\right)$ the distances between the absolute maximum point, ĝ, and the points corresponding to the optimal estimates in Problems (53)–(55) and (59)–(62), respectively. By virtue of the premise above, expression Equation (67) and strict concavity, we have:

$$\varrho \left(\widehat{\mathbf{g}},{\mathbf{g}}_1^{*}\right)\ge \varrho \left(\widehat{\mathbf{g}},{\mathbf{g}}_2^{*}\right)$$

(68)

These distances coincide only if ${\mathbf{g}}_1^{*}=\widehat{\mathbf{g}}$.

7. Examples

7.1. Entropy Estimation of the Parameters of Linear RSM-pq under a Small Amount of Data

Consider a linear RM with five random parameter Equations (51) belonging to the same intervals 𝒜 = [0, 10] and with the noise vector ξ = {ξ₁, ξ₂}, whose components lie within the intervals Ξ₁ = [−3, 3] and Ξ₂ = [−6, 6]. There are two “output” measurements, y = {y₁, y₂}.

The reference model has fixed parameters a⁰ = {1, 2, 2, 4, 1}. The deviation from these values is described by the relative square error:

$$\epsilon =\frac{\Vert {\mathbf{a}}^0-\mathbf{a}\Vert }{\Vert {\mathbf{a}}^0\Vert +\Vert \mathbf{a}\Vert }$$

The “input” measurement matrix makes up:

$$X=\left(\begin{array}{lllll}1.8\hfill & 2.1\hfill & 3.3\hfill & 2.0\hfill & 1.5\hfill \\ 4.1\hfill & 3.8\hfill & 3.0\hfill & 2.8\hfill & 1.9\hfill \end{array}\right)$$

The “output” measurement vector (distorted by noises from appropriate intervals) takes the form:

$$\mathbf{y}=\{21.1;32.8\}$$

The first quasi-moments for the parameters and noise are defined by:

$$a_i=10p_i,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=\overline{1,5};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\xi }_1=-3+6q_1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\xi }_2=-6+12q_2$$

Substituting the last equalities into Equation (51) yields the following RM-pq:

$$T\mathbf{p}+L\mathbf{q}=\mathbf{1}$$

with the matrix:

$$T=X{L}_{\mathbf{a}}=\left(\begin{array}{lllll}0.75\hfill & 0.87\hfill & 1.37\hfill & 0.83\hfill & 0.62\hfill \\ 1.06\hfill & 0.98\hfill & 0.77\hfill & 0.72\hfill & 0.45\hfill \end{array}\right)$$

the matrix:

$$L=L_{\xi }=\left(\begin{array}{cc}0.25& 0\\ 0& 0.31\end{array}\right)$$

and the vector 1 = {1, 1}.

As a priori information, we have selected three scenarios for the a priori probabilities (see

Table 1. The a priori probabilities, $p_i^0$, i ∈ [1, 5].

Scenario/i	1	2	3	4	5
A	1.0	1.0	1.0	1.0	1.0
B	0.1	0.2	0.3	0.3	0.1
C	0.3	0.4	0.1	0.05	0.15

and

Table 2. The a priori probabilities, $q_1^0$, $q_2^0$.

Scenario/i	1	2
D	0.2	0.8
E	1.0	1.0

).

Scenario AE corresponds to uniform a priori distributions of the parameters and noise. Next, Scenario BD reflects nonuniform a priori distributions of the parameters and noise. Finally, CE is the combined scenario (the parameters possess nonidentical a priori distributions, whereas the noise has uniform a priori distributions).

We study the ${𝒮}_{pq}^{\tilde{1}}$-estimates corresponding to Problems 1–2.

Problem 1. Normalized probabilities.

$$\begin{array}{c}H(\mathbf{p},\mathbf{q})=-\sum _{i=1}^5\left(p_i\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{p_i}{p_i^0}\right)-\sum _{j=1}^2\left(q_j\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{q_j}{q_j^0}\right)\Rightarrow \text{max}\\ \sum _{i=1}^5{p}_i=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{p}_i>0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sum _{j=1}^2{q}_j=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{q}_j>0\\ 0.75p_1+0.87p_2+1.37p_3+0.83p_4+0.62p_5+0.25q_1=1\\ 1.06q_1+0.98p_2+0.77p_3+0.72p_4+0.45p_5+0.31q_1=1\end{array}$$

Problem 2. Interval probabilities.

$$\begin{array}{c}H(\mathbf{p},\mathbf{q})=-\sum _{i=1}^5\left(p_i\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{p_i}{p_i^0}\right)-\sum _{j=1}^2\left(q_j\hspace{0.17em}\text{ln}\hspace{0.17em}\frac{q_j}{q_j^0}\right)\Rightarrow \text{max}\\ 0\le \mathbf{p}\le 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0\le \mathbf{q}\le 1\\ 0.75p_1+0.87p_2+1.37p_3+0.83p_4+0.62p_5+0.25q_1=1\\ 1.06p_1+0.98p_2+0.77p_3+0.72p_4+0.45p_5+0.31q_1=1\end{array}$$

The entropy, H, attains the absolute maximum at the point ${\widehat{p}}_i=0.36p_i^0$; ${\widehat{q}}_j=0.36q_j^0$; i ∈ [1, 5], j ∈ [1, 2].

Table 3. The components, p̂_i, i ∈ [1, 5].

Scenario/i	1	2	3	4	5
A	0.36	0.36	0.36	0.36	0.36
B	0.036	0.072	0.108	0.108	0.036
C	0.108	0.144	0.036	0.018	0.054

and

Table 4. The components, q̂₁, q̂₂.

Scenario/i	1	2
D	0.072	0.288
E	0.36	0.36

present the coordinates of the absolute maximum point for the above scenarios of the a priori probabilities (see Tables 1 and 2).

Solutions to Problem 1 can be found in

Table 5. The ${𝒮}_{pq}^{\tilde{1}}$-estimates of the probabilities, p, q, in Problem 1.

Scenario	${\mathit{p}}_{\mathbf{1}}^{*}$	${\mathit{p}}_{\mathbf{2}}^{*}$	${\mathit{p}}_{\mathbf{3}}^{*}$	${\mathit{p}}_{\mathbf{4}}^{*}$	${\mathit{p}}_{\mathbf{5}}^{*}$	${\mathit{q}}_{\mathbf{1}}^{*}$	${\mathit{q}}_{\mathbf{2}}^{*}$	H*
AD	0.14	0.17	0.28	0.20	0.21	0.28	0.72	1.56
BD	0.08	0.16	0.26	0.34	0.16	0.25	0.75	−0.03
CD	0.17	0.31	0.22	0.06	0.24	0.40	0.60	−0.23
AE	0.23	0.22	0.19	0.19	0.16	0.45	0.55	2.29
BE	0.14	0.24	0.18	0.32	0.12	0.39	0.61	0.63
CE	0.26	0.0.38	0.13	0.05	0.18	0.58	0.42	0.67

and

Table 6. The first quasi-moments of the parameters and noises in Problem 1.

Scenario	${\mathit{a}}_{\mathbf{1}}^{*}$	${\mathit{a}}_{\mathbf{2}}^{*}$	${\mathit{a}}_{\mathbf{3}}^{*}$	${\mathit{a}}_{\mathbf{4}}^{*}$	${\mathit{a}}_{\mathbf{5}}^{*}$	${\mathit{\xi }}_{\mathbf{1}}^{*}$	${\mathit{\xi }}_{\mathbf{2}}^{*}$	ε
AD	1.43	1.66	2.78	2.01	2.13	−1.33	2.65	0.13
BD	0.78	1.64	2.62	3.37	1.58	−1.52	3.04	0.03
CD	1.67	3.08	2.25	0.63	2.37	−0.62	1.23	0.30
AE	2.35	2.22	1.91	1.88	1.64	−0.31	0.62	0.15
BE	1.41	2.36	1.84	3.21	1.18	−0.65	1.30	0.02
CE	2.56	3.76	1.30	0.56	1.82	0.46	−0.91	0.36

.

Solutions to Problem 2 are shown by

Table 7. The ${𝒮}_{pq}^{\tilde{1}}$-estimates of the probabilities, p, q, in Problem 2.

Scenario	${\mathit{p}}_{\mathbf{1}}^{*}$	${\mathit{p}}_{\mathbf{2}}^{*}$	${\mathit{p}}_{\mathbf{3}}^{*}$	${\mathit{p}}_{\mathbf{4}}^{*}$	${\mathit{p}}_{\mathbf{5}}^{*}$	${\mathit{q}}_{\mathbf{1}}^{*}$	${\mathit{q}}_{\mathbf{2}}^{*}$	H*
AD	0.24	0.22	0.18	0.23	0.26	0.0.07	0.0.29	2.05
BD	0.15	0.26	0.27	0.27	0.07	0.07	0.46	0.27
CD	0.23	0.39	0.24	0.05	0.12	0.11	0.26	0.23
AE	0.25	0.22	0.15	0.22	0.25	0.30	0.39	2.37
BE	0.15	0.26	0.23	0.26	0.06	0.34	0.60	0.67
CE	0.26	0.40	0.17	0.05	0.11	0.48	0.37	0.71

and

Table 8. The 1̃-quasi-moments of the parameters and noise in Problem 2.

Scenario	${\mathit{a}}_{\mathbf{1}}^{*}$	${\mathit{a}}_{\mathbf{2}}^{*}$	${\mathit{a}}_{\mathbf{3}}^{*}$	${\mathit{a}}_{\mathbf{4}}^{*}$	${\mathit{a}}_{\mathbf{5}}^{*}$	${\mathit{\xi }}_{\mathbf{1}}^{*}$	${\mathit{\xi }}_{\mathbf{2}}^{*}$	ε
AD	2.37	2.26	1.83	2.35	2.64	−2.61	−2.57	0.14
BD	1.50	2.64	2.71	2.74	0.68	−2.58	−0.47	0.06
CD	2.34	3.95	2.44	0.52	1.23	−2.34	−2.92	0.33
AE	2.50	2.24	1.50	2.21	2.50	−1.17	−1.35	0.16
BE	1.52	2.58	2.35	2.60	0.66	−0.97	1.21	0.06
CE	2.62	3.98	1.68	0.47	1.11	−0.12	−1.51	0.36

.

The comparative analysis of these computations draws the following conclusions:

• the estimates of the parameters derived for interval probabilities have a larger constrained maximum of the entropy than the ones derived for normalized probabilities (see Theorem 2);

• the reference parameters and a priori probabilities appear interconnected: their “successful” choice (Scenario BD) leads to “better” approximation of the reference parameters in terms of the relative squared error than in the case of an “unsuccessful” choice (Scenario CD).

7.2. The Entropy Estimation of the pdfof Second-Order RDM-PQ Characteristics

Consider the following RDM-PQ:

$$\begin{array}{c}v[k]=\sum _{n_1=0}^2{w}^{(1)}[n_1]x[k-n_1]+\\ +\sum _{n_1,n_2=0}^2{w}^{(2)}[n_1,n_2]x[k-n_1]x[k-n_2]+\xi [k],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k\ge 2\end{array}$$

with the random impulse response:

$$\begin{array}{cc}{w}^{(1)}[n_1]=w^{(2)}[n_1,n_2]=0,& \text{if}\hspace{0.17em}\text{any}\hspace{0.17em}\text{of} \text{indexes}\hspace{0.17em}(n_1,n_2)>2\\ w_2[n_1,n_2]=w_2[n_2,n_1]& \end{array}$$

Therefore, the dynamic RM-PQ in question incorporates nine parameters:

$$\begin{array}{l}{a}_0=a_0^{(1)}=w^{(1)}[0],\hspace{0.17em}\hspace{0.17em}{a}_1=a_1^{(1)}=w^{(1)}[1],\hspace{0.17em}\hspace{0.17em}{a}_2=a_2^{(1)}=w^{(1)}[2]\\ a_3=a_0^{(2)}=w^{(2)}[0,0],\hspace{0.17em}\hspace{0.17em}{a}_4=a_1^{(2)}=w^{(2)}[0,1]+w^{(2)}[1,0]\\ a_5=a_2^{(2)}=w^{(2)}[0,2]+w^{(2)}[2,0],\hspace{0.17em}\hspace{0.17em}{a}_6=a_3^{(2)}=w^{(2)}[1,1]\\ a_7=a_4^{(2)}=w^{(2)}[1,2]+w^{(2)}[2,1],\hspace{0.17em}\hspace{0.17em}{a}_8=a_5^{(2)}=w^{(2)}[2,2]\end{array}$$

The values of constants in Equation (8) are combined in

Table 9. The values of ${\beta }_{+}^{(1,2)}$, β^(1,2)−, ${\alpha }_{+}^{(1,2)}$ and ${\alpha }_{-}^{(1,2)}$.

${\beta }_{+}^{(\mathbf{1})}$	${\beta }_{+}^{(\mathbf{2})}$	${\beta }_{-}^{(\mathbf{1})}$	${\beta }_{-}^{(\mathbf{2})}$	${\alpha }_{+}^{(\mathbf{1})}$	${\alpha }_{+}^{(\mathbf{2})}$	${\alpha }_{-}^{(\mathbf{1})}$	${\alpha }_{-}^{(\mathbf{2})}$
1.0	2.0	0.5	1.0	0.08	0.08	0.08	0.08

.

Table 10. The intervals for the parameters.

j	1	2	3	4	5	6	7	8	9
$a_j^{-}$	0.50	0.46	0.42	1.00	0.92	0.85	0.85	0.79	0.72
$a_j^{+}$	1.00	0.92	0.85	2.00	1.84	1.70	1.70	1.58	1.44

demonstrates interval Equation (6) covering the components of the vector, a.

We have two measurements of the “input” and “output.” Construct the blocks X⁽¹⁾, X⁽²⁾ Equation (11):

$$\begin{array}{c}{X}^{(1)}=\left(\begin{array}{ccc}x[2]& x[1]& x[0]\\ x[3]& x[2]& x[1]\end{array}\right)=\left(\begin{array}{ccc}{x}_{00}& x_{01}& x_{02}\\ x_{10}& x_{11}& x_{12}\end{array}\right)\\ X^{(2)}=\left(\begin{array}{cccccc}{x}^2[2]& x[2]x[1]& x[2]x[0]& x^2[1]& x[1]x[0]& x^2[0]\\ x^2[3]& x[3]x[2]& x[3]x[1]& x^2[2]& x[2]x[1]& x^2[1]\end{array}\right)=\\ =\left(\begin{array}{cccccc}{x}_{00}& x_{01}& x_{02}& x_{03}& x_{04}& x_{05}\\ x_{10}& x_{11}& x_{12}& x_{13}& x_{14}& x_{15}\end{array}\right)\end{array}$$

The matrix X = [X⁽¹⁾, X⁽²⁾] has the form:

$$X=\left(\begin{array}{ccccccccc}3.9& 1.9& 2.8& 0.9& 1.6& 5.2& 3.6& 1.9& 4.2\\ 9.3& 3.9& 1.9& 3.8& 8.5& 4.9& 0.9& 1.6& 2.6\end{array}\right)$$

The noise components become ξ[0] = ξ₀ ∈ Ξ₀ = [−3, 3]; ξ[1] = ξ₁ ∈ Ξ₁ = [−6, 6]. Additionally, the measured values of the “output” are given by y[2] = y₀ = 18.51, y[3] = y₁ = 43.36.

The ${𝒮}_{PQ}^1$-estimate of the pdf of the parameters, $p_j^{*}(a_j)$, j ∈ [0, 8], and the pdfs of the noise, $q_0^{*}({\xi }_0)$, $q_1^{*}({\xi }_1)$, is defined by:

$$\begin{array}{l}{p}_j^{*}(a_j)=\text{exp}\left[-1-{\gamma }_j-{\sum }_{l=0}^1{\theta }_l{x}_{lj}{a}_j\right],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j\in [0,8]\\ q_i^{*}({\xi }_i)=\text{exp}[-1-{\omega }_i-{\theta }_i{\xi }_i],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i\in [0,1]\end{array}$$

The Lagrange multipliers, α_j, β_i and θ_l, meet the following equations:

$$\begin{array}{l}{\Gamma }_j(\gamma ,\theta )={\int }_{a_j^{-}}^{a_j^{+}}\text{exp}\left[-1-{\gamma }_j-{\sum }_{l=0}^1{\theta }_l{x}_{lj}{a}_j\right]d{a}_j=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j\in [0,8]\\ {\Omega }_i(\omega ,\theta )={\int }_{{\xi }_i^{-}}^{{\xi }_i^{+}}\text{exp}\left[-1-{\omega }_i-{\theta }_i{\xi }_i\right]d{\xi }_i=1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i\in [0,1]\end{array}$$

$$\begin{array}{l}{\Phi }_i(\gamma ,\omega ,\theta )={\sum }_{j=0}^8{x}_{ij}{\int }_{a_j^{-}}^{a_j^{+}}{a}_j\text{exp}\left[-1-{\gamma }_j-{\sum }_{l=0}^1{\theta }_l{x}_{lj}{a}_j\right]d{a}_j+\\ +{\int }_{{\xi }_i^{-}}^{{\xi }_i^{+}}{\xi }_i\text{exp}\left[-1-{\omega }_i-{\theta }_i{\xi }_i\right]d{\xi }_i=y_i,\hspace{0.17em}\hspace{0.17em}i\in [0,1],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j\in [0,8]\end{array}$$

To solve these equations, we have used MATLAB for symbolic transformations and the numerical solution of nonlinear equations, known as the “trust-region dogleg” technique. The computed Lagrange multipliers form

Table 11. The Lagrange multipliers, γ_j.

j	0	1	2	3	4	5
γj	−10.0475	−6.8421	−13.7688	1.5448	11.2746	−37.2191

Table 12. The Lagrange multipliers, γ_j (continued).

j	6	7	8
γj	−32.9907	−15.0797	−29.7189

and

Table 13. The Lagrange multipliers, ω_i and θ_i.

i	0	1
ωi	26.6458	14.7240
θi	9.9822	−2.7918

.

Figure 5 shows the curves of the probability density functions of the parameters, p_j(a^(1)_j), j ∈ [0, 2] (curves 0,1,2) and p_j(a^(2)_j), j ∈ [3, 8] (curves 3–8). Finally, Figure 6 presents the curves of the probability density functions of the noise, q₀(ξ₀), q₁(ξ₁).

8. Conclusions

New methods of parametric (probabilities) and non-parametric (probability density functions) estimation of the randomized model characteristics are proposed. These methods are based on entropy functions or entropy functionals maximized under certain constraints. We can interpret the obtained estimate as a robust one, as the entropy function was used for its calculation. These methods are focused on the problems where data is limited and distorted by noises. It is shown that the entropy-robust estimation of the probabilities and probability density functions belong to the exponential class.