An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion

Abstract

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the α-order information potential (IP). 62B10

1. Introduction

Given two random variables: X ∈ ℝⁿ which is an unknown parameter to be estimated, and Y ∈ ℝ^m which is the observation or measurement. The estimation of X based on Y, is in general a measurable function of Y, denoted by X̂ = g(Y) ∈ G, where G stands for the collection of all Borel measurable functions with respect to the σ -field generated by Y. The optimal estimate g^* (Y) can be determined by minimizing a certain risk, which is usually a function of the error distribution. If X has conditional probability density function (PDF) p(x|y), then:

$$g^{*}=\underset{g\in \mathit{G}}{\text{arg\hspace{0.17em}min}}\mathcal{R}\left(p^g(x)\right)$$

(1)

where p^g (x) is the PDF of the estimate error E = X–g (Y), $ℛ$ (.) is the risk function: ℙ_E → ℝ, where ℙ_E denotes the collection of all possible PDFs of the error. Let F(y) be the distribution function of Y, the PDF p^g (x) will be:

$$p^g(x)={\int }_{{ℝ}^m}p(x+g(y)\mid y)dF(y)$$

(2)

As one can see from Equation (2), the problem of choosing an optimal g is actually the problem of shifting the components of a mixture of the conditional PDF so as to minimize the risk $ℛ$.

The risk function $ℛ$ plays a central role in estimation related problems since it determines the performance surface and hence governs the optimal solution and the performance of the search algorithms. Traditional Bayes risk functions are, in general, defined as the expected value of a certain loss function (usually a nonlinear mapping) of the error:

$${\mathcal{R}}_{Bayes}\left(p^g(x)\right)={\int }_{{ℝ}^n}l(x)p^g(x)dx$$

(3)

where l(.) is the loss function. The most common Bayes risk function used for estimation is the mean square error (MSE), also called the squared error or quadratic error risk, which is defined by ${\mathcal{R}}_{MSE}\left(p^g(x)\right)={\int }_{{ℝ}^n}{\Vert x\Vert }_2^2{p}^g(x)dx$ (in this paper, ||.||_p denotes the p -norm). Using the MSE as risk, the optimal estimate of X is simply the conditional mean π (y) ≜ mean [p(.|y)]. The popularity of the MSE is due to its simplicity and optimality for linear Gaussian cases [1–3]. However, MSE is not always a superior risk function especially for non-linear and non-Gaussian situations, since it only takes into account the second order statistics. Therefore, many alternative Bayes risk functions have been used in practical applications. The mean absolute deviation (MAD) $ℛ$_MAD (p^g (x)) = ∫_ℝⁿ ||x||₁ p^g (x)dx, with which the optimal estimate is the conditional median μ (y) ≜ median [p(.|y)] (here the median of a random vector is defined as the element-wise median vector), is a robust risk function and has been successfully used in adaptive filtering in impulsive noise environments [4]. The mean 0–1 loss $ℛ$_0–1 (p^g (x)) = ∫_ℝⁿ l_0–1 (x) p^g (x) dx, where l_0–1 (.)denotes the 0–1 loss function (the 0–1 loss function has been frequently used in statistics and decision theory. If the error is a discrete variable, l_0–1 (x) = (x ≠ 0), where (.) is the indicator function, whereas if the error is a continuous variable, l_0–1 (x) is defined as l_0–1 (x) = 1 – δ (x), where δ (.) is the Dirac delta function), yields the optimal estimate as ζ (y) ≜ mode [p(.|y)], i.e., the conditional mode (the mode of a continuous probability distribution is the value at which its PDF attains its maximum value), which is also the maximum a posteriori (MAP) (the MAP estimate is a limit of Bayes estimator (under the 0–1 loss function), but generally not a Bayes estimator) estimate if regarding p(.y) as the posterior density. Other important Bayes risk functions include the mean p-power error [5], Huber’s M-estimation cost [6], and the risk-sensitive cost [7], etc. For general Bayes risk Equation (3), there is no explicit expression for the optimal estimate unless some conditions on l(x) or/and conditional density p(x|y) are imposed. As shown in [8], if l(x) is even and convex, and the conditional density p(x|y) is symmetric in x, the optimal estimate will be the conditional mean (or equivalently, the conditional median).

Besides the traditional Bayes risk functions, the error entropy (EE) can also be used as a risk function in estimation problems. Using Shannon’s definition of entropy [9], the EE risk function is:

$${\mathcal{R}}_S\left(p^g(x)\right)=-{\int }_{{ℝ}^n}{p}^g(x)\hspace{0.17em}\text{log}\hspace{0.17em}{p}^g(x)dx$$

(4)

As the entropy measures the average dispersion or uncertainty of a random variable, its minimization makes the error concentrated. Different from conventional Bayes risks, the “loss function” of the EE risk (4) is – log p^g (x), which is directly related to the error’s PDF. Therefore, when using the EE risk, we are nonlinearly transforming the error by its own PDF. In 1970, Weidemann and Stear published a paper entitled Entropy Analysis of Estimating Systems [10] in which they studied the parameter estimation problem using the error entropy as a criterion functional. They proved minimizing the error entropy is equivalent to minimizing the mutual information between error and observation, and also proved that the reduced error entropy is upper-bounded by the amount of information obtained by observation. Later, Tomita et al. [11] and Kalata and Priemer [12] studied the estimation and filtering problems from the viewpoint of information theory and derived the famed Kalman filter as a special case of minimum-error-entropy (MEE) linear estimators. Like most Bayes risks, the EE risk (4) has no explicit expression for the optimal estimate unless some constraints on the conditional density p(x|y) are imposed. In a recent paper [13], Chen and Geman proved that, if p(x|y) is conditionally symmetric and unimodal (CSUM), the MEE estimate (the optimal estimate under EE risk) will be the conditional median (or equivalently, the conditional mean or mode).

Table 1. Optimal estimates for several risk functions.

Risk function	$ℛ$ (pg(x))	Optimal estimate
Mean square error (MSE)	${\int }_{{ℝ}^n}{\Vert x\Vert }_2^2{p}^g(x)dx$	g* (y) = π(y) ≜ mean [p(.|y)]
Mean absolute deviation (MAD)	∫ℝn ||x||1 pg (x) dx	g* (y) = μ(y) ≜ median [p(.|y)]
Mean 0–1 loss	∫ℝn l0–1 (x) pg (x)dx	g* (y) = ζ (y) ≜ mode [p(.|y)]
General Bayes risk	∫ℝn l (x) pg (x) dx	If l(x) is even and convex, and p(x|y) is symmetric in x, then g* (y) = π(y) = μ(y)
Error entropy (EE)	−∫ℝn pg (x) log pg (x) dx	If p(x|y) is CSUM, then g*(y) = π(y) = μ(y) = ζ(y)

gives a summary of the optimal estimates for several risk functions. Since the entropy of a PDF remains unchanged after shift, the MEE estimator is in general restricted to be an unbiased one (i.e., with zero-mean error).

In statistical information theory, there are many extensions to Shannon’s original definition of entropy. Renyi’s entropy is one of the parametrically extended entropies. Given a random variable X with PDF p(x), α -order Renyi entropy is defined by [14]:

$$H_{\alpha }(X)=\frac{1}{1-\alpha }\text{log}\left({\int }_{{ℝ}^n}{(p(x))}^{\alpha }\hspace{0.17em}dx\right)$$

(5)

where α > 0, and α ≠ 1. The entropy definition (5) becomes the usual Shannon entropy as α → 1. Renyi entropy can be used to define a generalized EE risk:

$${\mathcal{R}}_{\alpha }\left(p^g(x)\right)=\frac{1}{1-\alpha }\text{log}\left({\int }_{{ℝ}^n}{\left(p^g(x)\right)}^{\alpha }\hspace{0.17em}dx\right)$$

(6)

In recent years, the EE risk (6) has been successfully used as an adaptation cost in information theoretic learning (ITL) [15–22]. It has been shown that the nonparametric kernel (Parzen window) estimator of Renyi entropy (especially when α = 2) is more computationally efficient than that of Shannon entropy [15]. The argument of the logarithm in Renyi entropy, denoted by V_α (V_α = ∫_ℝⁿ (p(x))^α dx), is called the α-order information potential (IP) (this quantity is called information potential since each term in its kernel estimator can be interpreted as a potential between two particles (see [15] for the physical interpretation of kernel estimator of information potential) [15]. As the logarithm is a monotonic function, the minimization of Renyi entropy is equivalent to the minimization (when α < 1) or maximization (when α > 1) of information potential. In practical applications, information potential has been frequently used as an alternative to Renyi entropy [15].

A natural and important question now arises: what is the optimal estimate under the generalized EE risk (6)? We do not know the answer to this question in the general case. In this work, however, we will extend the results by Chen and Geman [13] to a more general case and show that, if the conditional density p(x|y)is CSUM, the generalized MEE estimate will also be the conditional median (or equivalently, the conditional mean or mode).

2. Main Theorem and the Proof

In this section, our focus is on the α -order information potential (IP), but the conclusions drawn can be immediately transferred to Renyi entropy. The main theorem of the paper is as follows.

Theorem 1

Assume for every value y ∈ ℝ^m, that the conditional PDF p(x|y) is conditionally symmetric (rotation invariant for multivariate case) and unimodal (CSUM) in x ∈ ℝ^n, and let μ (y) = median [p(.|y)]. If α -order information potential V_α (X – μ (Y)) < ∞ (α > 0, α α 1), then:

$$\{\begin{array}{ll}{V}_{\alpha }(X-\mu (Y))\le V_{\alpha }(X-g(Y))\hfill & if\hspace{0.17em}0<\alpha <1\hfill \\ V_{\alpha }(X-\mu (Y))\ge V_{\alpha }(X-g(Y))\hfill & if\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\alpha >1\hfill \end{array}$$

(7)

for all g :ℝ^m → ℝⁿ for which V_α (X – g(Y)) < ∞.

Remark

As p(x|y) is CSUM, the conditional median μ (y) in Theorem 1 is the same as the conditional mean π (y) and conditional mode ζ (y). According to the relationship between information potential and Renyi entropy, the inequalities in Equation (7) are equivalent to:

$$H_{\alpha }\left(X-\mu (Y)\right)\le H_{\alpha }\left(X-g(Y)\right)$$

(8)

Proof of the Theorem

In this work, we give a proof for the univariate case (n = 1). A similar proof can be easily extended to the multivariate case (n > 1). In the proof we assume, without loss of generality, that ∀y, p(x|y) has median at x = 0, since otherwise we could replace p(x|y)by p(x+ μ(y)|y) and work instead with conditional densities centered at x = 0. The road map of the proof is similar to that contained in [13]. There are, however, significant differences between our work and [13]: (1) we extend the entropy minimization problem to the generalized error entropy; (2) in our proof, the Holder inequality is applied, and there is no discretization procedure, which simplifies the proof significantly.

First, we prove the following proposition:

Proposition 1

Assume that f (x|y) (not necessarily a conditional density function) satisfies

(1)

non-negative, continuous and integrable in x for each y ∈ ℝ^m;

(2)

symmetric (rotation invariant for n > 1) around x = 0 and unimodal for each y ∈ ℝ^m;

(3)

uniformly bounded in (x, y);

(4)

V_α(f⁰) < ∞, where V_α (f⁰) = ∫_ℝ (f⁰(x))^α dx, and f⁰(x) = ∫_ℝ^m f (x|y)dF (y).

Then for all g :ℝ^m → ℝ for which V_α (f^g) < ∞, we have

$$\{\begin{array}{ll}{V}_{\alpha }(f^0)\le V_{\alpha }(f^g)\hfill & if\hspace{0.17em}0<\alpha <1\hfill \\ V_{\alpha }(f^0)\ge V_{\alpha }(f^g)\hfill & if\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\alpha >1\hfill \end{array}$$

(9)

where V_α (f^g) = ∫_ℝ (f^g(x))^α dx, f^g (x) = ∫_ℝ^m f (x+g (y)|y) dF (y).

Remark

It is easy to observe that ∫ f⁰ dx = ∫f^g dx ≤ sup_(x,y) f (x|y) < ∞ (not necessarily ∫ f⁰ dx = 1).

Proof of the Proposition

The proof is based on the following three lemmas.

Lemma 1[13]

Let non-negative function h : ℝ → [0, ∞) be bounded, continuous, and integrable, and define function O_h (z) by:

$$O_h(z)=\lambda \{x:h(x)\ge z\}$$

(10)

where λ is Lebesgue measure. Then the following results hold:

(a)

Define m^h (x) = sup {z : O_h (z) ≥ x}, x ∈ (0, ∞), and m^h (0) = sup_x h(x). Then m^h (x) is continuous and non-increasing on [0, ∞), and m^h (x) → 0 as x → ∞.

(b)

For any function G:[0, ∞) → ℝ with ∫_ℝ |G(h(x)) dx < ∞:

$${\int }_{ℝ}G\left(h(x)\right)\hspace{0.17em}dx={\int }_0^{\infty }G\hspace{0.17em}\left(m^h(x)\right)\hspace{0.17em}dx$$

(11)

(c)

For any x₀ ∈ [0, ∞):

$${\int }_0^{x_0}{m}^h(x)dx=\underset{A:\lambda (A)=x_0}{\text{sup}}{\int }_{A}h(x)dx$$

(12)

Proof of Lemma 1

See the proof of Lemma 1 in [13].

Remark

The transformation h → m^h in Lemma1 is also called the “rearrangement” of h [23]. By Lemma 1, we have V_α (m ^fg) = V_α (f^g) < ∞ and V_α (m ^f0) = V_α (f⁰) < ∞ (let G(x) = x ^α). Therefore, to prove Proposition 1, it suffices to prove:

$$\{\begin{array}{ll}{V}_{\alpha }(m^{f^0})\le V_{\alpha }(m^{f^g})\hfill & if\hspace{0.17em}0<\alpha <1\hfill \\ V_{\alpha }(m^{f^0})\ge V_{\alpha }(m^{f^g})\hfill & if\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\alpha >1\hfill \end{array}$$

(13)

Lemma 2

Denote m^g = m^{f g,} m⁰ = m^{f 0}. Then:

(a)

$${\int }_0^{\infty }{m}^g(x)dx={\int }_0^{\infty }{m}^0(x)dx<\infty$$

(14)

(b)

$${\int }_0^{x_0}{m}^g(x)dx\le {\int }_0^{x_0}{m}^0(x)dx,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\forall x_0\in [0,\infty )$$

(15)

Proof of Lemma 2

See the proof of Lemma 3 in [13].

Lemma 3

∀α > 0, let n be a non-negative integer such that n < α ≤ n+1. Then ∀x₀ ∈ [0, ∞):

(a)

$${\int }_0^{x_0}\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\int }_0^{x_0}{m}^0(x)dx$$

(16)

(b)

$${\int }_{x_0}^{\infty }\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\int }_{x_0}^{\infty }{m}^g(x)dx$$

(17)

Proof of Lemma 3

According to Holder inequality [23], we have ∀Ω ⊂ [0, ∞):

$${\int }_{\mathrm{\Omega }}\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\left({\int }_{\mathrm{\Omega }}{m}^g(x)dx\right)}^{\alpha -n}{\left({\int }_{\mathrm{\Omega }}{m}^0(x)dx\right)}^{n+1-\alpha }$$

(18)

By Lemma 2, ${\int }_0^{x_0}{m}^g(x)dx\le {\int }_0^{x_0}{m}^0(x)dx$, it follows that:

$$\begin{array}{l}{\int }_0^{x_0}\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\left({\int }_0^{x_0}{m}^g(x)dx\right)}^{\alpha -n}{\left({\int }_0^{x_0}{m}^0(x)dx\right)}^{n+1-\alpha }\\ \le {\left({\int }_0^{x_0}{m}^0(x)dx\right)}^{\alpha -n}{\left({\int }_0^{x_0}{m}^0(x)dx\right)}^{n+1-\alpha }\\ ={\int }_0^{x_0}{m}^0(x)dx\end{array}$$

(19)

Further, since ${\int }_0^{\infty }{m}^g(x)dx={\int }_0^{\infty }{m}^0(x)dx$, we have ${\int }_{x_0}^{\infty }{m}^g(x)dx\ge {\int }_{x_0}^{\infty }{m}^0(x)dx$, and hence:

$$\begin{array}{l}{\int }_{x_0}^{\infty }\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\left({\int }_{x_0}^{\infty }{m}^g(x)dx\right)}^{\alpha -n}{\left({\int }_{x_0}^{\infty }{m}^0(x)dx\right)}^{n+1-\alpha }\\ \le {\left({\int }_{x_0}^{\infty }{m}^g(x)dx\right)}^{\alpha -n}{\left({\int }_{x_0}^{\infty }{m}^g(x)dx\right)}^{n+1-\alpha }\\ ={\int }_{x_0}^{\infty }{m}^g(x)dx\end{array}$$

(20)

Q.E.D. (Lemma 3)

Let S_g = sup{x : m^g (x) > 0}, which is finite or infinite, Equation (17) can be rewritten as:

$${\int }_{x_0}^{S_g}\left\{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\right\}dx\le {\int }_{x_0}^{S_g}{m}^g(x)dx$$

(21)

Now we are in position to prove Equation (13):

(1)

0 < α < 1: In this case, we have:

$$\begin{array}{l}{\int }_0^{\infty }{\left(m^g(x)\right)}^{\alpha }dx={\int }_0^{S_g}{m}^g(x)\hspace{0.17em}{\left(m^g(x)\right)}^{\alpha -1}dx\\ ={\int }_0^{S_g}{m}^g(x)\hspace{0.17em}\left({\int }_0^{{\left(m^g(x)\right)}^{\alpha -1}}dy\right)dx\\ ={\int }_0^{S_g}\left\{{\int }_0^{\infty }{m}^g(x)\mathbb{I}\left(y\le {\left(m^g(x)\right)}^{\alpha -1}\right)dy\right\}dx\\ ={\int }_0^{\infty }\left({\int }_{\text{inf}\left\{x:{\left(m^g(x)\right)}^{\alpha -1}\ge y\right\}}^{S_g}{m}^g(x)dx\right)dy\\ \stackrel{(A)}{\ge }{\int }_0^{\infty }\left\{{\int }_{\text{inf}\left\{x:{\left(m^g(x)\right)}^{\alpha -1}\ge y\right\}}^{S_g}{\left(m^g(x)\right)}^{1-\alpha }{\left(m^0(x)\right)}^{\alpha }dx\right\}\hspace{0.17em}dy\\ ={\int }_0^{\infty }\left\{{\int }_0^{S_g}{\left(m^g(x)\right)}^{1-\alpha }{\left(m^0(x)\right)}^{\alpha }\mathbb{I}\left({\left(m^g(x)\right)}^{\alpha -1}\ge y\right)dx\right\}\hspace{0.17em}dy\\ ={\int }_0^{S_g}{\left(m^g(x)\right)}^{1-\alpha }{\left(m^0(x)\right)}^{\alpha }\left({\int }_0^{{\left(m^g(x)\right)}^{\alpha -1}}dy\right)\hspace{0.17em}dx\\ ={\int }_0^{S_g}{\left(m^0(x)\right)}^{\alpha }\hspace{0.17em}dx\\ \stackrel{(B)}{=}{\int }_0^{S_g}{\left(m^0(x)\right)}^{\alpha }\hspace{0.17em}dx+{\int }_{S_g}^0{\left(m^0(x)\right)}^{\alpha }\hspace{0.17em}dx\\ ={\int }_0^{\infty }{\left(m^0(x)\right)}^{\alpha }\hspace{0.17em}dx\end{array}$$

(22)

where (A) follows from Equation (21), and (B) is due to ${\int }_{S_g}^{\infty }{\left(m^0(x)\right)}^{\alpha }\hspace{0.17em}dx=0$, since

$$\begin{array}{c}0={\int }_{S_g}^{\infty }{m}^g(x)dx\ge {\int }_{S_g}^{\infty }{m}^0(x)dx\ge 0\\ \Rightarrow {\int }_{S_g}^{\infty }{m}^0(x)dx=0\end{array}$$

(23)

(2)

α > 1: First we have

$$\begin{array}{l}{\int }_0^{\infty }{\left(m^0(x)\right)}^{\alpha }dx={\int }_0^{\infty }{m}^0(x)\hspace{0.17em}{\left(m^0(x)\right)}^{\alpha -1}\hspace{0.17em}dx\\ ={\int }_0^{\infty }{m}^0(x)\left({\int }_0^{{\left(m^0(x)\right)}^{\alpha -1}}dy\right)dx\\ ={\int }_0^{\infty }\left\{{\int }_0^{\infty }{m}^0(x)\mathbb{I}\left(y\le {\left(m^0(x)\right)}^{\alpha -1}\right)dy\right\}dx\\ ={\int }_0^{\infty }\left({\int }_0^{\text{sup}\left\{x:{\left(m^0(x)\right)}^{\alpha -1}\ge y\right\}}{m}^0(x)dx\right)dy\\ \stackrel{(C)}{\ge }{\int }_0^{\infty }\left\{{\int }_0^{\text{sup}\left\{x:{\left(m^0(x)\right)}^{\alpha -1}\ge y\right\}}{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }dx\right\}dy\\ ={\int }_0^{\infty }\left\{{\int }_0^{\infty }{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\mathbb{I}\left({\left(m^0(x)\right)}^{\alpha -1}\ge y\right)dx\right\}dy\\ ={\int }_0^{\infty }{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n+1-\alpha }\left({\int }_0^{{\left(m^0(x)\right)}^{\alpha -1}}dy\right)\hspace{0.17em}dx\\ ={\int }_0^{\infty }{\left(m^0(x)\right)}^n{\left(m^g(x)\right)}^{\alpha -n}\hspace{0.17em}dx\end{array}$$

(24)

where (C) follows from Equation (16). Further, one can derive:

$$\begin{array}{l}{\int }_0^{\infty }{\left(m^0(x)\right)}^n{\left(m^g(x)\right)}^{\alpha -n}dx={\int }_0^{\infty }\left(m^0(x){\int }_0^{{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n-1}}dy\right)dx\\ ={\int }_0^{\infty }\left\{{\int }_0^{\infty }{m}^0(x)\mathbb{I}\left(y\le {\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n-1}\right)dy\right\}dx\\ ={\int }_0^{\infty }\left\{{\int }_0^{\text{sup}\left\{x:{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n-1}\ge y\right\}}{m}^0(x)dx\right\}dy\\ \stackrel{(D)}{\ge }{\int }_0^{\infty }\left\{{\int }_0^{\text{sup}\left\{x:{\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n-1}\ge y\right\}}{m}^g(x)dx\right\}dy\\ ={\int }_0^{\infty }\left\{{\int }_0^{\infty }{m}^g(x)\mathbb{I}\left({\left(m^g(x)\right)}^{\alpha -n}{\left(m^0(x)\right)}^{n-1}\ge y\right)dx\right\}dy\\ ={\int }_0^{\infty }{\left(m^0(x)\right)}^{n-1}{\left(m^g(x)\right)}^{\alpha -n+1}dx\\ ⋮\\ \ge {\int }_0^{\infty }{\left(m^0(x)\right)}^{n-2}{\left(m^g(x)\right)}^{\alpha -n+2}dx\\ ⋮\\ \ge {\int }_0^{\infty }{\left(m^g(x)\right)}^{\alpha }dx\end{array}$$

(25)

where (D) is because ${\int }_0^{x_0}{m}^g(x)dx\le {\int }_0^{x_0}{m}^0(x)dx,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\forall x_0\in [0,\infty )$, ∀x₀ ∈ [0, ∞). Combining Equations (24) and (25), we get ${\int }_0^{\infty }{\left(m^0(x)\right)}^{\alpha }dx\ge {\int }_0^{\infty }{\left(m^g(x)\right)}^{\alpha }dx$ (i.e., V_α (m⁰) ≥ V_α (m^g)).

Up to now, the proof of Proposition 1 has been completed. Let us come back to the proof of Theorem 1. The remaining task is just to remove the conditions of continuity and uniform boundedness imposed in Proposition 1. This can be easily accomplished by approximating p(x|y) by a sequence of functions {f_n (x|y)}, n = 1, 2,..., which satisfy the conditions of Proposition 1. Then similar to [13], we define:

$$f_n(x\mid y)=\{\begin{array}{l}n{\int }_x^{x+(1/n)}\text{min}\hspace{0.17em}\left(n,p(z\mid y)\right)dz\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\forall x\in [0,\infty )\hfill \\ f_n(-x\mid y)\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\forall x\in (-\infty ,0)\hfill \end{array}$$

(26)

It is easy to verify that for each n, f_n (x|y) satisfies all the conditions of Proposition 1. Here we only give the proof for condition 4). Let $f_n^0(x)={\int }_{{ℝ}^m}{f}_n(x\mid y)dF(y)$, we have:

$$\begin{array}{l}{V}_{\alpha }(f_n^0)={\int }_{ℝ}{\left(f_n^0(x)\right)}^{\alpha }\hspace{0.17em}dx\\ ={\int }_{ℝ}{\left({\int }_{{ℝ}^m}{f}_n(x\mid y)dF(y)\right)}^{\alpha }\hspace{0.17em}dx\\ =2{\int }_{{ℝ}_{+}}{\left({\int }_{{ℝ}^m}\left(n{\int }_x^{x+(1/n)}\text{min}\hspace{0.17em}\left(n,p(z\mid y)\right)dz\right)dF(y)\right)}^{\alpha }dx\\ \le 2{\int }_{{ℝ}_{+}}{\left({\int }_{{ℝ}^m}\left(n{\int }_x^{x+(1/n)}\underset{z\in [x,x+(1/n)]}{\text{sup}}p(z\mid y)dz\right)dF(y)\right)}^{\alpha }dx\\ \stackrel{(E)}{=}2{\int }_{{ℝ}_{+}}{\left({\int }_{{ℝ}^m}\left(n{\int }_x^{x+(1/n)}p(x\mid y)dz\right)dF(y)\right)}^{\alpha }dx\\ =2{\int }_{{ℝ}_{+}}{\left({\int }_{{ℝ}^m}p(x\mid y)dF(y)\right)}^{\alpha }dx\\ =V_{\alpha }(p^0)<\infty \end{array}$$

(27)

where (E) comes from the fact that ∀y, p(x|y)is non-increasing in x over [0, ∞), since it is CSUM.

According to Proposition 1, we have, for every n :

$$\{\begin{array}{ll}{V}_{\alpha }(f_n^0)\le V_{\alpha }(f_n^g)\hfill & if\hspace{0.17em}0<\alpha <1\hfill \\ V_{\alpha }(f_n^0)\ge V_{\alpha }(f_n^g)\hfill & if\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\alpha >1\hfill \end{array}$$

(28)

where $f_n^g(x){\int }_{{ℝ}^m}{f}_n(x+g(y)\mid y)dF(y)$. In order to complete the proof of Theorem 1, we only need to show that $V_{\alpha }(f_n^0)\to V_{\alpha }(p^0)$, and $V_{\alpha }(f_n^g)\to V_{\alpha }(p^g)$. This can be proved by the dominated convergence theorem. Here we only show $V_{\alpha }(f_n^0)\to V_{\alpha }(p^0)$, the proof for $V_{\alpha }(f_n^g)\to V_{\alpha }(p^g)$ is identical. First, it is clear that $f_n^0(x)\le p^0(x)$, ∀x, and hence ${\left(f_n^0(x)\right)}^{\alpha }\le {\left(p^0(x)\right)}^{\alpha }$, ∀x. Also, we can derive:

$${\int }_{ℝ}\mid f_n^0(x)-p^0(x)\mid dx\to 0$$

(29)

As V_α (p⁰) = ∫_ℝ (p⁰(x))^α < ∞, then by dominated convergence theorem, $V_{\alpha }(f_n^0)\to V_{\alpha }(p^0)$.

Q.E.D. (Theorem 1)

Remark

The condition of the CSUM in Theorem 1 is a little strong, but it can be easily relaxed to just requiring that the conditional PDF p(x|y) is generalized uniformly dominated (GUD) in x ∈ ℝⁿ (see [24] for the definition of GUD).

3. Conclusions

The problem of determining a minimum-error-entropy (MEE) estimator is actually the problem of shifting the components of a mixture of the conditional PDF so as to minimize the entropy of the mixture. It has been proved in a recent paper that, if the conditional distribution is conditionally symmetric and unimodal (CSUM), the Shannon entropy of the mixture distribution will be minimized by aligning the conditional median. In the present work, this result has been extended to a more general case. We show that if the conditional distribution is CSUM, the Renyi entropy of the mixture distribution will also be minimized by aligning the conditional median.