Some Further Results on the Minimum Error Entropy Estimation

Abstract

The minimum error entropy (MEE) criterion has been receiving increasing attention due to its promising perspectives for applications in signal processing and machine learning. In the context of Bayesian estimation, the MEE criterion is concerned with the estimation of a certain random variable based on another random variable, so that the error’s entropy is minimized. Several theoretical results on this topic have been reported. In this work, we present some further results on the MEE estimation. The contributions are twofold: (1) we extend a recent result on the minimum entropy of a mixture of unimodal and symmetric distributions to a more general case, and prove that if the conditional distributions are generalized uniformly dominated (GUD), the dominant alignment will be the MEE estimator; (2) we show by examples that the MEE estimator (not limited to singular cases) may be non-unique even if the error distribution is restricted to zero-mean (unbiased).

1. Introduction

A central concept in information theory is entropy, which is a mathematical measure of the uncertainty or the amount of missing information [1]. Entropy has been widely used in many areas, including physics, mathematics, communication, economics, signal processing, machine learning, etc. The maximum entropy principle is a powerful and widely accepted method for statistical inference or probabilistic reasoning with incomplete knowledge of probability distribution [2]. Another important entropy principle is the minimum entropy principle, which decreases the uncertainty associated with a system. In particular, the minimum error entropy (MEE) criterion can be applied in problems like estimation [3,4,5], identification [6,7], filtering [8,9,10], and system control [11,12]. In recent years, the MEE criterion, together with the nonparametric Renyi entropy estimator, has been successfully used in information theoretic learning (ITL) [13,14,15].

In the scenario of Bayesian estimation, the MEE criterion aims to minimize the entropy of the estimation error, and hence decrease the uncertainty in estimation. Given two random variables: $X\in {ℝ}^n$, an unknown parameter to be estimated, and $Y\in {ℝ}^m$, the observation (or measurement), the MEE estimation of $X$ based on $Y$ can be formulated as:

$$\begin{array}{l}{g}^{*}=\underset{g\in G}{\arg \min }H\left(X-g(Y)\right)\\ =\underset{g\in G}{\arg \min }E\left[-\log p^g\left(X-g(Y)\right)\right]\\ =\underset{g\in G}{\arg \min }-{\displaystyle {\int }_{{ℝ}^n}{p}^g(x)\log p^g(x)dx}\end{array}$$

(1)

where $g(Y)$ denotes an estimator of $X$ based on $Y$, $g$ is a measurable function, $G$ stands for the collection of all measurable functions of $Y$, $H\left(X-g(Y)\right)$ denotes the Shannon entropy of the estimation error $X-g(Y)$, and $p^g(x)$ denotes the probability density function (PDF) of the estimation error. Let $p(x|y)$ be the conditional PDF of $X$ given $Y$. Then:

$$p^g(x)={\displaystyle {\int }_{{ℝ}^m}p(x+g(y)|y)dF(y)}$$

(2)

where $F(y)$ denotes the distribution function of $Y$. From (2), one can see the error PDF $p^g(x)$ is actually a mixture of the shifted conditional PDF.

Different from conventional Bayesian risks, like mean square error (MSE) and risk-sensitive cost [16], the “loss function” in MEE is $-\log p^g(.)$, which is directly related to the error’s PDF, transforming nonlinearly the error by its own PDF. Some theoretical aspects of MEE estimation have been studied in the literature. In an early work [3], Weidemann and Stear proved that minimizing the error entropy is equivalent to minimizing the mutual information between the error and the observation, and also proved that the reduced error entropy is upper-bounded by the amount of information obtained by the observation. In [17], Janzura et al. proved that, for the case of finite mixtures ($Y$ is a discrete random variable with finite possible values), the MEE estimator equals the conditional median provided that the conditional PDFs are conditionally symmetric and unimodal (CSUM). Otahal [18] extended Janzura’s results to finite-dimensional Euclidean space. In a recent paper, Chen and Geman [19] employed a “function rearrangement” to study the minimum entropy of a mixture of CSUM distributions where no restriction on $Y$ was imposed. More recently, Chen et al. have investigated the robustness, non-uniqueness (for singular cases), sufficient condition, and the necessary condition involved in the MEE estimation [20]. Chen et al. have also presented a new interpretation on the MSE criterion as a robust MEE criterion [21].

In this work, we continue the study on the MEE estimation, and obtain some further results. Our contributions are twofold. First, we extend the results of Chen and Geman to a more general case, and show that when the conditional PDFs are generalized uniformly dominated (GUD), the MEE estimator equals the dominant alignment. Second, we show by examples that, the unbiased MEE estimator (not limited to singular cases) may be non-unique, and there can even be infinitely many optimal solutions. The rest of the paper is organized as follows. In Section 2, we study the minimum entropy of a mixture of generalized uniformly dominated conditional distributions. In Section 3, we present two examples to show the non-uniqueness of the unbiased MEE estimation. Finally, we give our conclusions in Section 4.

2. MEE Estimator for Generalized Uniformly Dominated Conditional Distributions

Before presenting the main theorem of this section, we give the following definitions.

Definition 1: Let $\mathit{ℱ}=\left\{f_t(x):{ℝ}^n\to {ℝ}_{+},t\in \mathbb{T}\right\}$ be a set of nonnegative, integrable functions, where $\mathbb{T}$ denotes an index set (possibly uncountable). Then $\mathit{ℱ}$ is said to be uniformly dominated in $x\in {ℝ}^n$ if and only if $\forall \epsilon \in \left[0,\infty \right)$, there exists a measurable set $D_{\epsilon }\subset {ℝ}^n$, satisfying $\lambda (D_{\epsilon })=\epsilon$ and:

$${\displaystyle \underset{D_{\epsilon }}{\int }{f}_t\left(x\right)dx}=\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle \underset{A}{\int }{f}_t\left(x\right)dx},\forall t\in \mathbb{T}$$

(3)

where $\lambda$ is Lebesgue measure. The set $D_{\epsilon }$ is called the $\epsilon$-volume dominant support of $\mathit{ℱ}$.

Definition 2: The nonnegative, integrable function set $\mathit{ℱ}$ is said to be generalized uniformly dominated (GUD) in $x\in {ℝ}^n$ if and only if there exists a function $\pi :\mathbb{T}\to {ℝ}^n$, such that ${\mathit{ℱ}}_{\pi }$ is uniformly dominated, where:

$${\mathit{ℱ}}_{\pi }=\left\{f_t^{\pi }:f_t^{\pi }\left(x\right)=f_t\left(x+\pi (t)\right),t\in \mathbb{T}\right\}$$

(4)

The function $\pi (t)$ is called the dominant alignment of $\mathit{ℱ}$.

Remark 1: The dominant alignment is, obviously, non-unique. If $\pi (t)$ is a dominant alignment of $\mathit{ℱ}$, then $\forall c\in {ℝ}^n$, $\pi (t)+c$ will also be a dominant alignment of $\mathit{ℱ}$.

When regarding $y$ as an index parameter, the conditional PDF $p(x|y)$ will represent a set of nonnegative and integrable functions, that is:

$$\mathit{𝒫}=\left\{p(x|y):{ℝ}^n\to {ℝ}_{+},y\in {ℝ}^m\right\}$$

(5)

If the above function set is (generalized) uniformly dominated in $x\in {ℝ}^n$, then we say that the conditional PDF $p(x|y)$ is (generalized) uniformly dominated in $x\in {ℝ}^n$.

Remark 2: The GUD is much more general than CSUM. Actually, if the conditional PDF is CSUM, it must also be GUD (with the conditional mean as the dominant alignment), but not vice versa. In Figure 1 we show two examples where two PDFs (solid and dotted lines) are uniformly dominated but not CSUM.

Figure 1. Uniformly dominated PDFs: (a) non-symmetric, (b) non-unimodal.

Figure 1. Uniformly dominated PDFs: (a) non-symmetric, (b) non-unimodal.

Theorem 1: Assume the conditional PDF $p(x|y)$ is generalized uniformly dominated in $x\in {ℝ}^n$, with dominant alignment $\pi (y)$. If $H(X-\pi (Y))$ exists (here “exists” means “exists in the extended sense” as defined in [19]) then $H(X-\pi (Y))\le H(X-g(Y))$ for all $g:{ℝ}^m\to {ℝ}^n$ for which $H(X-g(Y))$ also exists.

Proof of Theorem 1: The proof presented below is similar to that of the Theorem 1 in [19], except that the discretization procedure is avoided. In the following, we give a brief sketch of the proof, and consider only the case $n=1$ (the proof can be easily extended to $n>1$). First, one needs to prove the following proposition.

Proposition 1: Assume that the function $f(x|y)$, $x\in ℝ$, $y\in {ℝ}^m$ (not necessarily a conditional PDF) satisfies the following conditions:

(1)

non-negative, continuous, and integrable in $x\in ℝ$ for each $y\in {ℝ}^m$;

(2)

generalized uniformly dominated in $x$, with dominant alignment $\pi (y)$;

(3)

uniformly bounded in $(x,y)$.

Then for any $g:{ℝ}^m\to ℝ$, we have:

$$H\left(f^{\pi }\right)\le H\left(f^g\right)$$

(6)

where $H\left(f\right)=-{\displaystyle {\int }_{ℝ}f(x)\log f(x)dx}$ (here we extend the entropy definition to nonnegative $L^1$ functions), and:

$$\{\begin{array}{l}{f}^{\pi }(x)={\displaystyle {\int }_{{ℝ}^m}f(x+\pi (y)|y)dF(y)}\\ f^g(x)={\displaystyle {\int }_{{ℝ}^m}f(x+g(y)|y)dF(y)}\end{array}$$

(7)

Remark 3: It is easy to verify that ${\displaystyle \int f^{\pi }dx}={\displaystyle \int f^{g}dx}\le {\sup }_{(x,y)}f\left(x|y\right)<\infty$ (not necessarily ${\displaystyle \int f^{\pi }dx}=1$).

Proof of Proposition 1: The above proposition can be readily proved using the following two lemmas.

Lemma 1 [19]: Assume the nonnegative function $h:ℝ\to \left[0,\infty \right)$ is bounded, continuous, and integrable, and define the function $O_h(z)$ by ($\lambda$ is Lebesgue measure):

$$O_h(z)=\lambda \left\{x:h(x)\ge z\right\}$$

(8)

Then the following results hold:

(a)

Define $m^h(x)=\sup \left\{z:O_h(z)\ge x\right\}$, $x\in \left(0,\infty \right)$, and $m^h(0)={\sup }_{x}h(x)$. Then $m^h(x)$ is continuous and non-increasing on $\left[0,\infty \right)$, and $m^h(x)\to 0$ as $x\to \infty$.

(b)

For any function $G:\left[0,\infty \right)\to ℝ$ with ${\displaystyle {\int }_{ℝ}\left|G\left(h(x)\right)\right|dx}<\infty$

$${\displaystyle {\int }_{ℝ}G\left(h(x)\right)dx}={\displaystyle {\int }_0^{\infty }G\left(m^h(x)\right)dx}$$

(9)

(c)

For any $x_0\in \left[0,\infty \right)$

$${\displaystyle {\int }_0^{x_0}{m}^h(x)dx}=\underset{A:\lambda (A)=x_0}{\sup }{\displaystyle {\int }_{A}h(x)dx}$$

(10)

Proof of Lemma 1: See [19].

Remark 4: Denote $m^{\pi }=m^{f^{\pi }}$, and $m^g=m^{f^g}$. Then by Lemma 1, we have $H(m^{\pi })=H(f^{\pi })$ and $H(m^g)=H(f^g)$ (let $G(x)=-x\log x$). Therefore, to prove Proposition 1, it suffices to prove:

$$H(m^{\pi })\le H(m^g)$$

(11)

Lemma 2: Functions $m^{\pi }$ and $m^g$ satisfy:

(a)

$${\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)dx}={\displaystyle {\int }_0^{\infty }{m}^g(x)dx}<\infty$$

(12)

(b)

$${\displaystyle {\int }_0^{\epsilon }{m}^g(x)dx}\le {\displaystyle {\int }_0^{\epsilon }{m}^{\pi }(x)dx},\forall \epsilon \in \left[0,\infty \right)$$

(13)

Proof of Lemma 2: (a) comes directly from the fact ${\displaystyle \int f^{\pi }dx}={\displaystyle \int f^{g}dx}<\infty$. We only need to prove (b). We have:

$$\begin{array}{l}\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{f}^g(x)dx}\\ =\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{\displaystyle {\int }_{{ℝ}^m}f(x+g(y)|y)dF(y)}dx}\\ =\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_{{ℝ}^m}{\displaystyle {\int }_{A}f(x+g(y)|y)dx}dF(y)}\\ \le {\displaystyle {\int }_{{ℝ}^m}\left\{\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_{A}f(x+g(y)|y)dx}\right\}dF(y)}\\ ={\displaystyle {\int }_{{ℝ}^m}\left\{{\displaystyle {\int }_{D_{\epsilon }}f(x+\pi (y)|y)dx}\right\}dF(y)}\\ ={\displaystyle {\int }_{D_{\epsilon }}{\displaystyle {\int }_{{ℝ}^m}f(x+\pi (y)|y)dF(y)}dx}\\ =\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{\displaystyle {\int }_{{ℝ}^m}f(x+\pi (y)|y)dF(y)}dx}\\ =\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{f}^{\pi }(x)dx}\end{array}$$

(14)

where $D_{\epsilon }$ is the $\epsilon$-volume dominant support of $f(x+\pi (y)|y)$. By Lemma 1 (c):

$$\{\begin{array}{l}{\displaystyle {\int }_0^{\epsilon }{m}^g(x)dx}=\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{f}^g(x)dx}\\ {\displaystyle {\int }_0^{\epsilon }{m}^{\pi }(x)dx}=\underset{A:\lambda (A)=\epsilon }{\sup }{\displaystyle {\int }_A{f}^{\pi }(x)dx}\end{array}$$

(15)

Thus $\forall \epsilon \in \left[0,\infty \right)$, we have ${\displaystyle {\int }_0^{\epsilon }{m}^g(x)dx}\le {\displaystyle {\int }_0^{\epsilon }{m}^{\pi }(x)dx}$:

Q.E.D (Lemma 2)

We are now in position to prove (11):

$$\begin{array}{l}H\left(m^g\right)=-{\displaystyle {\int }_0^{\infty }{m}^g(x)\log m^g(x)dx}\\ ={\displaystyle {\int }_0^{\infty }{m}^g(x){\displaystyle {\int }_0^{-\log m^g(x)}dy}}dx\\ ={\displaystyle {\int }_0^{\infty }\left({\displaystyle {\int }_0^{\infty }{m}^g(x)\mathbb{I}\left[y\le -\log m^g(x)\right]dy}\right)dx}\\ ={\displaystyle {\int }_0^{\infty }\left({\displaystyle {\int }_{\inf \left\{x:-\log m^g(x)\ge y\right\}}^{\infty }{m}^g(x)dx}\right)dy}\\ \stackrel{Lemma2}{\ge }{\displaystyle {\int }_0^{\infty }\left({\displaystyle {\int }_{\inf \left\{x:-\log m^g(x)\ge y\right\}}^{\infty }{m}^{\pi }(x)dx}\right)dy}\\ =-{\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\log m^g(x)dx}\\ \stackrel{(a)}{\ge }-{\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\log m^{\pi }(x)dx}\\ =H(m^{\pi })\end{array}$$

(16)

where $\mathbb{I}[.]$ denotes the indicator function, and (a) follows from $\forall x>0$, $-\log x\ge 1-x$, that is:

$$\begin{array}{l}-{\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\log m^g(x)dx}+{\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\log m^{\pi }(x)dx}\\ =-{\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\log \left(\frac{m^g(x)}{m^{\pi }(x)}\right)dx}\\ \ge {\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)\left(1-\frac{m^g(x)}{m^{\pi }(x)}\right)dx}\\ ={\displaystyle {\int }_0^{\infty }{m}^{\pi }(x)dx}-{\displaystyle {\int }_0^{\infty }{m}^g(x)dx}\stackrel{Lemma2}{=}0\end{array}$$

(17)

In the above proof, we adopt the convention $0\cdot \infty =0$.

Q.E.D (Proposition 1)

Now the proof of Proposition 1 has been completed. To finish the proof of Theorem 1, we have to remove the conditions of continuity and uniform boundedness imposed in Proposition 1. This can be easily accomplished by approximating $p(x|y)$ by a sequence of functions $\left\{f_n(x|y)\right\}$, $n=1,2,\cdots$, which satisfy these conditions. The remaining proof is omitted here, since it is exactly the same as the last part of the proof for Theorem 1 in [19].

Q.E.D (Theorem 1)

Example 1: Consider an additive noise model:

$$X=\phi \left(Y\right)+\xi$$

(18)

where $\xi$ is an additive noise that is independent of $Y$. In this case, we have $p\left(x|y\right)=p_{\xi }\left(x-\phi (y)\right)$, where $p_{\xi }(.)$ denotes the noise PDF. It is clear that $p\left(x|y\right)$ is generalized uniformly dominated, with dominant alignment $\pi (y)=\phi (y)$. According to Theorem 1, we have $H\left(X-\phi (Y)\right)\le H\left(X-g(Y)\right)$. In fact, this result can also be proved by:

$$\begin{array}{l}H\left(X-\phi (Y)\right)=H\left(\xi \right)\\ \stackrel{(b)}{\le }H\left(\xi +\left(\phi (Y)-g(Y)\right)\right)\\ =H\left(X-g(Y)\right)\end{array}$$

(19)

where (b) comes from the fact that $\xi$ and $\left(\phi (Y)-g(Y)\right)$ are independent (For independent random variables $X$ and $Y$, the inequality $H(X+Y)\ge \max \left\{H(X),H(Y)\right\}$ holds). In this example, the conditional PDF $p\left(x|y\right)$ is, obviously, not necessarily CSUM.

Example 2: Suppose the joint PDF of random variables $X$, $Y$ ($X,Y\in ℝ$) is:

$$p\left(x,y\right)=\exp \left(1-x\exp (y)\right)$$

(20)

where $y\ge 0$, $x\exp (y)\ge 1$. Then the conditional PDF $p\left(x|y\right)$ will be:

$$\begin{array}{l}p\left(x|y\right)=\frac{p\left(x,y\right)}{p\left(y\right)}\\ =\frac{\exp \left(1-x\exp (y)\right)}{{\displaystyle {\int }_{\exp (-y)}^{\infty }\exp \left(1-x\exp (y)\right)dx}}\\ =\exp \left(1-x\exp (y)+y\right)\end{array}$$

(21)

One can easily verify that the above conditional PDF is non-symmetric but generalized uniformly dominated, with dominant alignment $\pi (y)=\exp (-y)$ (the ε-volume dominant support of $p(x+\pi (y)|y)$ is $D_{\epsilon }=\left[0,\epsilon \right]$). By Theorem 1, the function $\exp (-y)$ is the minimizer of error entropy.

3. Non-Uniqueness of Unbiased MEE Estimation

Because entropy is shift-invariant, the MEE estimator is obviously non-unique. In practical applications, in order to yield a unique solution, or to meet the desire for small error values, the MEE estimator is usually restricted to be unbiased, that is, the estimation error is restricted to be zero-mean [15]. The question of interest in this paper is whether the unbiased MEE estimator is unique. In [20], it has been shown that, for the singular case (in which the error entropy approaches minus infinity), the unbiased MEE estimation may yield non-unique (even infinitely many) solutions. In the following, we present two examples to show that this result still holds even for nonsingular case.

Example 3: Let the joint PDF of $X$ and $Y$ ($X,Y\in ℝ$) be a mixed-Gaussian density [20]:

$$p\left(x,y\right)=\frac{1}{4\pi \sqrt{1-{\rho }^2}}\left\{\exp \left\{\frac{-\left(y^2-2\rho y(x-\mu )+{(x-\mu )}^2\right)}{2\left(1-{\rho }^2\right)}\right\}+\exp \left\{\frac{-\left(y^2+2\rho y(x+\mu )+{(x+\mu )}^2\right)}{2\left(1-{\rho }^2\right)}\right\}\right\}$$

(22)

where $\mu >0$, $0\le \left|\rho \right|<1$. The conditional PDF of $X$ given $Y$ will be:

$$p\left(x|y\right)=\frac{1}{2\sqrt{2\pi \left(1-{\rho }^2\right)}}\left\{\exp \left(-\frac{{\left(x-\mu -\rho y\right)}^2}{2\left(1-{\rho }^2\right)}\right)+\exp \left(-\frac{{\left(x+\mu +\rho y\right)}^2}{2\left(1-{\rho }^2\right)}\right)\right\}$$

(23)

$\forall y\in ℝ$, $p\left(x|y\right)$ is symmetric around zero (but not unimodal in x). It can be shown that for some values of $\mu$, $\rho$, the MEE estimator of $X$ based on $Y$ does not equal zero (see [20], Example 3). In these cases, the MEE estimator will be non-unique, even if the error’s PDF is restricted to zero-mean (unbiased) distribution. This can be proved as follows:

Let $g^{*}$ be an unbiased MEE estimator of $X$ based on $Y$. Then $-g^{*}$ will also be an unbiased MEE estimator, because:

$$\begin{array}{l}{p}^{g^{*}}(x)={\displaystyle {\int }_{ℝ}p(x+g^{*}(y)|y)dF(y)}\\ \stackrel{(c)}{=}{\displaystyle {\int }_{ℝ}p(-x-g^{*}(y)|y)dF(y)}\\ =p^{-g^{*}}(-x)\end{array}$$

(24)

where (c) comes from the fact that $p\left(x|y\right)$ is symmetric around zero, and further:

$$\begin{array}{l}H\left(p^{g^{*}}\right)=-{\displaystyle {\int }_{-\infty }^{\infty }{p}^{g^{*}}(x)\log p^{g^{*}}(x)dx}\\ =-{\displaystyle {\int }_{-\infty }^{\infty }{p}^{-g^{*}}(-x)\log p^{-g^{*}}(-x)dx}\\ =-{\displaystyle {\int }_{-\infty }^{\infty }{p}^{-g^{*}}(x)\log p^{-g^{*}}(x)dx}\\ =H\left(p^{-g^{*}}\right)\end{array}$$

(25)

If the unbiased MEE estimator is unique, then we have $g^{*}=-g^{*}\Rightarrow g^{*}=0$, which contradicts the fact that $g^{*}$ does not equal zero. Therefore, the unbiased MEE estimator must be non-unique. Obviously, the above result can be extended to more general cases. In fact, we have the following proposition.

Proposition2: The unbiased MEE estimator will be non-unique if the conditional PDF $p\left(x|y\right)$ satisfies:

(1)

Symmetric in $x\in {ℝ}^n$ around the conditional mean $\mu (y)=E\left[X|Y=y\right]$ for each $y\in {ℝ}^m$;

(2)

There exists a function $\zeta :{ℝ}^m\to {ℝ}^n$ such that $\zeta (Y)\ne \mu (Y)$, $H\left(X-\zeta (Y)\right)\le H\left(X-\mu (Y)\right)$.

Proof: Similar to the proof presented above (Omitted).

In the next example, we show that, for some particular situations, there can be even infinitely many unbiased MEE estimators.

Example 4: Suppose$Y$ is a discrete random variable with Bernoulli distribution:

$$\mathrm{Pr}(Y=0)=\mathrm{Pr}(Y=1)=0.5$$

(26)

The conditional PDF of $X$ given $Y$ is (see Figure 2):

$$p\left(x|Y=0\right)=\{\begin{array}{l}\frac{1}{2a}if\left|x\right|\le a\\ 0other\end{array}$$

$$p\left(x|Y=1\right)=\{\begin{array}{l}\frac{1}{a}if\left|x\right|\le \frac{1}{2}a\\ 0other\end{array}$$

where $a>0$. Note that the above conditional PDF is uniformly dominated in $x$.

Figure 2. Conditional PDF of $X$ given $Y$: (a) $p(x|Y=0)$, (b) $p(x|Y=1)$.

Figure 2. Conditional PDF of $X$ given $Y$: (a) $p(x|Y=0)$, (b) $p(x|Y=1)$.

Given an estimator $\widehat{X}=g(Y)$, the error’s PDF will be:

$$p^g(x)=\frac{1}{2}\left\{p\left(x+g(0)|Y=0\right)+p\left(x+g(1)|Y=1\right)\right\}$$

(27)

Let $g$ be an unbiased estimator, then ${\displaystyle {\int }_{-\infty }^{\infty }x{p}^g(x)dx}=0$, and hence $g(0)=-g(1)$. In the following, we assume $g(0)\ge 0$ (due to symmetry, one can obtain similar results for $g(0)<0$), and consider three cases:

Case 1: $0\le g(0)\le \frac{1}{4}a$. In this case, the error PDF is:

$$p^g(x)=\{\begin{array}{l}\frac{1}{4a},-a-g(0)\le x<-\frac{a}{2}+g(0)\\ \frac{3}{4a},-\frac{a}{2}+g(0)\le x\le \frac{a}{2}+g(0)\\ \frac{1}{4a},\frac{a}{2}+g(0)<x\le a-g(0)\end{array}$$

(28)

Then the error entropy can be calculated as:

$$H\left(p^g(x)\right)=-\frac{3}{4}\log \left(3\right)+\log \left(4a\right)$$

(29)

Case 2: $\frac{1}{4}a<g(0)\le \frac{3}{4}a$. In this case, we have:

$$p^g(x)=\{\begin{array}{l}\frac{1}{4a},-a-g(0)\le x<-\frac{a}{2}+g(0)\\ \frac{3}{4a},-\frac{a}{2}+g(0)\le x\le a-g(0)\\ \frac{1}{2a},a-g(0)<x\le \frac{a}{2}+g(0)\end{array}$$

(30)

And hence:

$$H\left(p^g(x)\right)=\left(-\frac{9}{8}+\frac{3g(0)}{2a}\right)\log \left(3\right)+\left(\frac{1}{4}-\frac{g(0)}{a}\right)\log \left(2\right)+\log \left(4a\right)$$

(31)

Case 3: $g(0)>\frac{3}{4}a$. In this case:

$$p^g(x)=\{\begin{array}{l}\frac{1}{4a},-a-g(0)\le x\le a-g(0)\\ \frac{1}{2a},-\frac{a}{2}+g(0)\le x\le \frac{a}{2}+g(0)\end{array}$$

(32)

Thus:

$$H\left(p^g(x)\right)=-\frac{1}{2}\log \left(2\right)+\log \left(4a\right)$$

(33)

One can easily verify that the error entropy achieves its minimum value when $0\le g(0)\le a/4$ (the first case). There are, therefore, infinitely many unbiased estimators that minimize the error entropy.

4. Conclusion

Two issues involved in the minimum error entropy (MEE) estimation have been studied in this work. The first issue is about which estimator minimizes the error entropy. In general there is no explicit expression for the MEE estimator unless some constraints on the conditional distribution are imposed. In the past, several researchers have shown that, if the conditional density is conditionally symmetric and unimodal (CSUM), then the conditional mean (or median) will be the MEE estimator. We extend these results to a more general case, and show that if the conditional densities are generalized uniformly dominated (GUD), then the dominant alignment will minimize the error entropy. The second issue is about the non-uniqueness of the unbiased MEE estimation. It has been shown in a recent paper that for the singular case (in which the error entropy approaches minus infinity), the unbiased MEE estimation may yield non-unique (even infinitely many) solutions. In this work, we show by examples that this result still holds even for nonsingular case.