On the Smoothed Minimum Error Entropy Criterion

Abstract

Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel.

1. Introduction

The principles and methods in Shannon’s information theory have been widely applied in statistical estimation, filtering, and learning problems [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]. The learning process is essentially a procedure of information processing with the goal of decreasing data redundancy in the presence of uncertainty and encoding the data into a model, and hence it is intrinsically related to information theory. An information theoretic description of learning processes was given in [10], where learning is defined as a process in which the system’s subjective entropy or, equivalently, its missing information decreases in time. The mathematical concept of information was also brought to biologically plausible information processing [11]. In addition, a unifying framework for information theoretic learning (ITL) has been presented in [18].

From a statistical viewpoint, learning can be thought of as approximating the a posteriori distribution of the targets given a set of examples (training data). Figure 1 shows a general scheme of supervised machine learning, where the desired system output Y (the teacher) is assumed to be related to the input signal X through a conditional probability density function (PDF) $p_{Y|X}\left(y|x\right)$, and the learning machine (model) is represented by a parametric mapping g (.,W), where $W\in {ℝ}^d$ denotes a parameter vector that needs to be estimated.

Figure 1. A general scheme of supervised machine learning.

Figure 1. A general scheme of supervised machine learning.

The learning goal is then to adapt the parameter W such that the discrepancy between the model output $\widehat{Y}=g\left(X,W\right)$ and the desired output Y is minimized. How to measure the discrepancy (or model mismatch) is a key aspect in learning. One can use a statistical descriptor of the error ($E=Y-\widehat{Y}$) distribution as the measure of discrepancy. The most popular descriptors are the second order moments (variance, correlation, etc.), which combined with the Gaussian assumption, in general leads to mathematically convenient and analytically tractable optimal solutions. A typical example is the mean square error (MSE) criterion in least-squares regression. However, the second order statistics as optimality criteria may perform poorly especially in nonlinear and non-Gaussian (e.g., heavy-tail or finite range distributions) situations. Recently, the error entropy, as an information theoretic alternative to MSE, has been successfully applied in supervised adaptive system training [12,13,14,15,16,17]. The minimum error entropy (MEE) criterion usually outperforms MSE criterion in many realistic scenarios, since it captures higher-order statistics and information content of signals rather than simply their energy. Under the MEE criterion, the optimal weight in Figure 1 will be:

$$\begin{array}{l}{W}^{*}=\underset{W\in {ℝ}^d}{\arg \min }H\left(E\right)\\ =\underset{W\in {ℝ}^d}{\arg \min }-{\displaystyle \int p_E(x)\log p_E(x)dx}\\ =\underset{W\in {ℝ}^d}{\arg \min }{\rm E}\left[-\log p_E(E)\right]\end{array}$$

(1)

where H (E) denotes the Shannon entropy of the error E, p_E(.) denotes the error PDF, and $\text{Ε}$ denotes the expectation operator. Throughout this paper, “log” denotes the natural logarithm. The formulation (1) can be generalized to other entropy definitions, such as the α-order Renyi entropy [18]. Since entropy is shift invariant, the error PDF p_E(.) is in general restricted to zero mean in practice.

The learning machine in Figure 1 can also be a nonparametric and universal mapping g(.). Familiar examples include the support vector machine (SVM) [19,20] and kernel adaptive filtering [21]. In this case, the hypothesis space for learning is in general a high (possibly infinite) dimensional reproducing kernel Hilbert space (RKHS) $H$, and the optimal mapping under MEE criterion is:

$$g^{*}=\underset{g\in H}{\arg \min }-{\displaystyle \int p_E(x)\log p_E(x)dx}$$

(2)

To implement the MEE learning, we should evaluate the error entropy. In practice, however, the error distribution is usually unknown, and the analytical evaluation of error entropy is not possible. Thus we have to estimate the error entropy from the samples. One simple way is to estimate the error PDF based on available samples, and plug the estimated PDF directly into the entropy definition to obtain the entropy estimator. In the literature there are many techniques for estimating the PDF of a random variable based on its sample data. In ITL, the most widely used approach is the Parzen windowing (or kernel density estimation) [22]. By Parzen windowing, the estimated error PDF is:

$${\widehat{p}}_E(x)=\frac{1}{N}{\displaystyle \sum _{i=1}^N{\kappa }_{\lambda }\left(x-e_i\right)}$$

(3)

where ${\kappa }_{\lambda }(x)=(1/\lambda )\kappa \left(x/\lambda \right)$ is the kernel function with smoothing factor (or kernel size) λ > 0, $\kappa$ is a continuous density, and ${\left\{e_i\right\}}_{i=1}^N$ are error samples which are assumed to be independent and identically distributed (i.i.d.). As sample number N → ∞, the estimated PDF will uniformly converge (with probability 1) to the true PDF convolved with the kernel function, that is

$${\widehat{p}}_E(x)\stackrel{N\to \infty }{\to }\left(p_E*{\kappa }_{\lambda }\right)(x)$$

(4)

where * denotes the convolution operator. Then, by the Parzen windowing approach (with a fixed kernel function ${\kappa }_{\lambda }$), the estimated error entropy will converge almost surely (a.s.) to the entropy of the convolved density (see [22,23]). Thus, the actual entropy to be minimized is:

$$H\left(E+\lambda Z\right)=-{\displaystyle \int \left(p_E*{\kappa }_{\lambda }\right)(x)\log \left(p_E*{\kappa }_{\lambda }\right)(x)dx}$$

(5)

where Z denotes a random variable that is independent of X, Y, E, and has PDF $p_Z(x)=\kappa (x)$. Note that the PDF of the sum of two independent random variables equals the convolution of their individual PDFs. Here, we call the entropy H (E + λZ) the smoothed error entropy, and Z the smoothing variable. Under the smoothed MEE (SMEE) criterion, the optimal mapping in (2) becomes:

$$\begin{array}{l}{g}^{*}=\underset{g\in H}{\arg \min }H\left(E+\lambda Z\right)\\ =\underset{g\in H}{\arg \min }-{\displaystyle \int \left(p_E*{\kappa }_{\lambda }\right)(x)\log \left(p_E*{\kappa }_{\lambda }\right)(x)dx}\end{array}$$

(6)

Although SMEE is an actual learning criterion (as sample number N → ∞) in ITL, up to now its theoretical properties have been little studied. In this work, we study theoretically the SMEE criterion. The rest of the paper is organized as follows: in Section 2, we present some basic properties of the SMEE criterion. In Section 3, we investigate how the smoothing factor λ affects the optimal solution. Finally in Section 4, we give the conclusion.

2. Some Basic Properties of SMEE Criterion

In this section, some basic properties of SMEE criterion are presented. We assume from now on, without explicit mention, that the learning machine is a nonparametric and universal mapping g(.). In addition, the input vector X belongs to an m-dimensional Euclidean space, $X\in {ℝ}^m$, and for simplicity, the output Y is assumed to be a scalar signal, $Y\in ℝ$.

Property 1: Minimizing the smoothed error entropy will minimize an upper bound of the true error entropy H (E).

Proof: According to the entropy power inequality (EPI) [1], we have:

$$\exp \left(2H\left(E+\lambda Z\right)\right)\ge \exp \left(2H\left(E\right)\right)+\exp \left(2H\left(\lambda Z\right)\right)$$

(7)

It follows that:

$$H\left(E\right)\le \frac{1}{2}\log \left(\exp \left(2H\left(E+\lambda Z\right)\right)-\exp \left(2H\left(\lambda Z\right)\right)\right)$$

(8)

Thus, minimizing the smoothed error entropy H (E + λZ) minimizes an upper bound of H (E).

Remark 1: Although this property does not give a precise result concerning SMEE vs. MEE, it suggests that minimizing the smoothed error entropy will constrain the true error entropy to small values.

Property 2: The smoothed error entropy is upper bounded by $\frac{1}{2}\log \left(2\pi e\left({\sigma }_E^2+{\lambda }^2{\sigma }_Z^2\right)\right)$, where ${\sigma }_E^2$ and ${\sigma }_Z^2$ denote the variances of E and Z, respectively, and this upper bound is achieved if and only if both E and Z are Gaussian distributed.

Proof: The first part of this property is a direct consequence of the maximum entropy property of the Gaussian distribution. The second part comes from Cramer’s decomposition theorem [24], which states that if X₁ and X₂ are independent and their sum X₁ + X₂ is Gaussian, then both X₁ and X₂ must also be Gaussian.

Remark 2: According to Property 2, if both E and Z are Gaussian distributed, then minimizing the smoothed error entropy will minimize the error variance.

Property 3: The smoothed error entropy has the following Taylor approximation around λ = 0:

$$H\left(E+\lambda Z\right)=H\left(E\right)+\frac{1}{2}{\lambda }^2{\sigma }_Z^{2}J\left(E\right)+o\left({\lambda }^2\right)$$

(9)

where o (λ²) denotes the higher-order infinitesimal term of Taylor expansion, and J (E) is the Fisher information of E, defined as:

$$J\left(E\right)=\mathrm{{\rm E}}\left[{\left(\frac{\partial }{\partial E}\log p_E(E)\right)}^2\right]$$

(10)

Proof: This property can be easily proved by using De Bruijn’s identity [25]. For any two independent random variables X₁ and X₂, $X_1,X_2\in ℝ$, De Bruijn’s identity can be expressed as

$${\frac{\partial }{\partial t}H\left(X_1+\sqrt{t}{X}_2\right)|}_{t=0}=\frac{1}{2}{\sigma }_{X_2}^{2}J\left(X_1\right)$$

(11)

where ${\sigma }_{X_2}^2$ denotes the variance of X₂ (Classical deBruijn identity assumes that X₂ is Gaussian. Here, we use a generalized deBruijn identity for arbitrary (not necessarily Gaussian) X₂ [25]). So, we have:

$${\frac{\partial }{\partial {\lambda }^2}H\left(E+\lambda Z\right)|}_{{\lambda }^2=0}=\frac{1}{2}{\sigma }_Z^{2}J\left(E\right)$$

(12)

and hence, we obtain the first-order Taylor expansion:

$$\begin{array}{l}H\left(E+\lambda Z\right)=H\left(E\right)+\left({\frac{\partial }{\partial {\lambda }^2}H\left(E+\lambda Z\right)|}_{{\lambda }^2=0}\right){\lambda }^2+o\left({\lambda }^2\right)\\ =H\left(E\right)+\frac{1}{2}{\lambda }^2{\sigma }_Z^{2}J\left(E\right)+o\left({\lambda }^2\right)\end{array}$$

(13)

Remark 3: By Property 3, with small λ, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. This result is very interesting since minimizing the smoothed error entropy will minimize a weighted sum of the true error entropy and the Fisher information of error. Intuitively, minimizing the error entropy tends to result in a spikier error distribution, while minimizing the Fisher information makes the error distribution smoother (smaller Fisher information implies a smaller variance of the PDF gradient). Therefore, the SMEE criterion provides a nice balance between the spikiness and smoothness of the error distribution. In [26], the Fisher information of error has been used as a criterion in supervised adaptive filtering.

Property 4: Minimizing the smoothed error entropy H (E + λZ) is equivalent to minimizing the mutual information between E + λZ and the input X, i.e., $\underset{g\in H}{\arg \min }H\left(E+\lambda Z\right)=\underset{g\in H}{\arg \min }I\left(E+\lambda Z;X\right)$.

Proof: Since mutual information $I\left(X;Y\right)=H\left(X\right)-H\left(X|Y\right)$, where $H\left(X|Y\right)$ denotes the conditional entropy of X given Y, we derive:

$$\begin{array}{l}\underset{g\in H}{\arg \min }I\left(E+\lambda Z;X\right)=\underset{g\in H}{\arg \min }\left\{H\left(E+\lambda Z\right)-H\left(E+\lambda Z|X\right)\right\}\\ =\underset{g\in H}{\arg \min }\left\{H\left(E+\lambda Z\right)-H\left(Y-g\left(X\right)+\lambda Z|X\right)\right\}\\ =\underset{g\in H}{\arg \min }\left\{H\left(E+\lambda Z\right)-H\left(Y+\lambda Z|X\right)\right\}\\ \stackrel{(a)}{=}\underset{g\in H}{\arg \min }H\left(E+\lambda Z\right)\end{array}$$

(14)

where (a) comes from the fact that the conditional entropy $H\left(Y+\lambda Z|X\right)$ does not depend on the mapping g(.).

Property 5: The smoothed error entropy is lower bounded by the conditional entropy $H\left(Y+\lambda Z|X\right)$, and this lower bound is achieved if and only if the mutual information I (E + λZ; X) = 0.

Proof: As$I\left(E+\lambda Z;X\right)=H\left(E+\lambda Z\right)-H\left(Y+\lambda Z|X\right)$, we have:

$$\begin{array}{l}H\left(E+\lambda Z\right)=H\left(Y+\lambda Z|X\right)+I\left(E+\lambda Z;X\right)\\ \stackrel{(b)}{\ge }H\left(Y+\lambda Z|X\right)\end{array}$$

(15)

where (b) is because of the non-negativeness of the mutual information I (E + λZ; X).

Remark 4: The lower bound in Property 5 depends only on the conditional PDF of Y given X and the kernel function ${\kappa }_{\lambda }$, and is not related to the learning machine. Combining Property 5 and Property 2, the smoothed error entropy satisfies the following inequalities:

$$H\left(Y+\lambda Z|X\right)\le H\left(E+\lambda Z\right)\le \frac{1}{2}\log \left(2\pi e\left({\sigma }_E^2+{\lambda }^2{\sigma }_Z^2\right)\right)$$

(16)

Property 6: Let$\rho \left(y|x,{\kappa }_{\lambda }\right)\triangleq p_{Y|X}\left(y|x\right)*{\kappa }_{\lambda }(y)$ be the smoothed conditional PDF of Y given X = x, where the convolution is with respect to y. If for every $x\in {ℝ}^m$, $\rho \left(y|x,{\kappa }_{\lambda }\right)$ is symmetric (not necessarily about zero) and unimodal in $y\in ℝ$, then the optimal mapping in (6) equals (almost everywhere):

$$g^{*}(x)={\displaystyle {\int }_{ℝ}y\rho \left(y|x,{\kappa }_{\lambda }\right)dy}+c$$

(17)

where $c\in ℝ$ is any constant.

Proof: The smoothed error PDF $\left(p_E*{\kappa }_{\lambda }\right)(.)$ can be expressed as:

$$\begin{array}{l}\left(p_E*{\kappa }_{\lambda }\right)(\xi )={\displaystyle {\int }_{ℝ}{p}_E(\tau ){\kappa }_{\lambda }(\xi -\tau )d\tau }\\ ={\displaystyle {\int }_{ℝ}\left({\displaystyle {\int }_{{ℝ}^m}{p}_{Y|X}(\tau +g(x)|x)dF(x)}\right){\kappa }_{\lambda }(\xi -\tau )d\tau }\\ ={\displaystyle {\int }_{{ℝ}^m}\left({\displaystyle {\int }_{ℝ}{p}_{Y|X}(\tau +g(x)|x){\kappa }_{\lambda }(\xi -\tau )d\tau }\right)dF(x)}\\ ={\displaystyle {\int }_{{ℝ}^m}\left(\rho \left(\xi +g(x)|x,{\kappa }_{\lambda }\right)\right)dF(x)}\end{array}$$

(18)

where F(x) denotes the distribution function of X. From (18), we see that the SMEE criterion can be formulated as a problem of shifting the components of a mixture of the smoothed conditional PDFs so as to minimize the entropy of the mixture. Then Property 6 holds since it has already been proved in [27] that, if all components (conditional PDFs) in the mixture are symmetric and unimodal, the conditional mean (or median) will minimize the mixture entropy. The constant c is added since the entropy is shift-invariant (in practice we usually set c = 0).

3. How Smoothing Factor Affects the Optimal Solution

The smoothing factor λ is an important parameter in SMEE criterion, which controls the smoothness of the performance surface. In the following, we will investigate how the smoothing factor affects the optimal solution (optimal mapping) g*(.).

When λ = 0, the smoothed error entropy becomes the true error entropy, and the SMEE criterion reduces to the original MEE criterion. When λ > 0, the two criteria are different and, in general have different solutions. However, in some situations, for any λ, the SMEE criterion yields the same solution as the MEE criterion.

Proposition 1: If the desired output Y is related to the input signal X through the nonlinear regression model Y = f(X) + N, where f is an unknown nonlinear mapping, and N is an additive noise that is independent of X and Z, then for any λ, the optimal solution under SMEE criterion is:

$$g^{*}(x)=f(x)+c$$

(19)

Proof: For any mapping $g\in \mathscr{H}$, and any λ, we have:

$$\begin{array}{l}H\left(E+\lambda Z\right)=H\left(f(X)+N-g(X)+\lambda Z\right)\\ =H\left(\left[f(X)-g(X)\right]+\left[N+\lambda Z\right]\right)\\ =H\left(U+V\right)\\ \ge H\left(U+V|U\right)\\ =H\left(V|U\right)\\ \stackrel{\text{(c)}}{=}H\left(V\right)\end{array}$$

(20)

where U = f(X) − g(X), V = N + λZ, and (c) comes from the fact that U and V are independent. The equality in (20) holds if and only if U is δ distributed, that is, U is a constant. This can be easily proved as follows.

If U is a constant, the equality in (20) will hold. Conversely, if the equality in (20) holds, we can prove that U is a constant. Actually, in this case, U and U + V are independent, and hence, $\mathrm{E}\left[\left(U+V\right)U\right]=\mathrm{E}\left[U+V\right]\mathrm{E}\left[U\right]$. It follows that:

$$\begin{array}{l}\mathrm{E}\left[\left(U+V\right)U\right]=\mathrm{E}\left[U+V\right]\mathrm{E}\left[U\right]\\ \iff \mathrm{E}\left[U^2\right]+\mathrm{E}\left[UV\right]={\left(\mathrm{E}\left[U\right]\right)}^2+\mathrm{E}\left[U\right]\mathrm{E}\left[V\right]\\ \iff \mathrm{E}\left[U^2\right]+\mathrm{E}\left[U\right]\mathrm{E}\left[V\right]={\left(\mathrm{E}\left[U\right]\right)}^2+\mathrm{E}\left[U\right]\mathrm{E}\left[V\right]\\ \iff \mathrm{E}\left[U^2\right]={\left(\mathrm{E}\left[U\right]\right)}^2\\ \iff \mathrm{E}\left[{\left(U-\mathrm{E}\left[U\right]\right)}^2\right]=0\end{array}$$

(21)

which implies that the variance of U is zero (i.e., U is a constant). Therefore we have g*(x) = f(x) + c.

Remark 5: Proposition 1 implies that for the nonlinear regression problem, the optimal solution under the SMEE criterion does not change with the smoothing factor provided that the additive noise is independent of the input signal.

If the unknown system (the conditional PDF) is not restricted to a nonlinear regression model, under certain conditions the optimal solution of SMEE can still remain unchanged with the smoothing factor λ. Specifically, the following proposition holds.

Proposition 2: If the conditional PDF $p_{Y|X}\left(y|x\right)$ and the kernel function ${\kappa }_{\lambda }(y)$ are both symmetric (not necessarily about zero) and both unimodal in $y\in ℝ$, then for any λ, the SMEE criterion produces the same solution:

$$g^{*}(x)={\displaystyle {\int }_{ℝ}y{p}_{Y|X}\left(y|x\right)dy}+c$$

(22)

Proof: By Property 6, it suffices to prove that the smoothed conditional PDF $\rho \left(.|x,{\kappa }_{\lambda }\right)$ is symmetric and unimodal. This is a well-known result and a simple proof can be found in [28].

Remark 6: Note that the optimal solution under the minimum error variance criterion is also given by (22). In particular, if setting c = 0, the solution (22) becomes the conditional mean, which corresponds to the optimal solution under the MSE criterion.

Proposition 3: Assume that the conditional PDF $p_{Y|X}\left(.|x\right)$ is symmetric (not necessarily unimodal) with uniformly bounded support, and the kernel function ${\kappa }_{\lambda }$ is a zero-mean Gaussian PDF with variance λ². Then, if smoothing factor λ is larger than a certain value, the optimal solution under the SMEE is still given by (22).

Proof: By Property 6, it is sufficient to prove that if smoothing factor λ is larger than a certain value, the smoothed conditional PDF $\rho \left(.|x,{\kappa }_{\lambda }\right)$ is symmetric and unimodal. Suppose without loss of generality that the conditional PDF $p_{Y|X}\left(.|x\right)$ is symmetric about zero with bounded support [-B,B], B > 0. Since kernel function ${\kappa }_{\lambda }(.)$ is a zero-mean Gaussian PDF with variance λ², the smoothed PDF $\rho \left(.|x,{\kappa }_{\lambda }\right)$ can be expressed as:

$$\begin{array}{l}\rho \left(y|x,{\kappa }_{\lambda }\right)=\frac{1}{\sqrt{\text{2}\pi }\lambda }{\displaystyle {\int }_{-\infty }^{\infty }\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)p_{Y|X}\left(\tau |x\right)d\tau }\\ =\frac{1}{\sqrt{\text{2}\pi }\lambda }{\displaystyle {\int }_{-B}^B\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)p_{Y|X}\left(\tau |x\right)d\tau }\\ \stackrel{(d)}{=}\frac{1}{\sqrt{\text{2}\pi }\lambda }{\displaystyle {\int }_0^B\left\{\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)+\exp \left(-\frac{{\left(y+\tau \right)}^2}{2{\lambda }^2}\right)\right\}{p}_{Y|X}\left(\tau |x\right)d\tau }\end{array}$$

(23)

where (d) follows from $p_{Y|X}\left(\tau |x\right)=p_{Y|X}\left(-\tau |x\right)$. Clearly, $\rho \left(y|x,{\kappa }_{\lambda }\right)$ is symmetric in y. Further, we derive:

$$\frac{\partial }{\partial y}\rho \left(y|x,{\kappa }_{\lambda }\right)=\frac{-\text{1}}{\sqrt{\text{2}\pi }{\lambda }^3}{\displaystyle {\int }_0^B\left\{\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)\left(y-\tau \right)+\exp \left(-\frac{{\left(y+\tau \right)}^2}{2{\lambda }^2}\right)\left(y+\tau \right)\right\}{p}_{Y|X}\left(\tau |x\right)d\tau }$$

(24)

If λ ≥ 2B, we have:

$$\{\begin{array}{l}\frac{\partial }{\partial y}\rho \left(y|x,{\kappa }_{\lambda }\right)\le 0fory\ge 0\\ \frac{\partial }{\partial y}\rho \left(y|x,{\kappa }_{\lambda }\right)\ge 0fory<0\end{array}$$

(25)

We give below a simple proof of (25). It suffices to consider only the case y ≥ 0 In this case, we consider two subcases:

(1)

y ≥ B: In this case, we have $\forall \tau \in \left[0,B\right]$, (y - τ) ≥ 0, and hence:

$$\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)\left(y-\tau \right)+\exp \left(-\frac{{\left(y+\tau \right)}^2}{2{\lambda }^2}\right)\left(y+\tau \right)\ge 0，y\in \left[B,\infty \right),\tau \in \left[0,B\right]$$

(26)

(1)

(2) 0 ≤ y < B: In this case, we have$\forall \tau \in \left[0,B\right]$, $0\le \left|y-\tau \right|\le y+\tau \le 2B\le \lambda$. Since$\forall \left|x\right|\le \lambda$:

$$\frac{\partial }{\partial x}\left\{\exp \left(-\frac{x^2}{2{\lambda }^2}\right)x\right\}=\exp \left(-\frac{x^2}{2{\lambda }^2}\right)\frac{{\lambda }^2-x^2}{{\lambda }^2}\ge 0$$

(27)

we have $\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)\left|y-\tau \right|\le \exp \left(-\frac{{\left(y+\tau \right)}^2}{2{\lambda }^2}\right)\left(y+\tau \right)$, and it follows easily that:

$$\exp \left(-\frac{{\left(y-\tau \right)}^2}{2{\lambda }^2}\right)\left(y-\tau \right)+\exp \left(-\frac{{\left(y+\tau \right)}^2}{2{\lambda }^2}\right)\left(y+\tau \right)\ge 0, y\in \left[0,B\right),\tau \in \left[0,B\right]$$

(28)

Combining (24), (26) and (28), we get $\forall y\ge 0$, $\frac{\partial }{\partial y}\rho \left(y|x,{\kappa }_{\lambda }\right)\le 0$. Therefore (25) holds, which implies that if λ ≥ 2B the smoothed PDF $\rho \left(.|x,{\kappa }_{\lambda }\right)$ is symmetric and unimodal, and this completes the proof of the proposition.

Remark 7: Proposition 3 suggests that under certain conditions, when λ is larger than a certain value, the SMEE criterion yields the same solution as the minimum error variance criterion. In the next proposition, a similar but more interesting result is presented for general cases where no assumptions on the conditional PDF and on the kernel function are made.

Proposition 4: When the smoothing factor λ → ∞, minimizing the smoothed error entropy will be equivalent to minimizing the error variance plus an infinitesimal term.

Proof: The smoothed error entropy can be rewritten as:

$$\begin{array}{l}H\left(E+\lambda Z\right)=H\left(\lambda \left[\frac{1}{\lambda }E+Z\right]\right)=H\left(\frac{1}{\lambda }E+Z\right)+\log \lambda \\ =H\left(Z+\sqrt{t}E\right)-\frac{1}{2}\log t\end{array}$$

(29)

where t = 1/λ². Since the term $-\frac{1}{2}\log t$ does not depend on learning machine, minimizing H (E + λZ) is equivalent to minimizing $H\left(Z+\sqrt{t}E\right)$, that is:

$$\underset{g\in H}{\min }H\left(E+\lambda Z\right)\iff \underset{g\in H}{\min }H\left(Z+\sqrt{t}E\right)$$

(30)

By De Bruijn’s identity (11):

$${\frac{\partial }{\partial t}H\left(Z+\sqrt{t}E\right)|}_{t=0}=\frac{1}{2}{\sigma }_E^{2}J\left(Z\right)$$

(31)

When λ is very large (hence t is very small):

$$H\left(Z+\sqrt{t}E\right)=H\left(Z\right)+\frac{t}{2}{\sigma }_E^{2}J\left(Z\right)+o\left(t\right)$$

(32)

Combining (30) and (32) yields:

$$\underset{g\in H}{\min }H\left(E+\lambda Z\right)\iff \underset{g\in H}{\min }\left({\sigma }_E^2+\frac{2}{J\left(Z\right)t}o\left(t\right)\right)$$

(33)

which completes the proof.

Remark 8: The above result is very interesting: when the smoothing factor λ is very large, minimizing the smoothed error entropy will be approximately equivalent to minimizing the error variance (or the mean square error if the error PDF is restricted to zero-mean). This result holds for any conditional PDF and any kernel function. A similar result can be obtained for the nonparametric entropy estimator based on Parzen windows. It was proved in [14] that in the limit, as the kernel size (the smoothing factor) tends to infinity, the entropy estimator approaches a nonlinearly scaled version of the sample variance.

4. Conclusions

Traditional machine learning methods mainly exploit second order statistics (covariance, mean square error, correlation, etc.). The optimality criteria based on second order statistics are computationally simple, and optimal under linear and Gaussian assumptions. Although second order statistics are still prevalent today in the machine learning community and provide successful engineering solutions to most practical problems, it has become evident that this approach can be improved, especially when data possess non-Gaussian distributions (multi-modes, fat tails, finite range, etc.). In most situations, a more appropriate approach should be applied to capture higher order statistics or information content of signals rather than simple their energy. Recent studies suggest that the supervised machine learning can benefit greatly from the use of the minimum error entropy (MEE) criterion. To implement the MEE learning, however, one has to estimate the error entropy from the samples. In the limit (when sample size tends to infinity), the estimated error entropy by Parzen windowing converges to the smoothed error entropy, i.e., the entropy of the error plus an independent random variable with PDF equal to the kernel function used in Parzen windowing, so the smoothed error entropy is the actual entropy that is minimized in the MEE learning.

In this paper, we study theoretically the properties of the smoothed MEE (SMEE) criterion in supervised machine learning and, in particular, we investigate how the smoothing factor affects the optimal solution. Some interesting results are obtained. It is shown that when the smoothing factor is small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. In some special situations, the SMEE solution remains unchanged with increasing smoothing factor. In general cases, however, when the smoothing factor is very large, minimizing the smoothed error entropy will be approximately equivalent to minimizing the error variance (or the mean square error if the error distribution is restricted to zero-mean), regardless of the conditional PDF and the kernel function.

This work does not address the learning issues when the number of samples is limited. In this case, the problem becomes much more complex since there is an extra bias in the entropy estimation. We leave this problem open for future research. The results obtained in this paper, however, are still very useful since they provide theoretical solutions to which the empirical solution (with finite data) must approximate.