Relative Entropy Derivative Bounds

Abstract

We show that the derivative of the relative entropy with respect to its parameters is lower and upper bounded. We characterize the conditions under which this derivative can reach zero. We use these results to explain when the minimum relative entropy and the maximum log likelihood approaches can be valid. We show that these approaches naturally activate in the presence of large data sets and that they are inherent properties of any density estimation process involving large numbers of random variables.

1. Introduction

Given a large ensemble of i.i.d.random variables, all of them generated according to some common density function, the asymptotic equipartition principle [1] guarantees that only a fraction of the possible ensembles, which is called the typical set, gathers almost all the probability ([2] [p. 226]). This fact opens interesting possibilities: What if the properties of the typical set impose conditions on any estimation process, such that the parameter search is focused on just a small subset of the available parameter set? We explore this using generalizations of the Shannon differential entropy [1] and the Fisher information [3,4,5] under typical set considerations. There are other works that take into account both concepts [6,7,8]. However, this work focuses on something new: defining algebraic bounds on the behavior of the derivative of the relative entropy with respect to their parameters. Furthermore, we characterize the conditions under which seeking for the extreme of this expression is a valid approach. Most importantly, we prove that these conditions automatically activate if large data sets are used. We finish this work discussing the relation between these bounds and the density function estimation problem. Under the conditions that activate these bounds, we characterize when the minimum relative entropy and maximum log likelihood approaches render the same solution. Furthermore, we show that these equivalences become true by the sole fact of having a large number of random variables.

2. Known Density Functions Case

Let us assume for the ensuing discussion that there exists a known source density function, $f_X$, with support, $\mathcal{S}$. We also know a set of i.i.d. samples, ${\left\{x\right\}}_n\equiv \{x_1,\dots ,x_n\}$, that was generated by $f_X$. There is also another density function, $f_{Y|\mathbf{\theta }}\left(y\right)$, with the same support, $\mathcal{S}$. This density function is indexed by the parameter vector, $\mathbf{\theta }\in \Theta \subseteq {\mathbb{R}}^m$.

The first part of this work deals with this case, where all the density functions are known. The second part of this work studies the density function estimation problem, where the source density function, $f_X$, is not known.

3. Mixed Entropy Derivative: A Proxy for the Relative Entropy Derivative

We present the following definition:

Definition 1 

The relative entropy ([2][p. 231]), also called Kullback-Leibler divergence, is defined by:

$$D_{RE}\left(f_X\parallel f_{Y|\mathbf{\theta }}\right)\equiv {\int }_{\mathcal{S}}{f}_X\left(u\right)ln\frac{f_X\left(u\right)}{f_{Y|\mathbf{\theta }}\left(u\right)}du$$

(1)

Notice that ${lim}_{u\to 0^{+}}ulnu=0$. Even though the definition is valid for any pair of density functions, we are using those useful for the ensuing analysis. The domain of this integral, and that of all the integrals in this work, corresponds to the support, $\mathcal{S}$.

Definition 2 

The mixed entropy, $h_M(f_X,f_{Y|\mathbf{\theta }})$ of two density functions is defined by:

$$h_M(f_X,f_{Y|\mathbf{\theta }})\equiv -{\int }_{\mathcal{S}}{f}_X\left(u\right)ln{f}_{Y|\mathbf{\theta }}\left(u\right)du$$

(2)

when the integral exists.

Definition 3 

The Shannon differential entropy [1] is defined by:

$$h_S\left(f_X\right)\equiv -{\int }_{\mathcal{S}}{f}_X\left(u\right)ln{f}_X\left(u\right)du$$

(3)

From these definitions, when $f_{Y|\mathbf{\theta }}=f_X$, from a functional point of view, the mixed entropy, $h_M(f_X,f_{Y|\mathbf{\theta }})$, equals the differential entropy, $h_S\left(f_X\right)$

From examination of the relative entropy definition:

$$\begin{array}{ccc}\hfill D_{KL}\left(f_X\parallel f_{Y|\mathbf{\theta }}\right)& =& \int f_X\left(u\right)ln{f}_X\left(u\right)du-\int f_X\left(u\right)ln{f}_{Y|\mathbf{\theta }}\left(u\right)du\hfill \\ \mathrm{(4)}& & =& -h_S\left(f_X\right)+h_M(f_X,f_{Y|\mathbf{\theta }})\hfill \end{array}$$

Hence:

$$\frac{\partial D_{KL}\left(f_X\parallel f_{Y|\mathbf{\theta }}\right)}{\partial {\theta }_k}=\frac{h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}$$

(5)

In the previous equation it was assumed that the $\frac{\partial }{\partial {\theta }_k}$ are defined in the interior of Θ for $k\in \{1,\dots ,m\}$. In this work, where we assume that $f_X$ does not depend on any ${\theta }_k$, we use extensively the fact that the relative entropy derivative is equal to the mixed entropy derivative. Hence, in the following, we focus on studying the properties of the mixed entropy in order to be able to say something about the properties of the relative entropy derivatives.

4. Mixed Entropy Typical Set

An interesting insight related to sequences of i.i.d. random variables, discovered by Shannon [1], is the usefulness of the weak law of large numbers to characterize large ensembles of random variables. In the context of this work, this law implies:

Defining

Definition 4 

The likelihood, $f_{{\left\{Y\right\}}_n|\mathbf{\theta }}$, is defined by:

$$f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\equiv \prod _{k=1}^n{f}_{Y|\mathbf{\theta }}\left(x_k\right)$$

(6)

we can state:

Theorem 1 

For any positive real number, ε, and fixed parameter vector, $\mathbf{\theta }$:

$$\mathbb{P}\left\{\left|-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}-h_M(f_X,f_{Y|\mathbf{\theta }})\right|\le \epsilon \right\}\underset{n\to \infty }{\to }1$$

(7)

Proof: Use that the random variables are i.i.d., and the weak law of large numbers. ☐

Hence, it makes sense to define the following set ([2][p. 226]):

Definition 5 

For any positive real number, ε, fixed parameter vector, $\mathbf{\theta }$, and any $n\ge 1$, the mixed entropy typical set, ${\mathcal{M}}_{\epsilon }^n$, with respect to the density function, $f_X$, is defined by:

$${\mathcal{M}}_{\epsilon }^n=\left\{{\left\{x\right\}}_n\in {\mathcal{S}}^n:\left|-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}-h_M(f_X,f_{Y|\mathbf{\theta }})\right|\le \epsilon \right\}$$

(8)

Furthermore, using the weak law of large numbers, it is possible to prove:

Lemma 1 

For any positive real number, ε, and any parameter vector, $\mathbf{\theta }$, it is true that:

$$\mathbb{P}\left\{{\left\{x\right\}}_n\in {\mathcal{M}}_{\epsilon }^n\right\}\underset{n\to \infty }{\to }1$$

(9)

Proof: Use the weak law of large numbers in the mixed entropy typical set definition. ☐

This proves that for large values of n, almost all the probability is contained by the mixed entropy typical set and that the probability of being outside this set becomes negligible.

Then, it is possible to prove that:

Lemma 2 

For any positive real number, ε, and any fixed parameter vector, $\mathbf{\theta }$, it exists $n_0\equiv n_0(\epsilon ,\mathbf{\theta })$, such that for all $n>n_0$, the total probability of the typical set is lower bounded by:

$$1-\epsilon \le \mathbb{P}\left\{{\mathcal{M}}_{\epsilon }^n\right\}$$

(10)

Proof: Check ([2][p. 226], [9][p. 118]) for details. ☐

5. Micro-Differences in Mixed Entropy Typical Sets

Assuming that ${\left\{x\right\}}_n\in {\mathcal{M}}_{\epsilon }^n$ and from the definition of mixed entropy typical set:

$$ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}\in ]-n{h}_M(f_X,f_{Y|\mathbf{\theta }})-n\epsilon ,-n{h}_M(f_X,f_{Y|\mathbf{\theta }})+n\epsilon [$$

(11)

The width of this interval is $2\epsilon n$. This allows us to compare $ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}$ with $-n{h}_M(f_X,f_{Y|\mathbf{\theta }})$, the value that is obtained in the limit thanks to the weak law of large numbers. Their ratio is:

$$\left|\frac{2\epsilon n}{-n{h}_M(f_X,f_{Y|\mathbf{\theta }})}\right|=\left|\frac{2\epsilon }{h_M(f_X,f_{Y|\mathbf{\theta }})}\right|$$

(12)

In other words, given that ε can be chosen as small as needed, the range of values that $ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}$ can take may be indistinguishable from $-n{h}_M(f_X,f_{Y|\mathbf{\theta }})$, with high probability. This is another way of stating the weak law of large numbers.

The previous facts allow us to present the following definition:

Definition 6 

When ${\left\{x\right\}}_n\in {\mathcal{M}}_{\epsilon }^n$, its associated micro-difference is defined by the following function:

$$\delta \equiv \delta (f_X,n,{\left\{x\right\}}_n,f_{Y|\mathbf{\theta }},\mathbf{\theta },\epsilon )\in ]0,1[$$

(13)

such that:

$$ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}=-n(h_M(f_X,f_{Y|\mathbf{\theta }})+\epsilon )+2\epsilon n\delta$$

(14)

hence:

$$\delta =\frac{1}{2\epsilon n}\left(ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}+n(h_M(f_X,f_{Y|\mathbf{\theta }})+\epsilon )\right)$$

(15)

The analysis performed in the preceding paragraphs shows that the behavior of the micro-differences might be negligible in the case of large values of n. However, the following sections of this work deal with the the derivative of the micro-difference value with respect to the parameters; so, we cannot neglect it, and we need to study its behavior.

From Equation (14):

$$\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}}{\partial {\theta }_k}=-n\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}+2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}$$

(16)

for all $k\in [1,\dots ,m]$. Thus:

$$\begin{array}{ccc}\hfill 2\epsilon \frac{\partial \delta }{\partial {\theta }_k}& =& \frac{1}{n}\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}}{\partial {\theta }_k}+\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\hfill \end{array}$$

(17)

$$\begin{array}{ccc}& =& \frac{\partial }{\partial {\theta }_k}\left(\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}+h_M(f_X,f_{Y|\mathbf{\theta }})\right)\hfill \end{array}$$

(18)

$$\begin{array}{ccc}& =& -\frac{\partial }{\partial {\theta }_k}\left(-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}-h_M(f_X,f_{Y|\mathbf{\theta }})\right)\hfill \end{array}$$

(19)

This last expression helps us to understand that the derivative of δ with respect to the parameters is related to changes experienced by quantities upper bounded by ε.

6. Mixed Information

We define:

Definition 7 

The mixed information, $i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k$, is defined by:

$$i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k\equiv {\int }_{{\mathcal{S}}^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}}{\partial {\theta }_k}\right)}^{2}du$$

(20)

with:

$$f_{{\left\{X\right\}}_n}\equiv \prod _{k=1}^n{f}_X\left(x_k\right)$$

(21)

when the integral exists.

7. Mixed Entropy Derivative Bounds

The main contribution of this work is the following pair of inequalities, which are obtained thanks to a combination of the mixed information and the mixed entropy.

Theorem 2 

Given a positive real number, ε, any parameter vector, $\mathbf{\theta }$, and an i.i.d. sequence with n elements, then the components of ${\nabla }_{\mathbf{\theta }}{h}_M(f_X,f_{Y|\mathbf{\theta }})$ comply with:

$$\frac{\sqrt{I_R^k}-\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k}}{\sqrt{I_L}}\le \left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\le \frac{\sqrt{I_R^k}+\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k}}{\sqrt{I_L}}$$

(22)

where:

$$\begin{array}{ccc}\hfill I_R^k& \equiv & {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|2\epsilon \frac{\partial \delta }{\partial {\theta }_k}\right|}^{2}du\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill I_L& \equiv & {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}du\hfill \end{array}$$

(24)

Proof: From the mixed information definition:

$$i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k=I_{{\mathcal{M}}_{\epsilon }^n}^k+I_{\sim {\mathcal{M}}_{\epsilon }^n}^k\ge I_{{\mathcal{M}}_{\epsilon }^n}^k>0$$

(25)

where:

$$I_{{\mathcal{M}}_{\epsilon }^n}^k\equiv {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}}{\partial {\theta }_k}\right)}^{2}du$$

(26)

and the symbol, ∼, denotes the complement set.

Replacing Equation (16) in Equation (26), it is obtained:

$$\begin{array}{ccc}\hfill I_{{\mathcal{M}}_{\epsilon }^n}^k& =& n^2{\left(\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right)}^2{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}du-2n\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}\left(2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}\right)du\hfill \\ \mathrm{(27)}& & & +{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left(2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}\right)}^{2}du\hfill \end{array}$$

Using the Cauchy-Bunyakovsky-Schwarz inequality:

$$\begin{array}{ccc}\hfill {\left|{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}\left(2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}\right)du\right|}^2& =& {\left|{\int }_{{\mathcal{M}}_{\epsilon }^n}\sqrt{f_{{\left\{X\right\}}_n}}\sqrt{f_{{\left\{X\right\}}_n}}\left(2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}\right)du\right|}^2\hfill \\ & \le & {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}du·{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|2\epsilon n\frac{\partial \delta }{\partial {\theta }_k}\right|}^{2}du\hfill \\ \mathrm{(28)}& & =& I_L·n^2{I}_R^k\hfill \end{array}$$

Therefore:

$$\begin{array}{ccc}\hfill I_{{\mathcal{M}}_{\epsilon }^n}^k& \ge & n^2{\left(\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right)}^2{I}_L-2n^2\left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\sqrt{I_L}\sqrt{I_R^k}+n^2{I}_R^k\hfill \\ \mathrm{(29)}& & =& n^2{\left(\left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\sqrt{I_L}-\sqrt{I_R^k}\right)}^2\hfill \end{array}$$

which implies:

$$\frac{1}{n^2}{i}_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k\ge {\left(\left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\sqrt{I_L}-\sqrt{I_R^k}\right)}^2$$

(30)

Hence, using the negative solution of the square root:

$$\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k}\ge -\left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\sqrt{I_L}+\sqrt{I_R^k}$$

(31)

Therefore:

$$\frac{\sqrt{I_R^k}-\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k}}{\sqrt{I_L}}\le \left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|$$

(32)

Taking into account the positive solution of Equation (30):

$$\left|\frac{\partial h_M(f_X,f_{Y|\mathbf{\theta }})}{\partial {\theta }_k}\right|\le \frac{\sqrt{I_R^k}+\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)}_k}}{\sqrt{I_L}}$$

(33)

☐

The previous result allows us to state the following remarks:

Remark 1 

This theorem states algebraic bounds on the derivative of the mixed entropy that are valid for any value of n.

Remark 2 

The expression:

$$I_R^k\equiv {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|2\epsilon \frac{\partial \delta }{\partial {\theta }_k}\right|}^{2}du={\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|-\frac{\partial }{\partial {\theta }_k}\left(-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}-h_M(f_X,f_{Y|\mathbf{\theta }})\right)\right|}^{2}du$$

(34)

refers to the accumulation of derivatives of very small differences. This differences become smaller and smaller as the weak law of large numbers is enacted.

8. Unknown Source Density Function, $f_X$

Now, we study what happens when the source density function, $f_X$, is not known.

It can be proven that $D_{KL}\left(f_X\parallel f_{Y|\mathbf{\theta }}\right)\ge 0$ ([2][p. 232]). Thus, from the corresponding definitions given at the start of this work:

$$h_M(f_X,f_{Y|\mathbf{\theta }})\ge h_S\left(f_X\right)$$

(35)

Therefore, the density estimation problem can be framed as the following optimization program:

$${\mathbf{\theta }}_{\circ }\equiv {\mathbf{\theta }}_{\circ }(f_X,f_{Y|\mathbf{\theta }})=arg\underset{\mathbf{\theta }}{min}{h}_M(f_X,f_{Y|\mathbf{\theta }})$$

(36)

Remark 3 

The solutions of this optimization program pose the following options:

• $h_M(f_X,f_{Y|{\mathbf{\theta }}_{\circ }})=h_S\left(f_X\right)$, hence $f_{Y|{\mathbf{\theta }}_{\circ }}=f_X$.
• $h_M(f_X,f_{Y|{\mathbf{\theta }}_{\circ }})>h_S\left(f_X\right)$, thus $f_{Y|{\mathbf{\theta }}_{\circ }}\ne f_X$. This happens when the estimator cannot implement the desired density function, which is beyond its reach. One way of checking whether the global minimum has been reached is to compare the mixed entropy value to that of the Shannon differential entropy. If they differ, the global minimum has not been reached yet.

According to this result, it is straightforward to search for parameters that make true the following expression:

$${\left.{\nabla }_{\mathbf{\theta }}{h}_M(f_X,f_{Y|\mathbf{\theta }})\right|}_{\mathbf{\theta }={\mathbf{\theta }}_{\circ }}=0$$

(37)

However, this is out of the limits in our density estimation problem, given that we do not have access to $f_X$. Thank to the weak law of large numbers:

$$\underset{n\to \infty }{lim}-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}=h_M(f_X,f_{Y|\mathbf{\theta }})$$

(38)

in probability. Thus, for sufficiently large values of n:

$$-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}\approx h_M(f_X,f_{Y|\mathbf{\theta }})$$

(39)

Hence, the optimization program posed in Equation (36) can be approximated by:

$${\mathbf{\theta }}_{\circ }\equiv {\mathbf{\theta }}_{\circ }(n,{\left\{x\right\}}_n,f_{Y|\mathbf{\theta }})=arg\underset{\mathbf{\theta }}{min}\left(-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)$$

(40)

or:

$${\mathbf{\theta }}_{\circ }\equiv {\mathbf{\theta }}_{\circ }(n,{\left\{x\right\}}_n,f_{Y|\mathbf{\theta }})=arg\underset{\mathbf{\theta }}{max}\left(\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)$$

(41)

which is the maximum log likelihood framework ([3][p. 261], [4][260], [5][p. 65]).

Remark 4 

As in the mixed entropy case, the maximum log likelihood framework can exhibit the following cases:

• The optimization program finds a parameter combination that makes the estimator equal to the unknown source density function, thus making true the following expression for a sufficiently large n:

  $$-\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|{\mathbf{\theta }}_{\circ }}\approx h_M(f_X,f_{Y|{\mathbf{\theta }}_{\circ }})\approx h_S\left(f_X\right)$$

  (42)
• The estimator does not find the desired target function. This happens when the family of functions that can be implemented by the estimator does not include the target function. It also happens when the maximum log likelihood program is riddled with multiple local minima, and the optimization process gets trapped in one of them. Only under very specific conditions, it is possible to obtain the global maximum of the maximum log likelihood problem [10]. In this case, we cannot use a comparison between the value of the log likelihood term and that of the Shannon differential entropy to determine the goodness of our solution, as we could do in the case of the mixed entropy, because calculating the latter is analogous to the very problem we are trying to solve.

Hence, it is natural to work with this approximated program and look for parameters that make the following expression true:

$${\left.{\nabla }_{\mathbf{\theta }}\left(\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mathbf{\theta }}\right)\right|}_{\mathbf{\theta }={\mathbf{\theta }}_{\circ }}=0$$

(43)

9. An Example: Mismatched Density Functions

9.1. Known Density Functions Case

Let us define a source density function:

$$f_X\left(x\right)=\lambda e^{-\lambda x}H\left(x\right)$$

(44)

with $x\in \mathbb{R}$ and $H\left(x\right)$, the Heaviside step function. This density function is completely defined by the parameter, λ. Its differential entropy is given by:

$$h_S\left(f_X\right)=1-ln\lambda =1+ln\frac{1}{\lambda }$$

(45)

We also have a data set, ${\left\{x\right\}}_n$, generated by the source density function.

The density function that depends on parameters is:

$$f_{Y|\mu ,\sigma }=\frac{1}{\sigma \sqrt{2\pi }}exp\left(-\frac{{\left(\mu -x\right)}^2}{2{\sigma }^2}\right)$$

(46)

with $x\in \mathbb{R}$, μ its mean and σ its standard deviation. Its differential entropy is given by:

$$h_S\left(f_{Y|\mu ,\sigma }\right)=ln\left(\sigma \sqrt{2\pi e}\right)=1+ln\sigma +ln\sqrt{\frac{2\pi }{e}}$$

(47)

which does not depend on the mean, μ.

9.2. Compliance with Theorem 2

We first analyze the bounds associated with parameter μ. For these density functions:

$$\frac{1}{n}ln{f}_{{\left\{Y\right\}}_n|\mu ,\sigma }=\frac{1}{n}\sum _{k=1}^{n}ln{f}_{Y|\mu ,\sigma }\left(x_k\right)=ln\left(\frac{1}{\sigma \sqrt{2\pi }}\right)-\frac{1}{2n{\sigma }^2}\sum _{k=1}^n{\left(\mu -x_k\right)}^2$$

(48)

Hence:

$$\frac{1}{n}\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mu ,\sigma }}{\partial \mu }=-\frac{1}{{\sigma }^2}\left(\mu -\frac{1}{n}\sum _{k=1}^n{x}_k\right)$$

(49)

We calculate the mixed information for μ:

$$\begin{array}{ccc}\hfill i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mu ,\sigma }\right)}_{\mu }& =& {\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mu ,\sigma }}{\partial \mu }\right)}^{2}du\hfill \\ & =& {\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{n}{{\sigma }^2}\left(\mu -\frac{1}{n}\sum _{k=1}^n{u}_k\right)\right)}^{2}du\hfill \\ \mathrm{(50)}& & =& \frac{n^2}{{\sigma }^4}{\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\mu -\frac{1}{n}\sum _{k=1}^n{u}_k\right)}^{2}du\hfill \end{array}$$

Hence:

$$\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mu ,\sigma }\right)}_{\mu }}=\frac{1}{{\sigma }^2}\sqrt{{\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\mu -\frac{1}{n}\sum _{k=1}^n{u}_k\right)}^{2}du}$$

(51)

Also:

$$\begin{array}{ccc}\hfill h_M(f_X,f_{Y|\mu ,\sigma })& =& -{\int }_{-\infty }^{\infty }{f}_X\left(u\right)ln{f}_{Y|\mu ,\sigma }\left(u\right)du\hfill \\ & =& -{\int }_{-\infty }^{\infty }\lambda e^{-\lambda u}H\left(u\right)ln{f}_{Y|\mu ,\sigma }\left(u\right)du\hfill \\ & =& -{\int }_0^{\infty }\lambda e^{-\lambda u}\left(ln\left(\frac{1}{\sigma \sqrt{2\pi }}\right)-\frac{{\left(\mu -u\right)}^2}{2{\sigma }^2}\right)du\hfill \\ & =& ln\left(\sigma \sqrt{2\pi }\right){\int }_0^{\infty }\lambda e^{-\lambda u}du+\frac{\lambda }{2{\sigma }^2}{\int }_0^{\infty }{\left(\mu -u\right)}^2{e}^{-\lambda u}du\hfill \\ \mathrm{(52)}& & =& ln\left(\sigma \sqrt{2\pi }\right)+\frac{1}{{\sigma }^2}\left(\frac{{\mu }^2}{2}-\frac{\mu }{\lambda }+\frac{1}{{\lambda }^2}\right)\hfill \end{array}$$

and:

$$\frac{\partial h_M(f_X,f_{Y|\mu ,\sigma })}{\partial \mu }=\frac{1}{{\sigma }^2}\left(\mu -\frac{1}{\lambda }\right)$$

(53)

Finally, we calculate the micro-structure term using Equation (17):

$$2\epsilon \frac{\partial \delta }{\partial \mu }=\frac{1}{{\sigma }^2}\left(\frac{1}{n}\sum _{k=1}^n{x}_k-\frac{1}{\lambda }\right)$$

(54)

This allows us to calculate:

$$\begin{array}{ccc}\hfill I_R^{\mu }& =& {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|2\epsilon \frac{\partial \delta }{\partial \mu }\right|}^{2}du\hfill \\ & =& {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|\frac{1}{{\sigma }^2}\left(\frac{1}{n}\sum _{k=1}^n{u}_k-\frac{1}{\lambda }\right)\right|}^{2}du\hfill \\ \mathrm{(55)}& & =& \frac{1}{{\sigma }^4}{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{1}{n}\sum _{k=1}^n{u}_k-\frac{1}{\lambda }\right)}^{2}du\hfill \end{array}$$

Now, for σ, we have:

$$\frac{1}{n}\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mu ,\sigma }}{\partial \sigma }=\frac{1}{{\sigma }^3}\left(-{\sigma }^2+\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2\right)$$

(56)

The mixed information for σ is defined by:

$$\begin{array}{ccc}\hfill i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mu ,\sigma }\right)}_{\mu }& =& {\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{\partial ln{f}_{{\left\{Y\right\}}_n|\mu ,\sigma }}{\partial \sigma }\right)}^{2}du\hfill \\ & =& {\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(\frac{n}{{\sigma }^3}\left(-{\sigma }^2+\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2\right)\right)}^{2}du\hfill \\ \mathrm{(57)}& & =& \frac{n^2}{{\sigma }^6}{\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(-{\sigma }^2+\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2\right)}^{2}du\hfill \end{array}$$

Hence:

$$\frac{1}{n}\sqrt{i_F{\left(f_{{\left\{X\right\}}_n},f_{{\left\{Y\right\}}_n|\mu ,\sigma }\right)}_{\mu }}=\frac{1}{{\sigma }^3}\sqrt{{\int }_{{\mathbb{R}}_{+}^n}{f}_{{\left\{X\right\}}_n}{\left(-{\sigma }^2+\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2\right)}^{2}du}$$

(58)

Again, the derivative of the mixed entropy with respect to σ:

$$\frac{\partial h_M(f_X,f_{Y|\mu ,\sigma })}{\partial \sigma }=\frac{1}{{\sigma }^3}\left(\left({\sigma }^2-\frac{1}{{\lambda }^2}\right)-{\left(\mu -\frac{1}{\lambda }\right)}^2\right)$$

(59)

Hence, the associated micro-differences expression is:

$$2\epsilon \frac{\partial \delta }{\partial \sigma }=\frac{1}{{\sigma }^3}\left(\left(\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2-\frac{1}{{\lambda }^2}\right)-{\left(\mu -\frac{1}{\lambda }\right)}^2\right)$$

(60)

which allows us to calculate:

$$\begin{array}{ccc}\hfill I_R^{\sigma }& =& {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|2\epsilon \frac{\partial \delta }{\partial \sigma }\right|}^{2}du\hfill \\ & =& {\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left|\frac{1}{{\sigma }^3}\left(\left(\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2-\frac{1}{{\lambda }^2}\right)-{\left(\mu -\frac{1}{\lambda }\right)}^2\right)\right|}^{2}du\hfill \\ \mathrm{(61)}& & =& \frac{1}{{\sigma }^6}{\int }_{{\mathcal{M}}_{\epsilon }^n}{f}_{{\left\{X\right\}}_n}{\left(\left(\frac{1}{n}\sum _{k=1}^n{\left(\mu -u_k\right)}^2-\frac{1}{{\lambda }^2}\right)-{\left(\mu -\frac{1}{\lambda }\right)}^2\right)}^{2}du\hfill \end{array}$$

9.3. Unknown Source Density Function, $f_X$

If $f_X$ is unknown, then it is not possible to calculate the previous bounds. In this case, we resort to the optimization program specified in Equation (41). Hence, we use Equations (49) and (56), in order to obtain:

$$\begin{array}{ccc}\hfill {\mu }_n& =& \frac{1}{n}\sum _{k=1}^n{x}_k\hfill \end{array}$$

(62)

$$\begin{array}{ccc}\hfill {\sigma }_n^2& =& \frac{1}{n}\sum _{k=1}^n{\left({\mu }_n-x_k\right)}^2\hfill \end{array}$$

(63)

Using this choice of values immediately makes both mixed information expressions described by Equations (51) and (58) equal to zero, without the need for large values of n.

Furthermore, if we use:

$$\begin{array}{ccc}\hfill {\mu }_{\circ }=\underset{n\to \infty }{lim}{\mu }_n& =& \frac{1}{\lambda }\hfill \end{array}$$

(64)

$$\begin{array}{ccc}\hfill {\sigma }_{\circ }=\underset{n\to \infty }{lim}{\sigma }_n& =& \frac{1}{\lambda }\hfill \end{array}$$

(65)

then the micro-differences integrals also become equal to zero.

This example shows several things:

• The use of the maximum log likelihood solutions and large values of n guarantees that the derivative of the mixed entropy equals zero; hence, the derivative of the relative entropy with respect to its parameters is zero, too.
• However, this is not enough to guarantee that the density functions will match each other (a Gaussian cannot estimate an exponential). The maximum log likelihood solution only guarantees that the estimator will do the best it can do.
• If not enough data is available, the derivative of the mixed entropy will be different from zero, because the micro-differences terms are not zero.

10. Discussion

If one wanted to estimate a density function out of a data set, one could minimize the relative entropy: once we reach its minimum, we can guarantee that the density function implemented by the estimator will be equal to that which originated the data. This is true only if the family of functions that the estimator can effectively implement includes that of the source density function. Moreover, realizing that it is not possible to measure the relative entropy, one would quickly settle to maximize the log likelihood and obtain the desired density function. Keeping this in mind, the results presented in this work seem just another way of saying the same. However, this is not so. The difference is subtle. Whereas there are many density function estimation principles and it could be discussed whether minimizing the relative entropy is the most convenient one, the bounds presented in this work show that minimizing the relative entropy automatically activates in the presence of large data sets and that one does not need to decide whether that estimation framework is the most convenient or not. Something similar happens with the weak law of large numbers: there are many ways of determining the expected value of the density function that generated the data, but everybody knows that the weak law of large numbers guarantees that in the presence of large data sets, this value can be effectively approximated by the average of the values. Our bounds, also based on the weak law of large numbers, state something similar: in the presence of large data sets, the source density function is perceived as the solution of the maximum log likelihood solution.

11. Conclusions

Theorem 2, which is the main result of this work, consists in a set of bounds on the partial derivative of the mixed entropy, which is a proxy to the relative entropy, with respect to the parameters of the estimator. These bounds relate the mixed information value, the capacity of the estimator to produce micro-differences and the number of random variables present in the analysis in such a way that it is possible to determine that these bounds activate automatically when the data set is large. If the micro-differences integral is zero, the mixed information derivative is zero for large data sets, independently of any other considerations. In other words, minimizing the mixed entropy is the preferred density estimation framework when large data sets are available.

Given the impossibility of estimating the mixed entropy, it is shown that the correct numerical approximation to this problem is finding the solution to the maximum log likelihood problem. Again, when the estimator is associated to a micro-differences integral equal to zero, this framework becomes the optimal one in the presence of large data sets.