Duality of Maximum Entropy and Minimum Divergence

Abstract

We discuss a special class of generalized divergence measures by the use of generator functions. Any divergence measure in the class is separated into the difference between cross and diagonal entropy. The diagonal entropy measure in the class associates with a model of maximum entropy distributions; the divergence measure leads to statistical estimation via minimization, for arbitrarily giving a statistical model. The dualistic relationship between the maximum entropy model and the minimum divergence estimation is explored in the framework of information geometry. The model of maximum entropy distributions is characterized to be totally geodesic with respect to the linear connection associated with the divergence. A natural extension for the classical theory for the maximum likelihood method under the maximum entropy model in terms of the Boltzmann-Gibbs-Shannon entropy is given. We discuss the duality in detail for Tsallis entropy as a typical example.

1. Introduction

Information divergence plays a central role in the understanding of integrating statistics, information science, statistical physics and machine learning. Let $ℱ$ be the space of all the probability density functions with a common support with respect to a carrier measure Λ of a data space. Usually Λ is taken as the Lebesgue measure and the counting measure corresponding to continuous and discrete random variables, respectively. The most typical example of information divergence is the Kullback-Leibler divergence

$$D_0\left(f,g\right)=\int f\left(x\right)\left\{\text{log}\hspace{0.17em}f\left(x\right)-\text{log}\hspace{0.17em}g\left(x\right)\right\}d\Lambda \left(x\right)$$

on $ℱ$, which is decomposed into the difference of cross and diagonal entropy measures

$$C_0\left(f,g\right)=-\int f\left(x\right)\text{log}\hspace{0.17em}g\left(x\right)d\Lambda \left(x\right)$$

and

$$H_0\left(f,g\right)=-\int f\left(x\right)\text{log}\hspace{0.17em}f\left(x\right)d\Lambda \left(x\right).$$

The entropy H₀(f) is nothing but Boltzmann-Gibbs-Shannon entropy. In effect, D₀(f, g) connects the maximum likelihood [1,2], and the maximum entropy [3]. If we take a canonical statistic t(X), then the maximum entropy distribution under a moment constraint for t(X) belongs to the exponential model associated with t(X),

$$M^{\left(\text{e}\right)}=\left\{f_0\left(x,\theta \right):=\text{exp}\left\{{\theta }^{\top }t\left(x\right)-{\kappa }_0(\theta )\right\}:\theta \in \Theta \right\}$$

(1)

where κ₀(θ) = log ∫ exp{θ^⊤ t (x)}dΛ(x) and Θ = {θ : κ₀(θ) < ∞}. In this context, the statistic t(X) is minimally sufficient in the model, in which the maximum likelihood estimator (MLE) for the parameter θ of the model is given by one-to-one correspondence with t(X), see [4] for the convex geometry. If we consider the expectation parameter,

$$\mu ={𝔼}_{f_0\left(\cdot ,\theta \right)}\left\{t\left(X\right)\right\}$$

in place of θ, then for a given random sample X₁, ···, X_n, the MLE for μ is given by the sample mean of t(X_i)’s, that is

$${\widehat{\mu }}_0=\frac{1}{n}\sum _{i=1}^{n}t\left(X_i\right).$$

We define two kinds of geodesic curves connecting f and g in $ℱ$. We call a curve

$$C^{\left(\text{m}\right)}=\left\{C_t^{\left(\text{m}\right)}\left(x\right):=\left(1-t\right)f\left(x\right)+tg\left(x\right):t\in \left(0,1\right)\right\}$$

(2)

mixture-geodesic. Alternatively, we call a curve

$$C^{\left(\text{e}\right)}=\left\{C_t^{\left(\text{e}\right)}\left(x\right):=\text{exp}\left\{\left(1-t\right)\text{log}f\left(x\right)+t\text{log}g\left(x\right)-\kappa \left(t\right)\right\}:t\in \left(0,1\right)\right\}$$

(3)

exponential geodesic, where κ(t) = log ∫ f(x)^1–tg(x)^tdΛ(x). We denote Γ^(m) and Γ^(e) the two linear connections induced by the mixture and exponential geodesic curves on $ℱ$, which we call the mixture connection and exponential connection on $ℱ$, respectively, see [5,6]. Thus all tangent vectors on a mixture geodesic curve are parallel to each other with respect to Γ^(m); all tangent vectors on an exponential geodesic curve are parallel to each other with respect to Γ^(e). It is well-known that M^(e) is totally exponential-geodesic, that is, for any f₀(x, θ₀) and f₀(x, θ₁) in M^(e) it holds that the exponential geodesic curve connecting f₀(x, θ₀) and f₀(x, θ₁) is in M^(e). In effect we observe that $C_t^{(\text{e})}(x)=f_0(x,{\theta }_t)$ with θ_t = (1–t)θ₀ +tθ₁. Thus $C_t^{(\text{e})}(x)\in M^{(\text{e})}$ for all t ∈ (0, 1) because Θ is a convex set. Alternatively, consider a parametric model

$$M^{\left(\text{m}\right)}=\left\{f_1\left(x,\pi \right):=\sum _{j=0}^d{\pi }_j{f}_j\left(x\right):{\pi }_j>0\left(j=0,\cdots ,d\right),\sum _{j=0}^d{\pi }_j=1\right\}.$$

Then, M^(m) is totally mixture-geodesic. Because a mixture geodesic curve $C_t^{(\text{m})}(x)=(1-t)f_1(x,{\pi }_0)+t{f}_1(x,{\pi }_1)$ is in M^(m) for any t ∈ (0, 1) on account of $C_t^{(\text{m})}(x)=f_1(x,{\pi }_t)$, where (1 – t)π₀ + tπ₁.

We discuss a generalized entropy and divergence measures with applications in statistical models and estimation. There have been recent developments for the generalization of Boltzmann-Shannon entropy and Kullback-Leibler divergence. We focus on U-divergence with a generator function U, in which U-divergence is separated into the differences between cross entropy and diagonal entropy. We observe a dualistic property associated with U-divergence between statistical model and estimation. The U-loss function is given by an empirical approximation for U-divergence based on a given dataset under a statistical model, in which the U-estimator is defined by minimization of the U-loss function on the parameter space. On the other hand, the diagonal entropy leads to a maximum entropy distribution with a mean equal space, where we call the family of distributions U-model. In accordance with this, the U-divergence leads to a pair of U-model and U-estimator as a statistical model and estimation. The typical example is that U(t) = exp(t), which is associated with the Kullback-Leibler divergence D₀(f, g) generating a pair of an exponential family M^(e) and the minus log-likelihood function.

This aspect is characterized as a minimax game between a decision maker and Nature. The paper is organized as follows. Section 2 introduces the class of U-divergence measures. The information geometric framework associated with a divergence measure is given in Section 3. In Section 3 we discuss the maximum entropy model with respect to U-diagonal entropy. The minimum divergence method via U-divergence is discussed in Section 5. We next explore the duality between maximum U-entropy and minimum U-divergence in Section 6. Finally, we discuss the relation to the robust statistics by minimum divergence, and a future problem on MaxEnt in Section 7.

2. U-Divergence

A class of information divergence is constructed by a generator function U via a simple employment of conjugate convexity, see [7]. We introduce a class of generator functions by

$$𝒰=\left\{U:ℝ\to {ℝ}_{+}:\frac{d}{ds}U\left(s\right)\ge 0,\frac{d^2}{d{s}^2}U\left(s\right)\ge 0\right\}.$$

Then we consider the conjugate convex function defined on ℝ₊ of U in $𝒰$ as

$$U^{*}\left(t\right)=\underset{s\in ℝ}{\text{max}}\left\{st-U\left(s\right)\right\},$$

and hence U*(t) = tξ(t) – U(ξ(t)), where ξ(t) is the inverse function of the derivative of U(s), or equivalently (dU/ds)(ξ(t)) = t. The existence for ξ(t) is guaranteed from the assumption for U to be in $𝒰$, in which we observe an important property that the derivative of U* is the inverse of the derivative of U, that is

$$\frac{d}{dt}{U}^{*}\left(t\right)=\xi \left(t\right).$$

(4)

The conjugate function U* of U is reflexible, that is, U** = U. By definition, for any s ∈ ℝ and t ∈ ℝ₊,

$$U^{*}\left(t\right)\ge st-U\left(s\right)$$

(5)

with equality if and only if s = ξ(t). We consider an information divergence functional using the generator function U as

$$D_U\left(f,g\right)=\int \left\{U^{*}\left(f\right)-f\xi \left(g\right)+U\left(\xi \left(g\right)\right)\right\}d\Lambda ,$$

(6)

called U-divergence. We can easily confirm that D_U (f, g) satisfies the first axiom of a distance function since the integrand in Equation (6) is always nonnegative with equality of 0 if and only if f(x)= g(x) because Equation (5). It follows from the construction that D_U (f, g) is decomposed into C_U (f, g) and H_U (f) such that

$$D_U\left(f,g\right)=C_U\left(f,g\right)-H_U\left(f\right).$$

Here

$$C_U\left(f,g\right)=\int \left\{U\left(\xi \left(g\right)\right)-f\xi \left(g\right)\right\}d\Lambda ,$$

is called U-cross entropy;

$$H_U\left(f\right)=-\int U^{*}\left(f\right)d\Lambda$$

(7)

is called U-diagonal entropy. We can write H_U (f) = ∫ {U(ξ(f)) – fξ(f)}dΛ by the definition for U*, which equals the diagonal C_U (f, f). We note that the U-divergence is expressed as

$$D_U\left(f,g\right)=\int \left\{U^{*}\left(f\right)-U^{*}\left(g\right)+\xi \left(g\right)\left(f-g\right)\right\}d\Lambda ,$$

because of Equation (4), which implies that U* plays a role on a generator function in place of U. In fact, this is also called U*-Bregman divergence, cf. [8,9]

The first example of U is U₀(s) = exp(s), which leads to $U_0^{*}(t)=t\text{log}t-t$ and

$$\text{log}\left(t\right)=\underset{s\in ℝ}{\text{argmax}}\left\{st-\text{exp}\left(s\right)\right\},$$

Thus U₀-divergence, U₀-cross entropy and U₀-diagonal entropy equal D₀(f, g), C₀(f, g) and H₀(f) as defined in Introduction, respectively. As for the second example we consider

$$U_{\beta }\left(s\right)=\frac{1}{\beta +1}{\left(1+\beta s\right)}^{\frac{1+\beta }{\beta }}$$

(8)

where β is a scalar. The conjugate function becomes

$$U_{\beta }^{*}\left(t\right)=\frac{1}{\beta \left(\beta +1\right)}{t}^{\beta +1}-\frac{1}{\beta }t.$$

(9)

Then the generator function U_β associates with the β-power cross entropy

$$C_{\beta }\left(f,g\right)=\frac{1}{\beta +1}\int g^{\beta +1}d\Lambda -\frac{1}{\beta }\int f\left(g^{\beta }-1\right)d\Lambda ,$$

β-diagonal power entropy

$$H_{\beta }\left(f\right)=-\frac{1}{\beta \left(\beta +1\right)}\int f^{\beta +1}d\Lambda +\frac{1}{\beta }$$

and the β-power divergence D_β(f, g) = C_β(f, g) – H_β(f), that is,

$$D_{\beta }\left(f,g\right)=\frac{1}{\beta \left(\beta +1\right)}\int f^{\left(\beta +1\right)}d\Lambda -\frac{1}{\beta }\int f{g}^{\beta }d\Lambda +\frac{1}{\beta +1}\int g^{\beta +1}d\Lambda .$$

We observe that

$$\underset{\beta \to 0}{\text{lim}}\left(C_{\beta }\left(f,g\right),H_{\beta }\left(f\right)\right)=\left(C_0\left(f,g\right),H_0\left(f\right)\right).$$

The class of β-power divergence functionals includes the Kullback-Leibler divergence in the limiting sense of lim_β→0 D_β(f, g) = D₀(f, g). If β = 1, then $D_{\beta }(f,g)=\frac{1}{2}\int {(f-g)}^{2}d\Lambda$, which is a half of the squared L₂ norm. If we take a limit of β to −1, then D_β(f, g) becomes the Itakura-Saito divergence

$$D_{\text{IS}}\left(f,g\right)=\int \left(\text{log}g-\text{log}f+\frac{f}{g}-1\right)d\Lambda ,$$

which is widely applied in signal processing and speech recognition, cf. [10–12].

The β-power divergence D_β(p, q) is proposed in [13]; the β-power entropy H_β is equal to the Tsallis q-entropy with a relation q = β + 1, cf. [14–16]. Tsallis entropy is connected with spin glass relaxation, dissipative optical lattices and so on beyond the classical statistical physics associated with the Boltzmann-Shannon entropy H₀(p). See also [17,18] for the power entropy in the field of ecology. We will discuss the statistical property for the minimum β divergence method in the presence of outliers departing from a supposed model, cf. [19–21]. A robustness performance is elucidated by appropriate selection for β. Beyond robustness perspective, a property of spontaneous learning to apply to clustering analysis is focused in [22], see also [23] for nonnegative matrix analysis.

The third example of a generator function is U_η(s) = (1 – η)exp(s) – ηs with a scalar η. This generator function leads to the η-cross entropy

$$C_{\eta }\left(f,g\right)=-\int \left\{f\left(x\right)+\eta \right\}\text{log}\left\{g\left(x\right)+\eta \right\}d\Lambda \left(x\right)$$

and the η-entropy

$$H_{\beta }\left(f\right)=\int \left\{f\left(x\right)+\eta \right\}\text{log}\left\{f\left(x\right)+\eta \right\}d\Lambda \left(x\right),$$

so that the η-divegence is D_η(f, g) = C_η(f, g) – H_η(f), see [24–27] for applications for pattern recognition. Obviously, if we take a limit of η to 0, then C_η(f, g), H_η(f) and D_η(f, g) converge to C₀(f, g), H₀(f) and D₀(f, g), respectively. A mislabeled model is derived by a maximum η-entropy distribution with momentary constraint if we consider a binary regression model. See [25,27] for a detailed discussion.

3. Geometry Associated with U-Divergence

We investigate geometric properties associated with U-divergence, which will help the discussion in subsequent sections. Let us arbitrarily fix a statistical model M = {f_θ(x) : θ ∈ Θ} embedded in the total space $ℱ$ with mild regularity conditions. In fact, we consider the mixture geodesic curve C^(m), the exponential geodesic curve C^(e), the mixture model M^(m) and the exponential model M^(e) as typical examples of M. Here are difficult aspects to define $ℱ$ as a differentiable manifold of infinite dimension because the constraint for positivity on the support is intractable in the sense of the topology, see Section 2 in [6] for detailed discussion and historical remarks. On the other hand, if we confine ourselves to a statistical model M, then we can formulate M as a finite dimensional manifold, as in the following discussion. Thus, we produce a path geometry in which for any two elements f and g of $ℱ$ a class of geodesic curves connecting f and g including C^(m) and C^(e) is introduced so that the class of geodesic subspaces is derived as for M^(m) and M^(e).

3.1. Riemannian Metric and Linear Connections

We view the statistical model M as a d-dimensional differentiable manifold with the coordinate θ = (θ¹, ···, θ^d). Any information divergence associates with a Riemaniann metric and dual linear connections, see [28,29] for detailed discussion. We focus on the geometry generated by the U-divergence D_U (f, g) as follows. The Riemannian metric at f_θ of M is given by

$$G_{ij}^{\left(U\right)}(\theta )=-\int {\partial }_i{f}_{\theta }{\partial }_j\xi \left(f_{\theta }\right)d\Lambda ,$$

(10)

and linear connections are

$${\Gamma }_{i,j,k}^{\left(U\right)}(\theta )=-\int {\partial }_i{\partial }_j{f}_{\theta }{\partial }_k\xi \left(f_{\theta }\right)d\Lambda$$

(11)

and

$${{}^{*}\Gamma }_{i,j,k}^{(U)}(\theta )=-\int {\partial }_k{f}_{\theta }{\partial }_i{\partial }_j\xi \left(f_{\theta }\right)d\Lambda ,$$

(12)

where ∂_i = ∂/∂θⁱ, see Appendix. for the derivation. Now we can assert the following theorem under an assumption for $ℱ$: Let f be arbitrarily fixed in $ℱ$. If ∫ a(x){g(x) – f(x)}dΛ(x) = 0 for any g of $ℱ$, then a(x) is constant in x almost everywhere with respect to Λ.

Theorem 1. Let Γ^(U) be the linear connection defined in Equation (11). Then any Γ^(U)-geodesic curve is equal to the mixture-geodesic curve defined in Equation (2).

Proof. Let C^(U) := {f_t(x): t ∈ (0, 1)} be a Γ^(U)-geodesic curve with f₀ = f and f₁ = g. We consider a 2-dimensional model defined by f_θ(x) = (1 – s + u)f_t(x)+(s – u)g(x), where θ =(s, t, u). Then we observe that if u = s, then

$${\Gamma }_{11,2}^{\left(U\right)}(\theta )=-\int \left(\frac{d^2}{d{t}^2}{f}_t\right){\xi }^{\prime }\left(f_t\right)\left(g-f_t\right)d\Lambda$$

(13)

which identically 0 for any g of $ℱ$. It follows from the assumption for $ℱ$ that (d²/dt²)f_t(x)= c almost everywhere with respect to Λ, which solved by

$$f_t\left(x\right)=\frac{1}{2}ct\left(t-1\right)+\left(1-t\right)f\left(x\right)+tg\left(x\right)$$

from the endpoint condition for C^(U). We observe that c = 0 because f_t(x) ∈$ℱ$, which concludes that C^(U) equals the mixture-geodesic. The proof is complete.

This property is elemental to characterize the U-divergence class, which is closely related with the empirical reducibility as discussed in a subsequent section. The assumption for $ℱ$ holds if the carrier measure Λ is Lebesgue measure or the counting measure.

On the other hand, for a *Γ^(U)-geodesic curve ${{}^{*}C}^{(U)}:=\left\{f_t^{*}(x):t\in (0,1)\right\}$ with f₀ = f and f₁ = g we consider an embedding into a 2-dimensional model,

$$f_{\theta }^{*}\left(x\right)=u\left(\left(1-s+t\right)\xi \left(f_t^{*}\left(x\right)\right)+\left(s-t\right)\xi \left(g\left(x\right)\right)+{\kappa }_{\theta }\right),$$

where θ = (s, t), where u(s) = (d/dt)U(s) and κ_θ is a normalizing constant to satisfy $\int f_{\theta }^{*}(x)d\Lambda (x)=1$. By definition

$${{}^{*}\Gamma }_{11,2}^{(U)}(\theta )=-\int \left(\frac{d^2}{d{t}^2}\xi \left(f_t^{*}\right)\right)u^{\prime }\left({\xi }^{\prime }\left(f_t^{*}\right)\right)\left\{\xi \left(g\right)-\xi \left(f_t^{*}\right)\right\}d\Lambda =0$$

(14)

if s = t. This leads to $\left(d^2/d{t}^2\right)\xi \left(f_t^{*}(x)\right)=c$ almost everywhere with respect to Λ, which is solved by

$$f_t^{*}\left(x\right)=u\left(\left(1-t\right)\xi \left(f\left(x\right)\right)-t\xi \left(g\left(x\right)\right)-{\kappa }_t\right),$$

(15)

We confirm that, if U = exp, then *Γ^(U)-geodesic curve reduces to the exponential geodesic curve defined in Equation (3). □

3.2. Generalized Pythagorian Theorems

We next consider the Pythagorean theorem based on the U-divergence as an extension of the result associated with the Kullback-Leibler divergence in [6].

Theorem 2. Let p, q and r be in $ℱ$. We connect p with q by the mixture geodesic

$$f_t^{\left(\text{m}\right)}\left(x\right)=\left(1-t\right)p\left(x\right)+tq\left(x\right),$$

Alternatively we connect r and q by *Γ^(U)-geodesic curve

$$f_s^{\left(U\right)}\left(x\right)=u\left(\left(1-s\right)\xi \left(r\left(x\right)\right)-s\xi \left(q\left(x\right)\right)-\kappa \left(s\right)\right).$$

Two curves { $f_t^{(\text{m})}(x):t\in [0,1]$} and { $f_s^{(U)}(x):s\in [0,1]$} orthogonally intersect at q with respect to the Riemannian metric G^(U) defined in Equation (10) if and only if

$$D_U\left(p,r\right)=D_U\left(p,q\right)+D_U\left(q,r\right).$$

(16)

Proof. A straightforward calculus yields that

$${-\frac{{\partial }^2}{\partial t\partial s}{D}_U\left(f_t^{(\text{m})},f_s^{(U)}\right)|}_{t=1,s=1}=D_U(p,r)-\left\{D_U(p,q)+D_U(q,r)\right\}.$$

(17)

By the definition of G^(U) we see that $G_{12}^{(U)}(\theta )$ is nothing but the left side of Equation (17) when

$$f_{\theta }\left(x\right)=\left(1-t\right)p\left(x\right)+t{f}_s^{\left(U\right)}\left(x\right),$$

where θ = (t, s). Hence the orthogonality assumption is equivalent to Equation (16), which completes the proof.

Remark 1. We remark a further property such that, for any s and t in [0, 1],

$$D_U\left(p_t,r\right)=D_U\left(p_t,q\right)+D_U\left(q,r_s\right).$$

If U = exp, then Theorem 2 reduces to the Pythagoras theorem with the Kullback-Leibler divergence as shown in [6]. Consider two geodesic subspaces defined by

$$M^{\left(\text{m}\right)}=\left\{p_{\pi }\left(x\right)={\pi }_{0}q\left(x\right)+\sum _{j=1}^J{\pi }_j{p}_j\left(x\right):{\pi }_j\ge 0\left(j=0,\cdots ,J\right),\sum _{j=0}^J{\pi }_j=1\right\}$$

and

$$M^{\left(U\right)}=\left\{r_{∊}\left(x\right)=u\left({∊}_0\xi \left(q\left(x\right)\right)+\sum _{k=1}^K{∊}_k\xi \left(r_k\left(x\right)\right)-\kappa (∊)\right):{∊}_k\ge 0\left(k=0\cdots ,K\right),\sum _{k=0}^K{∊}_k=1\right\}.$$

(18)

For any m-geodesic curve C^(m) and U-geodesic curve *C^(U) connecting q we assume that C^(m) and C^(U) orthogonally intersect at q in the sense of the Riemannian metric G^(U). Then, for any p ∈ M^(m) and r ∈ M^(U)

$$D_U\left(p,r\right)=D_U\left(p,q\right)+D_U\left(q,r\right),$$

in which two-way projection is associated with as

$$D_U\left(p,q\right)=\underset{r\in M_2}{\text{min}}{D}_U\left(p,r\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{D}_U\left(q,r\right)=\underset{p\in M_1}{\text{min}}{D}_U\left(p,r\right).$$

First we confirm a kind of reduction property for the Kullback-Leibler divergence to the framework in information geometry such that (G^(D₀), Γ^(D₀), *Γ^(D₀)) = (G, Γ^(m), Γ^(e)), where G is the information metric. Second we return a case of the β-power divergence, which is reduced a special case of Theorem 2. Consider two curves $C^{(\text{m})}=\left\{C_t^{(\text{m})}(x)=(1-t)p(x)+tq(x):t\in [0,1]\right\}$ and

$$C^{\left(\beta \right)}=\left\{C_s^{\left(\beta \right)}\left(x\right)={\left\{\left(1-s\right)r{\left(x\right)}^{\beta }+tq{\left(x\right)}^{\beta }+c\left(s\right)\right\}}^{\frac{1}{\beta }}:s\in \left[0,s\right]\right\}.$$

Then we observe for the Riemannian metric G^(β) generated by β-power divergence that

$$G^{\left(\beta \right)}\left({\dot{C}}_1^{\text{m}},{\dot{C}}_1^{\beta }\right)\left(q\right)=D_{\beta }\left(p,r\right)-\left\{D_{\beta }\left(p,q\right)+D_{\beta }\left(q,r\right)\right\},$$

(19)

which is ∫ (p – q)(p^β – q^β)dΛ. We observe that if C^(m) and C^(β) orthogonally intersect at q, then

$$D_{\beta }\left(p,r\right)=D_{\beta }\left(p,q\right)+D_{\beta }\left(q,r\right).$$

4. Maximum Entropy Distribution

The maximum entropy principle is based on the Boltzmann-Shannon entropy in which the maximum entropy distribution is characterized by an exponential model. The maximum entropy method has been widely enhanced in fields of natural language processing, ecological analysis and so forth. However, there are other types of entropy measures proposed as the Hill diversity index, the Gini-Simpson index, the Tsallis entropy and so on, cf. [14,17,18] in different fields. We introduced the class of U-entropy functionals, which include all the entropy measures mentioned above. In this subsection, we discuss the maximum entropy distribution based on an arbitrarily fixed U-entropy.

We check a finite discrete case with K + 1 cells as a special situation, where $ℱ$ reduces to a K-dimensional simplex $𝒮$_K. The maximum U-entropy distribution is defined by

$$f^{*}=\underset{f\in {𝒮}_k}{\text{argmax}}{H}_U\left(f\right).$$

The Lagrange function is

$$L\left(f,\lambda \right)=\sum _{i=1}^{K+1}\left\{-\xi \left(f_i\right)f_i+U\left(\xi \left(f_i\right)\right)\right\}+\lambda \left(\sum _{i=1}^{K+1}{f}_i-1\right).$$

We observe that

$$\frac{\partial }{\partial f_i}L\left(f,\lambda \right)=-\xi \left(f_i\right)+\lambda =0,$$

which implies $f_i^{*}=1/(K+1)$ for i = 1, ···, K + 1. Therefore the maximum U-entropy distribution f^* is a uniform distribution on $𝒮$_K for any generator function U.

In general the U-entropy is an unbounded functional on $ℱ$ unless $ℱ$ is finite discrete. For this we introduce a moment constraint as follows. Let t(X) be a k-dimensional statistic vector. Henceforth we assume that 𝔼_f {‖ t(X) ‖ ²} < ∞ for all f of $ℱ$. We consider a mean equal space for t(X) as

$$\Gamma \left(\tau \right)=\left\{f\in ℱ:{𝔼}_f\left\{t\left(X\right)\right\}=\tau \right\},$$

where τ is a fixed vector in ℝ^k. By definition Γ(τ) is totally mixture geodesic, that is, if f and g are in Γ(τ), then (1 – t)f + tg is also in Γ(τ) for any t ∈ (0, 1).

Theorem 3. Let$f_{\tau }^{*}=\text{argmax}\left\{H_U(f):f\in \Gamma (\tau )\right\}$, where H_U (f) is U-diagonal entropy defined in Equation (7). Then the maximum U-entropy distribution is given by

$$f_{\tau }^{*}\left(x\right)=u\left({\theta }^{\top }t\left(x\right)-{\kappa }_U\left(\theta \right)\right),$$

(20)

where κ_U (θ) is the normalizing factor and θ is a parameter vector determined by the moment constraint

$$\int t\left(x\right)u\left({\theta }^{\top }t\left(x\right)-{\kappa }_U(\theta )\right)d\Lambda \left(x\right)=\tau .$$

Proof. The Eular-Lagrange functional is given by

$$\Phi \left(f,\theta ,\lambda \right)-H_U\left(f\right)-{\theta }^{\top }\left[{𝔼}_f\left\{t\left(X\right)\right\}-\tau \right]-\lambda \left\{\int f\left(x\right)d\Lambda \left(x\right)-1\right\}$$

If g_τ ∈ Γ(τ) and $f_{\tau }^{*}\left(x\right)+t{g}_{\tau }\left(x\right)$, then f_t ∈ Γ(τ), and

$${\frac{d}{dt}\Phi \left(f_t,\theta ,\lambda \right)|}_{t=0}=0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\frac{d^2}{d{t}^2}\Phi \left(f_t,\theta ,\lambda \right)|}_{t=0}<0.$$

(21)

The equation in Equation (21) yields that

$$\int \left\{\xi \left(f_{\tau }^{*}\left(x\right)\right)-{\theta }^{\top }\left(t\left(x\right)-\tau \right)-\lambda \right\}\left\{g\left(x\right)-f^{*}\left(x\right)\right\}d\Lambda \left(x\right)=0$$

for any g_τ (x) in Γ(τ), which concludes Equation (20). Since ξ(t) is an increasing function, we observe that

$$\frac{d^2}{d{t}^2}\Phi \left(f_t,\theta ,\lambda \right)=-\int {\xi }^{\prime }\left(f_t\left(g\right)\right){\left\{g\left(x\right)-f_{\tau }^{*}\left(x\right)\right\}}^{2}d\Lambda \left(x\right)<0$$

(22)

for any t ∈ [0, 1], which implies the inequality in Equation (21). Since g_τ ∈ Γ(τ), we observe that

$${𝔼}_{g_{\tau }}\left\{\xi \left(f_{\tau }^{*}\left(X\right)\right)\right\}={𝔼}_{f_{\tau }^{*}}\left\{\xi \left(f_{\tau }^{*}\left(X\right)\right)\right\}$$

Therefore we can confirm that $H_U\left(f_{\tau }^{*}\right)\ge H_U(g_{\tau })$ for any g_τ ∈ Γ(τ) since

$$H_U\left(f_{\tau }^{*}\right)-H_U\left(g_{\tau }\right)=D_U\left(g_{\tau },f_{\tau }^{*}\right),$$

which is nonnegative by the definition of U-divergence. The proof is complete.

Here we give a definition of the model of maximum U-entropy distributions as follows.

Definition 1. We define a k-dimensional model

$$M_U=\left\{f_U\left(x,\theta \right):=u\left({\theta }^{\top }t\left(x\right)-{\kappa }_U(\theta )\right):\theta \in \Theta \right\},$$

(23)

which is called U-model, where Θ = {θ ∈ ℝ^k : κ_U (θ) < ∞}.

The Naudts’ deformed exponential family discussed from a statistical physical viewpoint as in [15]is closely related with U-model. The one-parameter family {r_s(x): s ∈ [0, 1]} as defined in Equation (15) is a one-dimensional U-model and M^(U) defined in Equation (18) is a K-dimensional U-model. For a U-model M_U defined in Equation (23), the parameter θ is an affine parameter for the linear connection *Γ^(U) defined in Equation (12). In fact, we observe from the definition Equation (12) that

$${{}^{*}\Gamma }_{i,j,k}^{(U)}(\theta )={\partial }_j{\partial }_k{\kappa }_U(\theta )\int {\partial }_k{f}_U(\theta ,x)d\Lambda \left(x\right)$$

which is identically 0 for all θ ∈ Θ. We have a geometric understanding for the U-model similar to the exponential model discussed in Introduction.

Theorem 4. Assume for U(t) that U^′″ (t) > 0 for any t in ℝ. Then, the U-model is totally *Γ^(U)-geodesic.

Proof. For arbitrarily fixed θ₁ and θ₂ in Θ, we define the U-geodesic curve connecting between f_U (x, θ₁) and f_U (x, θ₂) such that, for λ ∈ (0, 1),

$$f_{\lambda }\left(x\right)=u\left(\lambda \xi \left(f_U\left(x,{\theta }_1\right)\right)+\left(1-\lambda \right)\xi \left(f_U\left(x,{\theta }_2\right)\right)-\kappa \left(\lambda \right)\right)$$

with a normalizing factor κ(λ), which is written by f_λ(x) = f_U (x, θ_λ), where θ_λ = λθ₁ +(1 – λ)θ₂. Hence it suffices to show θ_λ ∈ Θ for all λ ∈ (0, 1), where Θ is defined in Definition 1. We look at the identity ∫ f_U (x, θ)dΛ(x) = 1 from a fact that f_U (x, θ) is a probability density function. This implies that the first derivative gives

$$\int u^{\prime }\left({\theta }^{\top }t\left(x\right)-{\kappa }_U(\theta )\right)\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U(\theta )\right\}d\Lambda \left(x\right)=0$$

and the second derivative gives

$$\begin{array}{l}\int u^{″}\left({\theta }^{\top }t\left(x\right)-{\kappa }_U(\theta )\right)\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U(\theta )\right\}{\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U(\theta )\right\}}^{\top }d\Lambda \left(x\right)\\ -\int u^{\prime }\left({\theta }^{\top }t\left(x\right)-{\kappa }_U(\theta )\right)d\Lambda \left(x\right)\frac{{\partial }^2}{\partial \theta \partial {\theta }^{\top }}{\kappa }_U(\theta )=0\end{array}$$

(24)

Since the identity Equation (24) shows that the Hessian of κ_U (θ) is proportional to a Gramian matrix, which implies that κ_U (θ) is convex in θ. Since κ_U (θ_λ) ≤ (1 – λ)κ_U (θ₁)+ λκ_U (θ₂) and θ₁ and θ₂ in Θ, κ_U (θ_λ) ≤ ∞. This concludes that θ_λ ∈ Θ for any λ ∈ (0, 1), which completes the proof.

We discuss a typical example by the power entropy H_β(f), see [15,30–34] from a viewpoint of statistical physics. First we consider a mean equal space of univariate distributions on (0, ∞)

$$\Gamma \left(\mu \right)=\left\{f:{𝔼}_f\left\{t\left(X\right)\right\}=\mu \right\}$$

where

$$t\left(x\right)={\left(x,\frac{x^{\beta \left(\kappa -1\right)}-1}{\beta }\right)}^{\top }$$

Note that lim_β→0 t(x) = (x, (κ – 1) log x). To get the maximum entropy distribution with H_β we consider the Euler-Lagrange function given by

$$E_{\beta }\left(f,\lambda \right)=\frac{1}{\beta \left(\beta +1\right)}{\int }_0^{\infty }{\left(x\right)}^{1+\beta }dx+{\theta }^{\top }\left\{{\int }_0^{\infty }t\left(x\right)f\left(x\right)dx-\mu \right\}+\lambda \left\{{\int }_0^{\infty }f\left(x\right)dx-1\right\},$$

where θ and λ are Lagrange multiplier parameters. This yields that the maximum entropy distribution is

$$\begin{array}{ll}{f}_{\beta }(x,\theta )\hfill & =Z_{\beta }{(\theta )}^{-1}{\left(1+\beta {\theta }^{\top }t\left(x\right)\right)}^{\frac{1}{\beta }}\hfill \\ \hfill & =Z_{\beta }{(\theta )}^{-1}{\left(\beta {\theta }_{1}x+{\theta }_2{x}^{\beta \left(\kappa -1\right)}\right)}^{\frac{1}{\beta }}\hfill \\ \hfill & =Z_{\beta }{(\theta )}^{-1}{x}^{\kappa -1}{\left({\theta }_2-\beta {\theta }_1{x}^{1-\beta \left(\kappa -1\right)}\right)}^{\frac{1}{\beta }},\hfill \end{array}$$

where θ is determined by μ such that 𝔼_fβ(·,θ)t(X) = μ and

$$Z_{\beta }(\theta )={\int }_0^{\infty }x{\left({\theta }_2-\beta {\theta }_1{x}^{1-\beta }\right)}^{\frac{1}{\beta }}dx.$$

A gamma distribution is defined by the density function

$$f\left(x,\kappa ,\theta \right)=\frac{x^{\kappa -1}\text{exp}\left(-\frac{x}{\theta }\right)}{\Gamma \left(\kappa \right){\theta }^{\kappa }}$$

Second, we consider a case of multivariate distributions, where the moment constraints are supposed that for a fixed p-dimensional vector μ and matrix V of size p × p

$$\Gamma (\mu ,V)=\left\{f\in ℱ:{𝔼}_f(X)=\mu ,{𝕍}_f(X)=V\right\}.$$

Let

$$f_{\beta }(\cdot ,\mu ,V)=\underset{f\in \Gamma (\mu ,V)}{\text{argmax}}{H}_{\beta }(f).$$

If we consider a limit case of β to 0, then H_β(f) reduces to the Boltzmann-Shannon entropy and the maximum entropy distribution is the Gaussian distribution with the density function

$$\phi (x,\mu ,V)={\left\{\text{det}(2\pi V)\right\}}^{p/2}\text{exp}\left\{-\frac{1}{2}{(x-\mu )}^{\top }{V}^{-1}(x-\mu )\right\}.$$

In general we deduce that if $\beta >-\frac{2}{p+2}$, then the maximum β-power entropy distribution uniquely exists such that the density function is given by

$$f_{\beta }(x,\mu ,V)=\frac{c_{\beta }}{\text{det}{(2\pi V)}^{\frac{1}{2}}}{\left\{1-\frac{\beta }{2+p\beta +2\beta }{(x-\mu )}^{\top }{V}^{-1}(x-\mu )\right\}}_{+},$$

where

$$c_{\beta }=\{\begin{array}{lll}{\left(\frac{2\beta }{2+p\beta +2\beta }\right)}^{\frac{p}{2}}\Gamma \left(1+\frac{p}{2}+\frac{1}{\beta }\right){\left\{\Gamma \left(1+\frac{1}{\beta }\right)\right\}}^{-1}\hfill & \text{if}\hfill & \beta \ge 0\hfill \\ {\left(\frac{-2\beta }{2+p\beta +2\beta }\right)}^{\frac{p}{2}}\Gamma \left(-\frac{1}{\beta }\right){\left\{\Gamma \left(-\frac{1}{\beta }-\frac{p}{2}\right)\right\}}^{-1}\hfill & \text{if}\hfill & -\frac{2}{p+2}<\beta \le 0\hfill \end{array}$$

See [35,36] for the detailed discussion [37,38] for the discussion on group invariance. Thus, if β> 0, then the maximum β-power entropy distribution has a compact support

$$\left\{x\in {ℝ}^p:{(x-\mu )}^{\top }{V}^{-1}(x-\mu )\le \frac{2}{\beta }+p+2\right\}$$

The typical case is β =2, which is called the Wigner semicircle distribution. On the other hand, if $-\frac{2}{p+2}<\beta <0$, the maximum β-power entropy distribution has a full support of ℝ^p, and equals a p-variate t-distribution with a degree of freedom depending on β.

5. Minimum Divergence Method

We have shown a variety of U-divergence functionals using various generator functions in which the minimum divergence methods are applied to analyses in statistics and statistical machine learning. In effect the U-cross entropy C_U (f, g) is convex-linear in f, that is,

$$C_U\left(\sum _{j=1}^J{\lambda }_j{f}_j,g\right)=\sum _{j=1}^J{\lambda }_j{C}_U(f_j,g)$$

for any λ_j > 0 with ${\sum }_{j=1}^J{\lambda }_j=1$. It is closely related with a characteristic property that the linear connection Γ^(U) associated with U-divergence is equal to the mixture connection Γ^(m) as discussed in Theorem 1. Furthermore, for a fixed g, C_U (f, g) can be viewed as a functional of F in place of f as follows:

$$C_U(F,g)=\int \left\{\xi (g(x))-\int U(\xi (g(x)))d\Lambda (x)\right\}dF(x),$$

where F is the probability distribution generated from f(x). If we assume to have a random sequence X₁, ···, X_n from a density function f(x), then the U-cross entropy is approximated as

$$C_U({\overline{F}}_n,g)=-\frac{1}{n}\sum _{i=1}^n\xi (g(X_i))+\int U(\xi (g))d\Lambda ,$$

(25)

where F̄_n is the empirical distribution based on the data X₁, ···, X_n, that is, ${\overline{F}}_n(B)=\frac{1}{n}{\sum }_{i=1}^{n}I(X_i\in B)$ for any Borel measurable set B. By definition,

$$\int \xi \left(g(x)\right)d{\overline{F}}_n(x)=\frac{1}{n}\sum _{i=1}^n\xi (g(X_i)).$$

Consequently, if we model g by a model function f(·,θ), then the right side of Equation (25) depends only on the data set ${\left(X_i\right)}_{i=1}^n$ and parameter θ without any knowledge for the underlying density function f(x). This gives the empirical approximation, which is advantageous over other classes of divergence measures. The minimum U-divergence method is directly applied to minimization of the empirical approximation with respect to θ. We note that the minimum divergence is equivalent to the minimum cross entropy, in which the diagonal entropy is just a constant in θ. In particular, in the classical case,

$$C_0\left({\overline{F}}_n,f(\cdot ,\theta )\right)=-\frac{1}{n}\sum _{i=1}^n\text{log}f(X_i,\theta )+1,$$

which is equivalent to the minus log-likelihood function.

Let X₁, ···, X_n be independently and identically distributed from an underlying density function f(x) which is approximated by a statistical model M = {f(x, θ): θ ∈ Θ}. The U-loss function is introduced by

$$L_U(\theta )=-\frac{1}{n}\sum _{i=1}^n\xi \left(f(X_i,\theta )\right)+b_U(\theta ),$$

where b_U (θ) = ∫ U(ξ(f(x, θ)))dΛ(x). We call θ̂_U = argmin_θ∈Θ L_U (θ) U-estimator for the parameter θ. By definition 𝔼_f {L_U (θ)} = C_U (F, f(·,θ)) for all θ in Θ, which implies that L_U (θ) almost surely converges to C_U (F, f(·,θ)) as n goes to ∞. Let us define a statistical functional as

$${\theta }_U(F)=\underset{\theta \in \Theta }{\text{argmax}}{C}_U(F,f(\cdot ,\theta )),$$

where C_U (F, g) is written C_U (f, g) placing f into the probability distribution F generated from f. Then θ_U (F) is model-consistent, or θ_U (F_θ) = θ for any θ ∈ Θ because

$$C_U\left(F_{\theta },f(\cdot ,{\theta }^{\prime })\right)\le H_U(f(\cdot ,\theta ))$$

with equality if and if θ ^′ = θ, where F_θ is the probability distribution induced form f(x, θ).

Hence U-estimator θ̂_U is asymptotically consistent. The estimating function is given by

$$s_U\left(x,\theta \right)=\frac{\partial }{\partial \theta }\xi \left(f\left(x,\theta \right)\right)-{𝔼}_{f\left(\cdot ,f\right)}\left\{\frac{\partial }{\partial \theta }\xi \left(f\left(X,\theta \right)\right)\right\}.$$

(26)

Consequently we confirm that s_U (x, θ) is unbiased in the sense that 𝔼_f(_·,θ){s_U (X, θ)} = 0.

We next investigate the asymptotic normality for U-estimator. The estimating equation for the U-estimator is given by

$$\frac{1}{n}\sum _{i=1}^n{s}_U\left(X_i,{\widehat{\theta }}_U\right)=0,$$

of which the Taylor approximation gives

$$\frac{1}{n}\sum _{i=1}^n\left\{s_U\left(X_i,{\theta }_U\left(F\right)\right)\right\}+\frac{\partial s_U}{\partial {\theta }^{\top }}\left(X_i,{\theta }_U\right)\left({\widehat{\theta }}_U-{\theta }_U\left(F\right)\right)=o\left(n_P^{-1}\right).$$

In accordance with this, we get the asymptotic approximation,

$$\sqrt{n}\left\{{\widehat{\theta }}_U-{\theta }_U\left(F\right)\right\}=\frac{1}{\sqrt{n}}J{\left({\theta }_U\left(F\right)\right)}^{-1}\sum _{i=1}^n{s}_U\left(X_i,{\theta }_U\left(F\right)\right)+o\left(n_P^{-\frac{1}{2}}\right),$$

where

$$J\left(\theta \right)={𝔼}_{f\left(\cdot ,\theta \right)}\left\{\frac{\partial s_U}{\partial {\theta }^{\top }}\left(X,\theta \right)\right\}.$$

Because the strong law of large number gives

$$\frac{1}{n}\sum _{i=1}^n\frac{\partial s_U}{\partial {\theta }^{\top }}\left(X_i,{\theta }_U\left(f\right)\right)\stackrel{\text{a.s.}}{\to }J\left({\theta }_U\left(F\right)\right)$$

as n goes to ∞, where $\stackrel{\text{a.s.}}{\to }$ denotes almost sure convergence. If the underlying density function is in the model M, that is f(x) = f(x, θ), then it follows from the model consistency for θ_U (F) that

$$\sqrt{n}\left({\widehat{\theta }}_U-\theta \right)=\frac{1}{\sqrt{n}}J{\left(\theta \right)}^{-1}\sum _{i=1}^n{s}_U\left(X_i,\theta \right)+o\left(n_P^{-\frac{1}{2}}\right),$$

which implies that

$$\sqrt{n}\left({\widehat{\theta }}_U-\theta \right)\stackrel{\text{D}}{\to }N\left(0,J{\left(\theta \right)}^{-1}V\left(\theta \right)J{\left(\theta \right)}^{-1}\right),$$

where $\stackrel{\text{D}}{\to }$ denotes convergence in distribution and

$$V\left(\theta \right)={𝕍}_{f\left(\cdot ,\theta \right)}\left\{s_U\left(X,\theta \right)\right\}.$$

If the generator function is taken as U(s) = exp(s), then the U-estimator reduces to the MLE with the asymptotic normality to N(0,G(θ)^–1), where G(θ) is the Fisher information matrix for θ.

Consider U-estimator for the parameter θ of the exponential model M^(e) in Equation (1). In particular we are concerned with a possible outlying contaminated in the exponential model, and hence a ∊-contamination model is defined as

$$F_{\theta ,∊,y}\left(x\right)=\left(1-∊\right)F_0\left(x,\theta \right)+∊{\delta }_y\left(x\right),$$

where ∊, 0 < ∊ < 1 is a sufficiently small constant, F₀(x, θ) is the cumulative distribution function of the exponential model, and δ_y(x) denotes a degenerate distribution at y. The influence function for U-estimator is given by

$$\text{IF}\left({\widehat{\theta }}_U,y\right):=\underset{∊\to 0}{\text{lim}}\frac{{\theta }_U\left(F_{\theta ,∊,y}\right)-\theta }{∊}=J{\left(\theta \right)}^{-1}{s}_U\left(y,\theta \right),$$

See [19,20,27]. Thus we can check the robustness for U-estimator whether the influence function is bounded in y or not. For example, if we adopt as U(s) = (1+ βs)^1/β, then

$$\text{IF}\left({\widehat{\theta }}_U,y\right)=J{\left(\theta \right)}^{-1}\left[\left\{t\left(y\right)-\mu \right\}{f}_0{\left(y,\theta \right)}^{\beta }-b\left(\theta ,\beta \right)\right],$$

(27)

where b(θ, β)= ∫ {t(x) – μ}f₀(x, θ)^βdΛ(x). Thus, if β> 0, then the influence function is confirmed to be bounded in y for almost cases including a normal, exponential and Poisson distribution models since the term {t(y) – μ}f₀(y, θ)^β in Equation (27) is bounded in y for these models. On the other hand, If β = 0, that is the maximum likelihood estimator entails the unbounded influence functions because the term t(y) – μ is unbounded in y for theses models.

6. Duality of Maximum Entropy and Minimum Divergence

In this section, we discuss a dualistic interplay between statistical model and estimation. In statistical literature, the maximum likelihood estimation has a special position over other estimation methods in the sense of efficiency, invariance and sufficiency; while the statistical model has been explored various candidates in the presence of misspecification. For example, we frequently consider a Laplace distribution for estimating a Gaussian mean, which leads to the sample median as the maximum likelihood estimator for the mean of the Laplace distribution. In this sense, there is an unbalance in the employment for the model and estimator. In principle, we can select arbitrarily different generator functions U₀ and U₁ so that the U₁-estimation gives consistency under the U₀-model. There is a natural question which situation happens if we consider the U-estimation under the U-model?

Let M_U be a U-model defined by

$$M_U=\left\{f_U\left(x,\theta \right):=u\left({\theta }^{\top }t\left(x\right)-{\kappa }_U\left(\theta \right)\right):\theta \in \Theta \right\},$$

(28)

where Θ = {θ ∈ ℝ^k : κ_U (θ) < ∞}. The the U-loss function under the U-model for a given data set {X₁, ···, X_n} is defined by

$$L_U\left(\theta \right)=-\frac{1}{n}\sum _{i=1}^n\xi \left(f_U\left(X_i,\theta \right)\right)+\int U\left(\xi \left(f_U\left(x,\theta \right)\right)\right)d\Lambda \left(x\right),$$

which is reduced to

$$L_U\left(\theta \right)=-{\theta }^{\top }\overline{t}+{\kappa }_U\left(\theta \right)+b_U\left(\theta \right),$$

(29)

where $\overline{t}=\frac{1}{n}{\sum }_{i=1}^{n}t(X_i)$ and

$$b_U\left(\theta \right)=\int U\left(\xi \left({\theta }^{\top }t\left(x\right)-{\kappa }_U\left(\theta \right)\right)\right)d\Lambda \left(x\right).$$

(30)

The estimating equation is given by

$$\frac{\partial }{\partial \theta }{L}_U\left(\theta \right)-\overline{t}+\frac{\partial }{\partial \theta }{\kappa }_U\left(\theta \right)+\frac{\theta }{\partial \theta }{b}_U\left(\theta \right),$$

which is written by

$$\frac{\partial }{\partial \theta }{L}_U\left(\theta \right)=-\overline{t}+{𝔼}_{f\left(\cdot ,\theta \right)}\left\{t\left(X\right)\right\}.$$

Hence, if we consider the U-estimator for a parameter η by the transformation of θ defined by ϕ(θ) = 𝔼_f(_·,θ){t(X)}, then the U-estimator η̂_U is nothing but the sample mean t̄. Here we confirm that the transformation ϕ(θ) is one-to-one as follows. The Jacobian matrix of the transformation is given by

$$\frac{\partial }{\partial \theta }\varphi \left(\theta \right)=\int u^{\prime }\left({\theta }^{\top }t\left(x\right)-{\kappa }_U\left(\theta \right)\right)\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U\left(\theta \right))\right\}{\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U\left(\theta \right))\right\}}^{\top }d\Lambda \left(x\right),$$

since the first identity for M_U leads to

$$\frac{\partial }{\partial \theta }\int f_U\left(x,\theta \right)d\Lambda \left(x\right)=\int u^{\prime }\left({\theta }^{\top }t\left(x\right)-{\kappa }_U\left(\theta \right)\right)\left\{t\left(x\right)-\frac{\partial }{\partial \theta }{\kappa }_U\left(\theta \right)\right\}d\Lambda \left(x\right)=0.$$

Therefore, we conclude that the Jacobian matrix is symmetric and positive-definite since u′ (t) is a positive function from the assumption of the convexity for U, which implies that ϕ(θ) is one-to-one. Consequently, the estimator θ̂ _U for θ is given by ϕ^–1(t̄). We summarize these results in the following theorem.

Theorem 5. Let M_U be a U-model with a canonical statistic t(X) as defined in Equation (28). Then the U-estimator for the expectation parameter η of t(X) is always t̄, where $\overline{t}=\frac{1}{n}{\sum }_{i=1}^{n}t(X_i)$.

Remark 2. We remark that the empirical Pythagorean theorem holds as in

$$L_U\left(\theta \right)=L_U\left({\widehat{\theta }}_U\right)+D_U\left({\widehat{\theta }}_U,\theta \right),$$

since we observe that

$$L_U\left(\theta \right)-L_U\left({\widehat{\theta }}_U\right)={\left({\widehat{\theta }}_U-\theta \right)}^{\top }\overline{t}+{\kappa }_U\left(\theta \right)+b_U\left(\theta \right)-{\kappa }_U\left({\widehat{\theta }}_U\right)+b_U\left({\widehat{\theta }}_U\right),$$

which gives another proof for which θ̂_U is ϕ^–1(t̄). The statistic t̄ is a sufficient statistic in the sense that the U-loss function L_U (θ) is a function of t̄ as in Equation (29). Accordingly, the U-estimator under U-model is a function only of t̄ from the observations X₁, ···, X_n. In this extension, the MLE is a function of t̄under the exponential model with the canonical statistic t(X).

Let us look at the case of the β-power divergence. Under the β-power model given by

$$M_{\beta }=\left\{f_{\beta }\left(x,\theta \right):={\left\{{\kappa }_{\beta }\left(\theta \right)+\beta {\theta }^{\top }t\left(x\right)\right\}}^{\frac{1}{\beta }}:\theta \in \Theta \right\},$$

the β-loss function is written by

$$L_{\beta }\left(\theta \right)=-\beta {\theta }^{\top }\overline{t}+{\kappa }_{\beta }\left(\theta \right)+b_{\beta }\left(\theta \right),$$

where

$$b_{\beta }\left(\theta \right)=\frac{1}{\beta +1}\int {\left\{{\kappa }_{\beta }\left(\theta \right)+\beta {\theta }^{\top }t\left(x\right)\right\}}^{\frac{1+\beta }{\beta }}d\Lambda \left(x\right).$$

The β-power estimator for the expectation parameter of t(X) is exactly given by t̄.

7. Discussion

We concentrate on elucidating the dual structure of the U-estimator under the U-model, in which the perspective extends the relation of the maximum likelihood under the exponential model with a functional degree of freedom. Thus, we explore a rich and practical class of duality structures; however, there remains an unsolved problem when we directly treat the space $ℱ$ as an differentiable manifold, see [39] for an infinite dimensional exponential family. The approach here is not a direct extension of an infinite dimensional manifold, but a path geometry in the following sense. For all pairs of elements of $ℱ$ the geodesic curve connecting the pair is represented in an explicit form in the class of *Γ^(U) connections in our context.

The U-divergence approach was the first trial to introduce a dually flat structure to $ℱ$ which is different from the alpha-geometry. However, there are many related studies. For example, a nonparametric information geometry on the space of all functions without constraints for positivity and normalizing is discussed in Zhang [40]. Amari [41] characterizes (ρ, τ)-divergence with decomposable dually flat structure, see also [42]. If ρ is an identity function and τ(s) = (d/ds)U(s), (ρ, τ)-divergence is no less than U-divergence. In effect we confine ourselves to discussing the U-divergence class for the sake of the direct estimability for U-estimator.

The duality between the maximum entropy and the minimum divergence has been explored in the minimax theorem for a zero-sum game between a decision maker and Nature. The pay-off function is taken by the cross U-entropy in which Nature tries to maximize the pay-off function under the mean equal constraint; the decision maker tries to minimize the pay-off function. The equilibrium is given by the minimax solution, which is the maximum U-entropy distribution, see [43] for the extensive discussion and the relation with Bayesean robustness. The observation explored in this paper is closely related with this minimax argument, however the duality between the statistical model and estimation is focused on, where the minimum U-divergence leads to projection onto the U-model.

In principle, the U-estimator is applicable for all the statistical model since U-loss function is written by a sample as well as the log-likelihood function. If the choice of the model is different from the U-model, then U-estimator has different performance from the present situation. For example, we consider an exponential model (U(s) = exp(s)), and a β-estimator (U(s) = (1 – βs)^1/β for getting a robustness property for outlying observations, cf. [19,20]. In such situations, the duality property is no longer valid, since the β-estimator for the parameter of the exponential model is not a function of the sufficient statistic t̄ defined in Theorem 5. Thus, we have to pay attention to another aspect than the duality structure in the presence of outlying, or misspecification for the statistical model. Furthermore, another type of divergence measures including projective power divergence is recommended to perform super robustness, cf. [21,44].

We presented the method of generalized maximum entropy based on the proposed entropy measure, as an extension of the classical maximum entropy method based on the Boltzmann-Gibbs-Shannon entropy. Practical applications of MaxEnt are actively followed in ecological and computational linguistic researches based on the classical maximum entropy, cf. [45,46]. Difficult aspects are discussed, in which the MaxEnt is apt to be over-learning on data sets because it basically employs the maximum likelihood estimator. There is a great potential for the proposed method to implement these research fields in order to overcome these difficult aspects, by selecting an appropriate generator function. A detailed discussion is beyond the scope of the present paper; however, it will be challenged in the near future with concrete objectives motivated by real data analysis.