Nonparametric Information Geometry: From Divergence Function to Referential-Representational Biduality on Statistical Manifolds

Abstract

Divergence functions are the non-symmetric “distance” on the manifold, ${\mathcal{M}}_{\theta }$, of parametric probability density functions over a measure space, $(X,\mu )$. Classical information geometry prescribes, on ${\mathcal{M}}_{\theta }$: (i) a Riemannian metric given by the Fisher information; (ii) a pair of dual connections (giving rise to the family of α-connections) that preserve the metric under parallel transport by their joint actions; and (iii) a family of divergence functions (α-divergence) defined on ${\mathcal{M}}_{\theta }\times {\mathcal{M}}_{\theta }$, which induce the metric and the dual connections. Here, we construct an extension of this differential geometric structure from ${\mathcal{M}}_{\theta }$ (that of parametric probability density functions) to the manifold, $\mathcal{M}$, of non-parametric functions on X, removing the positivity and normalization constraints. The generalized Fisher information and α-connections on $\mathcal{M}$ are induced by an α-parameterized family of divergence functions, reflecting the fundamental convex inequality associated with any smooth and strictly convex function. The infinite-dimensional manifold, $\mathcal{M}$, has zero curvature for all these α-connections; hence, the generally non-zero curvature of ${\mathcal{M}}_{\theta }$ can be interpreted as arising from an embedding of ${\mathcal{M}}_{\theta }$ into $\mathcal{M}$. Furthermore, when a parametric model (after a monotonic scaling) forms an affine submanifold, its natural and expectation parameters form biorthogonal coordinates, and such a submanifold is dually flat for $\alpha =\pm 1$, generalizing the results of Amari’s α-embedding. The present analysis illuminates two different types of duality in information geometry, one concerning the referential status of a point (measurable function) expressed in the divergence function (“referential duality”) and the other concerning its representation under an arbitrary monotone scaling (“representational duality”).

1. Introduction

Information geometry is a differential geometric study of the manifold of probability measures or probability density functions [1]. Its role in understanding asymptotic inference was summarized in [2,3,4,5,6,7]. Information geometric methods have been applied to many areas of interest to statisticians, such as the study of estimating functions (e.g., [8,9]) and invariant priors for Bayesian inference (e.g., [10,11]), and to the machine learning community, such as the natural gradient descent method [12,13], support vector machine [14], boosting [15], turbo decoding [16], etc.

The differential geometric structure of statistical models with finite parameters is now well understood. Consider a family of probability functions (i.e., probability measures on discrete support or probability density functions on continuous support) as parameterized by $\theta =[{\theta }^1,\cdots ,{\theta }^n]$. The collection of such probability functions, where each function is indexed by a point, $\theta \in {\mathbb{R}}^n$, forms a manifold, ${\mathcal{M}}_{\theta }$, under suitable conditions. Rao [17] identified Fisher information to be the Riemannian metric for ${\mathcal{M}}_{\theta }$. Efron [18], through investigating a one-parameter family of statistical models, elucidated the meaning of curvature for asymptotic statistical inference and pointed out its flatness for the exponential model. In his reaction to Efron’s work, Dawid [19] invoked the differential geometric notion of linear connections on a manifold as preserving parallelism during vector transportation and pointed out other possible constructions of linear connections on ${\mathcal{M}}_{\theta }$, in addition to the non-flat Levi-Civita connection associated with the Fisher metric. Amari [2,20], in his path-breaking work, systematically advanced the theory of information geometry by constructing a parametric family of α-connections, ${\Gamma }^{\left(\alpha \right)}$, $\alpha \in \mathbb{R}$, along with a dualistic interpretation of $\alpha \leftrightarrow -\alpha$ as conjugate connections on the manifold, ${\mathcal{M}}_{\theta }$. The e-connection ($\alpha =1$) vanishes (i.e., becomes identically zero) on the manifold of the exponential family of probability functions under natural parameters, whereas the m-connection ($\alpha =-1$) vanishes on the manifold of the mixture family of probability functions under mixture parameters. Therefore, not only have ${\Gamma }^{(\pm 1)}$ zero curvatures for both exponential and mixture families, but affine coordinates were found to yield ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{(-1)}$, themselves zero for the exponential and mixture families, respectively.

This classic information geometry dealing with parametric statistical models has been investigated in the non-parametric setting using the tools of infinite-dimensional analysis [21,22,23], with non-parametric Fisher information given by [23]. This is made possible, because topological issues were resolved by the pioneering work of [24] using the theory of Orlicz space for charting the exponential statistical manifold. Zhang and Hasto [25] characterized the probability manifold modeled on an ambient affine space via functional equations and generalized exponential charts. The goal of the present paper is to extend these non-parametric results by showing links among three inter-connected mathematical topics that underlie information geometry, namely: (i) divergence functions measuring the non-symmetric distance of any two points (density or measurable functions) on the manifold (the referential duality); (ii) convex analysis and the associated Legendre–Fenchel transformation linking the natural and expectation parameters of parametric models (the representational duality); and (iii) the resulting dual Riemannian structure involving the Fisher metric and the family of α-connections. Results in the parametric setting were summarized in [26].

The Riemannian manifold of parametric statistical models is a special kind, one that involves dual (as known as conjugate) connections; historically, such a mathematical theory was independently developed to investigate hypersurface immersion (see [27,28]). Lauritzen [29] characterized the general differential geometric context under which a one-parameter family of α-connections arise, as well as the meaning of conjugacy for a pair of connections on statistical manifolds [30]. Kurose [31,32] and then Matsuzoe [33,34] elucidated information geometry from an affine differential geometric perspective. See, also [35] for a generalized notion of conjugate connections. It was Eguchi [36,37,38] who provided a generic way for inducing a metric and a pair of conjugate connections from an arbitrary divergence (what he called “contrast”) function. The current exposition will build on this “Eguchi relation” between the metric and conjugate connections of the Riemannian manifold, ${\mathcal{M}}_{\theta }$, and the divergence function defined on ${\mathcal{M}}_{\theta }\times {\mathcal{M}}_{\theta }$.

The main results of this paper include the introduction of an α-parametric family of divergence functionals on measurable functions (including probability functions) using any smooth and strictly convex function and the induction by such divergence a metric and a family of conjugate connections that resemble, but generalize, the Fisher information proper and α-connections proper. In particular, we derive explicit expressions of the metric and conjugate connections on the infinite-dimensional manifold of all functions defined on the same support of the sample space. When finite-dimensional affine embedding is allowed, our formulae reduce to the familiar ones associated with the exponential family established in [2]. We carefully delineate two senses of duality associated with such manifolds, one related to the reference/comparison status of any pair of points (functions) and the other related to properly scaled representations of them.

1.1. Parametric Information Geometry Revisited

Here, we briefly summarize the well-known results of parametric information geometry in the classical (as opposed to quantum) sense. The motivation is two-fold. First, by reviewing the basic parametric results, we want to make sure that any generalization of the framework of information geometry will reduce to those formulae under appropriate conditions. Secondly, understanding how a divergence function is related to the dual Riemannian structure will enable us to approach the infinite-dimensional case by analogy, that is, through constructing more general classes of divergence functionals defined on function spaces.

1.1.1. Riemannian Manifold, Fisher Metric and α-Connections

Let $(\mathcal{X},\mu )$ be a measure space with σ-algebra built upon the atoms, $d\zeta$, of $\mathcal{X}$. Let ${\mathcal{M}}_{\mu }$ denote the space of probability density functions, $p:\mathcal{X}\to {\mathbb{R}}_{+}(\equiv {\mathbb{R}}^{+}\cup \left\{0\right\})$, defined on the sample space, $\mathcal{X}$, with background measure $d\mu =\mu \left(d\zeta \right)$:

$${\mathcal{M}}_{\mu }=\left\{p\left(\zeta \right):E_{\mu }\left\{p\left(\zeta \right)\right\}=1;p\left(\zeta \right)>0,\forall \zeta \in \mathcal{X}\right\}$$

(1)

Here, and throughout this paper, $E_{\mu }\{·\}={\int }_{\mathcal{X}}\{·\}d\mu$ denotes the expectation of a measurable function (in curly brackets) with respect to the background measure, μ. We also denote $E_p\{·\}={\int }_{\mathcal{X}}\{·\}pd\mu$.

A parametric family of density functions, $p(·|\theta )$, called a parametric statistical model, is the association of a density function, $\theta \mapsto p(·|\theta )$, for each n-dimensional vector $\theta =[{\theta }^1,\cdots ,{\theta }^n]$. The space of parametric statistical models forms a Riemannian manifold (where θ is treated as the local chart):

$${\mathcal{M}}_{\theta }=\{p\left(\zeta \right|\theta )\in {\mathcal{M}}_{\mu }:\theta \in \Theta \subset {\mathbb{R}}^n\}\subset {\mathcal{M}}_{\mu }$$

(2)

with the so-called Fisher metric [17]:

$$g_{ij}\left(\theta \right)=E_{\mu }\left\{p\left(\zeta \right|\theta )\frac{\partial logp\left(\zeta \right|\theta )}{\partial {\theta }^i}\frac{\partial logp\left(\zeta \right|\theta )}{\partial {\theta }^j}\right\}$$

(3)

and α-connections [20,39]:

$${\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)=E_{\mu }\left\{p\left(\zeta \right|\theta )\left(\frac{1-\alpha }{2}\frac{\partial logp\left(\zeta \right|\theta )}{\partial {\theta }^i}\frac{\partial logp\left(\zeta \right|\theta )}{\partial {\theta }^j}+\frac{{\partial }^{2}logp\left(\zeta \right|\theta )}{\partial {\theta }^i\partial {\theta }^j}\right)\frac{\partial logp\left(\zeta \right|\theta )}{\partial {\theta }^k}\right\}$$

(4)

with the α-connections satisfying the dualistic relation:

$${\Gamma }_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)={\Gamma }_{ij,k}^{(-\alpha )}\left(\theta \right)$$

(5)

Here, *, denotes conjugate (dual) connection. Recall that, in general, a metric is a bilinear map on the tangent space, and an affine connection is used to define parallel transport of vectors. The conjugacy in a pair of connections, $\Gamma ⟷{\Gamma }^{*}$, is defined by their jointly preserving the metric when each acts on one of the two tangent vectors; that is, when the tangent vectors undergo parallel transport according to Γ or ${\Gamma }^{*}$ respectively. Equivalently, and perhaps more fundamentally, the pair of conjugate connections preserve the dual pairing of vectors in the tangent space with co-vectors in the cotangent space [30]. Any Riemannian manifold with its metric, g, and conjugate connections, $\Gamma ,{\Gamma }^{*}$, given in the form of Equations (3)–(5) is called a statistical manifold (in the narrower sense) and is denoted as $\{{\mathcal{M}}_{\theta },g,{\Gamma }^{(\pm \alpha )}\}$. In the broader sense, a statistical manifold $\{\mathcal{M},g,\Gamma ,{\Gamma }^{*}\}$ is a differentiable manifold equipped with a Riemannian metric g and a pair of torsion-free conjugate connections $\Gamma \equiv {\Gamma }^{\left(1\right)},{\Gamma }^{*}\equiv {\Gamma }^{(-1)}$, without necessarily requiring g and $\Gamma ,{\Gamma }^{*}$ to take the forms of Equations (3)–(5).

1.1.2. Exponential Family, Mixture Family and Their Generalization

An exponential family of probability density functions is defined as:

$$p^{\left(e\right)}\left(\zeta \right|\theta )=exp\left(F_0\left(\zeta \right)+\sum _i{\theta }^i{F}_i\left(\zeta \right)-\mathrm{\Phi }\left(\theta \right)\right)$$

(6)

where θ is its natural parameter and $F_i\left(\zeta \right)(i=1,\cdots ,n)$ is a set of linearly independent functions with the same support in $\mathcal{X}$, and the cumulant generating function (“potential function”) $\mathrm{\Phi }\left(\theta \right)$ is:

$$\mathrm{\Phi }\left(\theta \right)=log{E}_{\mu }\left\{exp\left(F_0\left(\zeta \right)+\sum _i{\theta }^i{F}_i\left(\zeta \right)\right)\right\}$$

(7)

Substituting Equation (6) into Equations (3) and (4), the Fisher metric and the α-connections are simply:

$$g_{ij}\left(\theta \right)=\frac{{\partial }^2\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j}$$

(8)

and:

$${\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)=\frac{1-\alpha }{2}\frac{{\partial }^3\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j\partial {\theta }^k}$$

(9)

whereas the Riemannian curvature tensor (of an α-connection) is given by [2], p.106:

$$R_{ij\mu \nu }^{\left(\alpha \right)}\left(\theta \right)=\frac{1-{\alpha }^2}{4}\sum _{l,k}({\mathrm{\Phi }}_{il\nu }{\mathrm{\Phi }}_{jk\mu }-{\mathrm{\Phi }}_{il\mu }{\mathrm{\Phi }}_{jk\nu }){\mathrm{\Phi }}^{lk}$$

(10)

where ${\mathrm{\Phi }}^{ij}=g^{ij}$ is the matrix inverse of $g_{ij}$ and subscripts of Φ indicate partial derivatives. Therefore, the α-connection for the exponential family is dually flat when $\alpha =\pm 1$. In particular, all components of ${\Gamma }_{ij,k}^{\left(1\right)}$ vanish, due to Equation (9), on the manifold formed by $p^{\left(e\right)}(·|\theta )$ in which the natural parameter, θ, serves as the local coordinates.

On the other hand, the mixture family:

$$p^{\left(m\right)}\left(\zeta \right|\theta )=\sum _i{\theta }^i{F}_i\left(\zeta \right)$$

(11)

when viewed as a manifold charted by its mixture parameter, θ, with the constraints, ${\sum }_i{\theta }^i=1$ and ${\int }_X{F}_i\left(\zeta \right)d\mu =1$, turns out to have identically zero ${\Gamma }_{ij,k}^{(-1)}$. The connections, ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{(-1)}$, are also called the exponential and mixture connections, or e- and m-connection, respectively. The exponential family and the mixture family are special cases of the α-family [1,2] of density functions, $p\left(\zeta \right|\theta )$, whose denormalization satisfies (with constant κ):

$$l^{\left(\alpha \right)}\left(\kappa p\right)=F_0\left(\zeta \right)+\sum _i{\theta }^i{F}_i\left(\zeta \right)$$

(12)

under the α-embedding function, $l^{\left(\alpha \right)}:{\mathbb{R}}^{+}\to \mathbb{R}$, defined as:

$$l^{\left(\alpha \right)}\left(t\right)=\left\{\begin{array}{cc}logt& \alpha =1\\ \frac{2}{1-\alpha }{t}^{(1-\alpha )/2}& \alpha \ne 1\end{array}\right.$$

(13)

The α-embedding of a probability density function plays an important role in Tsallis statistics; see, e.g., [40]. Under α-embedding, the denormalized density functions form the so-called α-affine manifold [1], p.46. The Fisher metric and α-connections, under such α-representation, have the following expressions:

$$\begin{array}{ccc}\hfill g_{ij}\left(\theta \right)& =& E_{\mu }\left\{\frac{\partial l^{\left(\alpha \right)}\left(p(·|\theta )\right)}{\partial {\theta }^i}\frac{\partial l^{(-\alpha )}\left(p(·|\theta )\right)}{\partial {\theta }^j}\right\}\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)& =& E_{\mu }\left\{\frac{{\partial }^2{l}^{\left(\alpha \right)}\left(p(·|\theta )\right)}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial l^{(-\alpha )}\left(p(·|\theta )\right)}{\partial {\theta }^k}\right\}\hfill \end{array}$$

(15)

Clearly, on an α-affine manifold with any given α value, components of ${\Gamma }^{\left(\alpha \right)}$ are all identically zero by virtue of the definition of α-family Equation (12), and hence, $\pm \alpha$-connections are dually flat.

1.2. Divergence Function and Induced Statistical Manifold

It is well known that the statistical manifold, $\{{\mathcal{M}}_{\theta },g,{\Gamma }^{(\pm \alpha )}\}$, with Fisher information as the metric, g, and the $(\pm \alpha )$-connections, ${\Gamma }^{(\pm \alpha )}$, as conjugate connections, can be induced from a parametric family of divergence functions called “α-divergence”. Here, we briefly review the link of divergence functions to the dual Riemannian geometry of statistical manifolds.

1.2.1. Kullback-Leibler Divergence, Bregman Divergence and α-Divergence

Divergence functions are distance-like quantities; they measure the directed (non-symmetric) difference of two probability density functions in the infinite-dimensional function space or two points in a finite-dimensional vector space of the parameters of a statistical model. An example is the Kullback-Leibler divergence (also known as, KL cross-entropy) between two probability densities, $p,q\in {\mathcal{M}}_{\mu }$, here expressed in its extended form (i.e., without requiring p and q to be normalized):

$$K(p,q)=\int \left(q-p-plog\frac{q}{p}\right)d\mu =K^{*}(q,p)$$

(16)

with a unique, global minimum of zero when $p=q$. For the exponential family Equation (6), the expression (16) takes the form of the so-called Bregman divergence [41] defined on $\Theta \times \Theta \subseteq {\mathbb{R}}^n\times {\mathbb{R}}^n$:

$${\mathrm{B}}_{\mathrm{\Phi }}({\theta }_p,{\theta }_q)=\mathrm{\Phi }\left({\theta }_p\right)-\mathrm{\Phi }\left({\theta }_q\right)-\langle {\theta }_p-{\theta }_q,\partial \mathrm{\Phi }\left({\theta }_q\right)\rangle$$

(17)

where Φ is the potential function (7), ∂ is the gradient operator and $\langle ·,·\rangle$ denotes the standard bilinear form (pairing) of a vector with a co-vector. The Bregman divergence (17) expresses the directed-distance of two members, p and q, of the exponential family as indexed, respectively, by the two parameters, ${\theta }_p$ and ${\theta }_q$.

A generalization of the Kullback-Leibler divergence is the α-divergence, defined as:

$${\mathcal{A}}^{\left(\alpha \right)}(p,q)=\frac{4}{1-{\alpha }^2}{\text{E}}_{\mu }\left\{\frac{1-\alpha }{2}p+\frac{1+\alpha }{2}q-p^{\frac{1-\alpha }{2}}{q}^{\frac{1+\alpha }{2}}\right\}$$

(18)

measuring the directed distance between any two density functions, p and q. It is easily seen that:

$$\underset{\alpha \to -1}{lim}{\mathcal{A}}^{\left(\alpha \right)}(p,q)=K(p,q)=K^{*}(q,p)$$

(19)

$$\underset{\alpha \to 1}{lim}{\mathcal{A}}^{\left(\alpha \right)}(p,q)=K^{*}(p,q)=K(q,p)$$

(20)

Note that traditionally (see [2,20]), the term $\frac{1-\alpha }{2}p+\frac{1+\alpha }{2}q$ is replaced by 1 in the integrand of Equation (18), and the term $q-p$ is absent in the integrand of Equation (16); this is trivially true when $p,q$ are probability densities with a normalization of one. Zhu and Rohwer [42,43], in what they called the δ-divergence, $\delta =\frac{1-\alpha }{2}$, supplied these extra terms as the “extended” forms of α-divergence and of Kullback-Leibler divergence). The importance of these terms will be seen later (Section 2.2).

Note that, strictly speaking, when the underlying space is a finite-dimensional vector space, that is, the space, ${\mathbb{R}}^n$, for the parameters, θ, of a statistical model, $p(·|\theta )$, then the term “divergence function” is appropriate. However, if the underlying sample space is infinite-dimensional that may be uncountable, that is, the manifold, ${\mathcal{M}}_{\mu }$, of non-parametric probability densities, p and q, then the term “divergence functional” seems more appropriate. The latter implicitly defines a divergence function (through pullback) if the probability densities are embedded into a finite-dimensional submanifold, ${\mathcal{M}}_{\theta }$, in the case of a parametric statistical model, $p(·|\theta )$. As an example, for the exponential family Equation (6), the Kullback-Leibler divergence Equation (16) in terms of p and q, implicitly defines a divergence in terms of ${\theta }_p$, ${\theta }_q$, i.e., the Bregman divergence Equation (17). In the following, we use the term divergence function when we intend to blur the distinction between whether it is defined on the finite-dimensional vector space or on the infinite-dimensional function space and, in the latter case, whether it is pulled back into the finite dimensional submanifold. We will, however, use the term divergence functional when we emphasize the infinite-dimensional setting sans parametric embedding.

In general, a divergence function (also called “contrast function”) is non-negative for all $p,q$ and vanishes only when $p=q$; it is assumed to be sufficiently smooth. A divergence function will induce a Riemannian metric, g, in the form of Equation (3) by its second order properties and a pair of conjugate connections, $\Gamma ,{\Gamma }^{*}$, in the forms of Equations (4) and (5) by its third order properties (relations were first formulated by Eguchi [36,37], which we are going to review next).

1.2.2. Induced Dual Riemannian Geometry

Let $\mathcal{M}$ be a Riemannian manifold endowed with a metric tensor field, g, whose restriction to any point, p, is a symmetric, positive bilinear form, $\langle ,\rangle$, on $T_p\left(\mathcal{M}\right)\times T_p\left(\mathcal{M}\right)$. Here, $T_p\left(\mathcal{M}\right)$ denotes the space of all tangent vectors at the point, $p\in \mathcal{M}$, and $\Sigma \left(\mathcal{M}\right)$ denotes the collection of all vector fields on $\mathcal{M}$. Then:

$$g(u,v)=\langle u,v\rangle$$

(21)

with $u,v\in \Sigma \left(\mathcal{M}\right)$. Let $w\in \Sigma \left(\mathcal{M}\right)$ be another vector field, and $d_w$ denotes the directional derivative (of a function, vector field, etc.) along the direction corresponding to w (taken at any given point, p, if explicitly written out). An affine connection, ∇, is a map, $\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \Sigma \left(\mathcal{M}\right),(w,u)\mapsto {\mathrm{\nabla }}_{w}u$, that is linear in $u,w$, while $\mathcal{F}$-linear in w, but not in u. A pair of connections, ∇, ${\mathrm{\nabla }}^{*}$, are said to be conjugate to each other if:

$$d_{w}g(u,v)=\langle {\mathrm{\nabla }}_{w}u,v\rangle +\langle u,{\mathrm{\nabla }}_w^{*}v\rangle$$

(22)

or in component form, denoted by $\Gamma ,{\Gamma }^{*}$:

$${\partial }_k{g}_{ij}={\Gamma }_{ki,j}+{\Gamma }_{kj,i}^{*}$$

(23)

The “contravariant” form, ${\Gamma }_{ij}^l$, of the affine connection defined by:

$${\mathrm{\nabla }}_{{\partial }_i}{\partial }_j=\sum _l{\Gamma }_{ij}^l{\partial }_l$$

(24)

is related to the “covariant” form, ${\Gamma }_{ij,k}$ through:

$$\sum _l{g}_{lk}{\Gamma }_{ij}^l={\Gamma }_{ij,k}$$

(25)

The Riemannian metric, g, and conjugate connections, $\mathrm{\nabla },{\mathrm{\nabla }}^{*}$, on a statistical manifold can be induced by a divergence function, $\mathcal{D}:\mathcal{M}\times \mathcal{M}\to {\mathbb{R}}_{+}$, which, by definition, satisfies:

(i)

$\mathcal{D}(p,q)\ge 0\forall p,q\in \mathcal{M}$ with equality holding iff $p=q$;

(ii)

${\left(d_u\right)}_p{\left.\mathcal{D}(p,q)\right|}_{p=q}={\left(d_v\right)}_q{\left.\mathcal{D}(p,q)\right|}_{p=q}=0$;

(iii)

$-{\left(d_u\right)}_p{\left(d_v\right)}_q{\left.\mathcal{D}(p,q)\right|}_{p=q}$ is positive definite;

where the subscript, $p,q$, means that the directional derivative is taken with respect to the first and second arguments in $\mathcal{D}(p,q)$, respectively, along the direction, u or v. Eguchi [36,37] showed that any such divergence function, $\mathcal{D}$, satisfying (i)–(iii) will induce a Riemannian metric, g, and a pair of connections, $\mathrm{\nabla },{\mathrm{\nabla }}^{*}$ via:

$$\begin{array}{ccc}\hfill g(u,v)& =& {\left.-{\left(d_u\right)}_p{\left(d_v\right)}_q\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill \langle {\mathrm{\nabla }}_{w}u,v\rangle & =& {\left.-{\left(d_w\right)}_p{\left(d_u\right)}_p{\left(d_v\right)}_q\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill \langle u,{\mathrm{\nabla }}_w^{*}v\rangle & =& {\left.-{\left(d_w\right)}_q{\left(d_v\right)}_q{\left(d_u\right)}_p\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(28)

In index-laden component forms, they are:

$$\begin{array}{ccc}\hfill g_{ij}& =& {\left.-{\left({\partial }_i\right)}_p{\left({\partial }_j\right)}_q\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill {\Gamma }_{ij,k}& \equiv & \langle {\mathrm{\nabla }}_{{\partial }_i}{\partial }_j,{\partial }_k\rangle ={\left.-{\left({\partial }_i\right)}_p{\left({\partial }_j\right)}_p{\left({\partial }_k\right)}_q\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill {\Gamma }_{ij,k}^{*}& \equiv & \langle {\partial }_k,{\mathrm{\nabla }}_{{\partial }_i}^{*}{\partial }_j\rangle ={\left.-{\left({\partial }_i\right)}_q{\left({\partial }_j\right)}_q{\left({\partial }_k\right)}_p\mathcal{D}(p,q)\right|}_{p=q}\hfill \end{array}$$

(31)

Equations (26)–(28) in coordinate-free form, or Equations (29)–(31) in index-laden form, link a divergence function, $\mathcal{D}$, to the Riemannian metric, g, and conjugate connections, $\mathrm{\nabla },{\mathrm{\nabla }}^{*}$; henceforth, they will be called the Eguchi relation. It is easily verifiable that they satisfy Equation (22) or Equation (23), respectively. These relations are the stepping stones going from a divergence function defining (generally) non-symmetric distances between a pair of points on a manifold at large to the dual Riemannian geometric structure on the same manifold in the small. To apply to the infinite-dimensional context, we provide a proof (in Section 4) for the coordinate-free version Equations (26)–(28). This will allow us to first construct divergence functional on the infinite-dimensional function space (the Kullback-Leibler divergence being a special example) and then derive explicit expressions for the non-parametric Riemannian metric and conjugate connections by explicating $d_u,d_v,d_w$.

1.3. Goals and Approach

Our goals in this paper are several-fold. First, we want to provide a unified perspective for the divergence functions encountered in the literature. There are two broad classes, those defined on the infinite-dimensional function space and those defined on the finite-dimensional vector space. The former class include the one-parameter family of α-divergence (equivalently, the δ-divergence in [42,43]), the family of Jensen difference related to the Shannon entropy function [44], both specializing to Kullback-Leibler divergence as a limiting case. The latter class includes the Bregman divergence [41], also called “geometric divergence” [32], which turns out to be identical to the “canonical divergence” [1] on a dually flat manifold expressed in a pair of biorthogonal coordinates; those coordinates are induced by a pair of conjugate convex functions via the Legendre–Fenchel transform [2,20]. [15] recently investigated an infinite-dimensional version of the Bregman divergence, called the U-divergence. It will be shown in this paper that all of the above-mentioned divergence functions can be understood as convex inequalities associated with some real-valued, strictly convex function defined on $\mathbb{R}$ (for the infinite-dimensional case) or ${\mathbb{R}}^n$ (for the finite-dimensional case), with the convex mixture parameter assuming the role of α in the induced α-connection. Note that $\alpha ⟷-\alpha$ in such divergence functions corresponds to an exchange of the two points the divergence functions measure (generally in a non-symmetric fashion), while $\alpha ⟷-\alpha$ in the induced connections corresponds to the conjugacy operation for the pairing of two metric-compatible connections. Hence, our approach to divergence functions from convex analysis will address both of these aspects coherently, and an intimate relation between these two senses of duality is expected to emerge from our formulation (see below).

The second goal of our paper is to provide a more general form for the Fisher metric Equation (3) and the α-connections Equation (4) (or equivalently, Equations (14) and (15) under α-embedding), while still staying within the framework of [29] in characterizing statistical manifolds. One specific aim is to derive explicit expressions for the Fisher metric and α-connections for the infinite-dimensional case. In the past, infinite-dimensional expression for the α-connection ${\mathrm{\nabla }}^{\left(\alpha \right)}$, as a mixture of ${\mathrm{\nabla }}^{\left(1\right)}$ and ${\mathrm{\nabla }}^{(-1)}$, has emerged, but was given only implicitly with their interpretations debated [22,23]. Our approach exploits the coordinate-free version of the Eguchi relations Equations (26)–(28) directly, and derives Fisher metric and α-connections from the general form of divergence functions mentioned in the last paragraph. The affine connection, ${\mathrm{\nabla }}^{\left(\alpha \right)}$, is formulated as the covariant derivative, which is characterized by a bilinear form. Since our divergence functional will be defined on the infinite-dimensional manifold, $\mathcal{M}$, without restricting the underlying functions (individual points on $\mathcal{M}$) to be normalized and positively-valued, the affine connections we derive are expected to have zero Riemann curvature as those in the ambient space. From this perspective, statistical curvature (the curvature of a statistical manifold) can be viewed as an embedding curvature, that is, curvature arising out of restricting to the submanifold, ${\mathcal{M}}_{\mu }$, of normalized and positive-valued functions (i.e., non-parametric statistical manifold), and further to the finite-dimensional submanifold ${\mathcal{M}}_{\theta }$ (i.e., parametric statistical models).

Our third goal here is to clarify some fundamental issues in information geometry, including the meaning of duality and its relation to submanifold embedding. In its original development starting from [19], the flatness of the e-connection (or m-connection) is with respect to a particular family of density functions, namely, the exponential family (or mixture family). Later, Amari [2,20] generalized this observation to any α-family (i.e., a density function that is, after denormalization, affine under α-embedding): the α-connection is flat (indeed, ${\Gamma }_{ij,k}^{\left(\alpha \right)}$ vanishes) for the α-affine manifold (which is reduced to the exponential model for $\alpha =1$ and the mixture model for $\alpha =-1$). One may be led to infer that the α parameter in the α-connection and the α parameter in α-embedding are one and the same and, thereby, conclude that ${\mathrm{\nabla }}^{\left(1\right)}$-flatness (or ${\mathrm{\nabla }}^{(-1)}$-flatness) is exclusively associated with the exponential family expressed in its natural parameter (or the mixture family expressed in its mixture parameter). Here, we point out that these conclusions are unwarranted: the flatness of an α-connection and the embedding of a probability function into an affine submanifold under α-representation are two related, but separate, issues. We will show that the α-connections for the infinite-dimensional ambient manifold, $\mathcal{M}$, which contains the manifold of probability density functions, ${\mathcal{M}}_{\mu }$, as a submanifold, has zero (ambient) curvature for all α values. For finite-dimensional parametric statistical models, it is known that the α-connection will not in general have zero curvature even when $\alpha =\pm 1$. Here, we will give precise conditions under which ${\mathrm{\nabla }}^{(\pm 1)}$ will be dually flat—i.e., when the denormalized statistical model can be affine embedded under any ρ-representation, where a strictly increasing function $\rho :\mathbb{R}\to \mathbb{R}$ generalizes the α-embedding function (13). In such cases, there exists a strictly convex potential function, akin to Equation (7), for the exponential statistical model, that will reduce the Fisher metric and α-connections to the forms of Equations (8) and (9). One may define the natural parameter and expectation parameter that are dual to each other and that form biorthogonal coordinates for the underlying manifold, just as for the exponential family.

Our analysis will clarify two different kinds of duality in information geometry, one related to the different status of a reference probability function and a comparison probability function (referential duality), the other related to the representation of each probability function via a pair of conjugate scaling (representational duality). Roughly speaking, the $(\pm 1)$-duality reflects the former, whereas the $e/m$-duality reflects the latter. Previously, they were non-distinguished; in our analysis, we are able to disambiguate these two senses of duality. For instance, we are able to devise a two-parameter family of divergence functions, where the two parameters play distinct roles in the induced geometry, one capturing referential duality and the other capturing representational duality. Interestingly, this two-parameter family of connections still takes the same form of the α-connection proper (with a single parameter), indicating that this extension is still within [29]’s conceptualization of dual connections in information geometry.

The technical challenge that we have to overcome in our derivations is doing calculus in the infinite-dimensional setting. Consider the set of measurable functions from $\mathcal{X}$ to $\mathbb{R}$, which, in the presence of charts modeled on (open) subsets, ${\left\{E_i\right\}}_{i\in I}$, of a Banach space, form a manifold, $\mathcal{M}$, of infinite dimension. Each point on $\mathcal{M}$ is a function, $p:\mathcal{X}\to \mathbb{R}$, over the sample space $\mathcal{X}$; and each chart, $\mathcal{U}\subset \mathcal{M}$, is afforded with a bijective map to the Banach space with a suitable norm (e.g., Orlicz space, as adopted by [21,22,23,24,45]). For non-parametric statistical models, [24] provided exponential charts modeled on Orlicz spaces, which was followed by the rest of the above-referenced works. We do not restrict ourselves to probability density functions and work, in general, with measurable functions (without positivity and normalization requirements); we treat probability functions as forming a submanifold in $\mathcal{M}$ defined by the positivity and normalization conditions. This approach gives us certain advantages in deriving, from divergence functions directly, the Riemannian geometry on $\mathcal{M}$, whereby $\mathcal{M}$ serves as an ambient space to embed a statistical manifold, ${\mathcal{M}}_{\mu }$, as a submanifold in a standard way (by restricting the tangent vector field of $\mathcal{M}$). The usual interpretation of the affine connection on ${\mathcal{M}}_{\mu }$ as the projection of a natural connection on $\mathcal{M}$ is then “borrowed” over from the finite-dimensional setting to this infinite-dimensional setting. Our approach followed that of [46], who treats the infinite dimensional manifold as a generic $C^{\infty }$-Banach manifold and used the theory of canonical spray (and the Morse-Palais Lemma) to construct Riemannian metric and affine connections on such manifolds. However, we fell short of providing a topology on $\mathcal{M}$ as induced from the divergence functions and compare it with the one endowed by [24]. In particular, the conditions under which ${\mathcal{M}}_{\mu }$ forms a proper submanifold of $\mathcal{M}$ remain to be identified. Neither have we addressed topological issues for the well-definedness of conjugate connections on such infinite-dimensional manifolds. We refer the readers to [23], who investigated whether the entire family of α-connections is well-defined for $\mathcal{M}$ endowed with the same topology.

The structure of the rest of the paper is as follows. Section 2 will deal with information geometry under the infinite-dimensional setting and Section 3 under the finite-dimensional setting. For ease of presentation, results will be provided in the main text, while their proofs will be deferred to Section 4. Section 5 closes with a discussion of the implications of the current framework. A preliminary report of this work was presented to IGAIA2 (Tokyo) and appeared in [47].

2. Information Geometry on Infinite-Dimensional Function Space

In this section, we first review the basic apparatus of the differentiable manifold with particular emphasis paid to the infinite-dimensional (non-parametric) setting (Section 2.1). We then define a family of divergence functionals based on convex analysis (Section 2.2) and use them to induce the dual Riemannian geometry on the infinite-dimensional manifold (Section 2.3). The section is concluded with an investigation of a special case of homogeneous divergence, called $(\alpha ,\beta )$-divergence, in which the two parameters play distinct, but interrelated, roles for referential duality and representational duality, thereby generalizing the familiar α-divergence in a sensible way (Section 2.4).

2.1. Differentiable Manifold in the Infinite-Dimensional Setting

Let $\mathcal{U}$ be an open set on the base manifold, $\mathcal{M}$, containing a representative point, $x_0$, and $\mathrm{F}:\mathcal{U}\to \mathbb{R}$, a smooth function defined on this local patch, $\mathcal{U}\subset \mathcal{M}$. The set of smooth functions on $\mathcal{M}$ is denoted $\mathcal{F}\left(\mathcal{M}\right)$. A curve, $t\mapsto x\left(t\right)$, on the manifold is a collection of points, $\left\{x\right(t)\in \mathcal{U}:t\in [0,1\left]\right\}$, whereas a tangent vector (or simply “vector”), v at $x_0\in \mathcal{U}$, represents an equivalent class of curves passing through $x_0=x\left(0\right)$, all with the same direction and speed as specified by the vector, $v={\left.\frac{dx}{dt}\right|}_{t=0}$. We use $T_{x_0}\left(\mathcal{M}\right)$ to denote the space of all tangent vectors (“tangent space”) at a given $x_0$; it is obviously a vector space. The tangent manifold, $\mathcal{TM}$, is then the collection of tangent spaces for all points on $\mathcal{M}$: $\mathcal{TM}=\{\cup T_x\left(\mathcal{M}\right),x\in \mathcal{M}\}$. A vector field, $v\left(x\right)$, is the association of a vector, v, at each point, x, of the manifold, $\mathcal{M}$; it is a cross-section of $\mathcal{TM}$. The set of all smooth vector fields on $\mathcal{M}$ is denoted $\Sigma \left(\mathcal{M}\right)$. The tangent vector, v, acting on a function, $\mathrm{F}$, will yield a scalar, denoted $d_v\mathrm{F}$, called the direction derivative of $\mathrm{F}$:

$$d_v\mathrm{F}=\underset{t\to 0}{lim}\frac{1}{t}(\mathrm{F}\left(x\left(t\right)\right)-\mathrm{F}\left(x_0\right))$$

(32)

The tangent vector, v, acting on a vector field, $u\left(x\right)$, is defined analogously:

$$d_{v}u=\underset{t\to 0}{lim}\frac{1}{t}(u\left(x\left(t\right)\right)-u\left(x_0\right))$$

(33)

In our setting, given a measure space, $(\mathcal{X},\mu )$, where samples are drawn from the set $\mathcal{X}$ and μ is the background measure, we call any function that maps $\mathcal{X}\to \mathbb{R}$ a ζ-function. The set of all ζ-functions forms a vector space, where vector addition is point-wise: $(f_1+f_2)\left(\zeta \right)=f_1\left(\zeta \right)+f_2\left(\zeta \right)$, and scalar multiplication is simple multiplication: $\left(cf\right)\left(\zeta \right)=cf\left(\zeta \right)$. We now consider the set of all ζ-functions with common support μ, which is assumed to form a manifold denoted as ${\mathcal{M}}_{\mu }$. A typical point of this manifold denotes a specific ζ-function, $p\left(\zeta \right):\zeta \to p\left(\zeta \right)$, defined over $\mathcal{X}$, the sample space, which is infinite dimensional or even uncountable in general. Under suitable topology (e.g., [24]), all points, ${\mathcal{M}}_{\mu }$, form a manifold. On this manifold, any function, $\mathrm{F}:p\to \mathrm{F}\left(p\right)$, is referred to (in this paper) as a ζ-functional, because it takes in a ζ-function $p(·)$ and outputs a scalar. The set of ζ-functionals on $\mathcal{M}$ is denoted $\mathcal{F}\left(\mathcal{M}\right)$. (Note that “ζ-function” and “ζ-functional” are both functions (also called “maps” or “mappings”) in the mathematical sense, with pre-specified domains and ranges. We make the distinction that the ζ-function refers to a real-valued function (e.g., density functions, random variables) defined on the sample space, $\mathcal{X}$, and ζ-functional refers to a mapping from one or more ζ-functions to a real number.) A curve on $\mathcal{M}$ passing through a typical point, p, is nothing but a one-parameter family of ζ-functions, denoted as $p\left(\zeta \right|t)$, with $p\left(\zeta \right|0)=p$. Here, $·|t$ is read as “given t”, “indexed by t”, “parameterized” by t—a one-parameter family of ζ-functions, $p\left(\zeta \right|t)$, is formed as t varies. For each fixed t, $p\left(\zeta \right|t)$ is a function, $\mathcal{X}\times I\to \mathbb{R}$. More generally, $p\left(\zeta \right|\theta )$, where $\theta =[{\theta }^1,\cdots ,{\theta }^n]\subseteq {\mathbb{R}}^n$, is a ζ-function indexed by n parameters, ${\theta }^1,\cdots ,{\theta }^n$. As θ varies, $p\left(\zeta \right|\theta )$ represents a finite dimensional submanifold, ${\tilde{\mathcal{M}}}_{\theta }\subset \mathcal{M}$ where:

$${\tilde{\mathcal{M}}}_{\theta }=\{p\left(\zeta \right|\theta )\in \mathcal{M}:\theta \subseteq {\mathbb{R}}^n\}\subset \mathcal{M}$$

(34)

In this paper, they are referred to as parametric models (and parametric statistical model if $p\left(\zeta \right|\theta )$ is normalized and positive-valued).

In the infinite-dimensional setting, the following tangent vector, v:

$$v\left(\zeta \right)={\left.\frac{\partial p\left(\zeta \right|t)}{\partial t}\right|}_{t=0}$$

(35)

is also a ζ-function. When the tangent vector, v, operates on the ζ-functional $\mathrm{F}\left(p\right)$:

$$d_v\left(\mathrm{F}\left(p\right)\right)=\underset{t\to 0}{lim}\frac{\mathrm{F}\left(p\right(·\left|t\right))-\mathrm{F}(p(·|0\left)\right)}{t}$$

(36)

the outcome is another ζ-functional of both $p\left(\zeta \right)$ and $v\left(\zeta \right)$ and linear in the latter. A particular ζ-functional of interest in this paper is of the following form:

$$\mathrm{F}\left(p\right)={\int }_{\mathcal{X}}f\left(p\left(\zeta \right)\right)d\mu =E_{\mu }\left\{f\left(p(·)\right)\right\}$$

(37)

where $f:\mathbb{R}\to \mathbb{R}$ is a strictly convex function defined on the real line. In this case, $p\left(\zeta \right|t)=p\left(\zeta \right)+v\left(\zeta \right)t+o\left(t^2\right)$, so:

$$d_v\left(\mathrm{F}\left(p\right)\right)={\int }_{\mathcal{X}}{f}^{\prime }\left(p\left(\zeta \right)\right)v\left(\zeta \right)d\mu$$

(38)

which is linear in $v(·)$.

A vector field, as a cross-section of $\mathcal{TM}$, takes $p\left(\zeta \right)$ and associates a ζ-function. We denote a vector field as $u\left(\zeta \right|p)\in \Sigma (\mathcal{M})$, where the variable following the “|” sign indicates that u depends on the point, $p\left(\zeta \right)$, an element of the base manifold, $\mathcal{M}$ (we could also write it as $u\left(p\right(\zeta \left)\right)\left(\zeta \right)$ or $u_p\left(\zeta \right)$). Though the vector fields defined above are not necessarily smooth, we will concentrate on smooth ones below. Of particular interest to us is the vector field, $\rho \left(p\right(\zeta \left)\right)$, for some strictly increasing function, $\rho :\mathbb{R}\to \mathbb{R}$.

Differentiation of smooth vector fields can be defined analogously. The directional derivative, $d_{v}u$, of a vector field, $u\left(\zeta \right|p)$, which is a ζ-function also dependent on $p\left(\zeta \right)$, in the direction of $v=v\left(\zeta \right)$, which is another ζ-function, is:

$$d_{v}u\left(\zeta \right|p)=\underset{t\to 0}{lim}\frac{u\left(\zeta \right|p\left(\zeta \right|t\left)\right)-u\left(\zeta \right|p\left(\zeta \right))}{t}$$

(39)

Note that $d_{v}u$ is another ζ-function; that is why we can write $d_{v}u\left(\zeta \right|p)$ also as $\left(d_{v}u\right)\left(\zeta \right)$. As an example, the derivative of the vector field, $\rho \left(p\right(\zeta \left)\right)$, where $\rho :\mathbb{R}\to \mathbb{R}$, in the direction of $v\left(\zeta \right)$ is:

$$d_v\rho \left(p\left(\zeta \right|t)\right)=\underset{t\to 0}{lim}\frac{\rho \left(p\right(\zeta \left|t\right))-\rho (p)}{t}={\rho }^{\prime }\left(p\left(\zeta \right)\right)v\left(\zeta \right)$$

(40)

With differentiation of vector fields defined, one can define the covariant derivative operation, ${\mathrm{\nabla }}_w$. When operating on a ζ-functional, the covariant derivative is simply the directional derivative (along direction w):

$${\mathrm{\nabla }}_w\mathrm{F}\left(p\right)=d_w\mathrm{F}\left(p\right)$$

(41)

when operating on a vector field, say $u\left(\zeta \right|p)$, ${\mathrm{\nabla }}_w$ is defined as (see [46]):

$${\mathrm{\nabla }}_{w}u=d_{w}u+B(·|w(·|p),u(·|p))$$

(42)

where $B:\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \Sigma \left(\mathcal{M}\right)$ is a ζ-function, which is bilinear in the two tangent vectors (ζ-functions), w and u; it is the infinite-dimensional counterpart of the Christoffel symbol, Γ (for finite dimensions). We denote the conjugate covariant derivative, ${\mathrm{\nabla }}_w^{*}$ (as defined by Equation (22)) in terms of $B^{*}$ (with an asterisk denoting conjugacy):

$$\left({\mathrm{\nabla }}_w^{*}u\right)\left(\zeta \right)=\left(d_{w}u\right)\left(\zeta \right)+B^{*}\left(\zeta \right|w\left(\zeta \right|p\left(\zeta \right)),u\left(\zeta \right|p\left(\zeta \right)))$$

(43)

(here, we write out the explicit dependency on ζ).

The Riemann curvature tensor, R, which measures the curvature of a connection, ∇ (as specified by B), is defined by the map, $\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \Sigma \left(\mathcal{M}\right)$:

$$R(u,v,w)=R(u,v)w={\mathrm{\nabla }}_u{\mathrm{\nabla }}_{v}w-{\mathrm{\nabla }}_v{\mathrm{\nabla }}_{u}w-{\mathrm{\nabla }}_{[u,v]}w$$

(44)

where:

$$[u,v]=d_{u}v-d_{v}u$$

(45)

The torsion tensor, $T:\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \Sigma \left(\mathcal{M}\right),$ is given by:

$$T(u,v)={\mathrm{\nabla }}_{u}v-{\mathrm{\nabla }}_{v}u-[u,v]$$

(46)

2.2. ${\mathcal{D}}^{\left(\alpha \right)}$-Divergence, a Family of Generalized Divergence Functionals

Divergence functionals are defined with respect to a pair of ζ-functions in an infinite-dimensional function space. A divergence functional, $\mathcal{D}:\mathcal{M}\times \mathcal{M}\to {\mathbb{R}}_{+}$, maps two ζ-functions to a non-negative real number. To the extent that ζ-functions can be parameterized by finite-dimensional vectors, $\theta \subseteq {\mathbb{R}}^n$, a divergence functional on $\mathcal{M}\times \mathcal{M}$ will implicitly induce a divergence function on the parameter space, which is a subset of ${\mathbb{R}}^n\times {\mathbb{R}}^n$. In this section, we will discuss the general form of the divergence functional and the associated infinite-dimensional manifold. Finite-dimensional embedding of ζ-functions (i.e., parametric models) will be discussed in Section 3.

2.2.1. Fundamental Convex Inequality and Divergence

We start our exposition by reviewing the notion of a convex function on the real line, $f:\mathbb{R}\to \mathbb{R}$. We recall the fundamental convex inequality that defines a strictly convex function, f:

$$f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\le \frac{1-\alpha }{2}f\left(\gamma \right)+\frac{1+\alpha }{2}f\left(\delta \right)$$

(47)

for all $\gamma ,\delta \in \mathbb{R}$, with equality holding, if and only if $\gamma =\delta$, for all $\alpha \in (-1,1)$. Geometrically, the value of the function, f, at any point, ϵ, in between two end points, γ and δ, lies on or below the chord connecting its values at these two points. This property of a strictly convex function can also be stated in elementary algebra as the Chord Theorem, namely:

$$\frac{f\left(ϵ\right)-f\left(\gamma \right)}{ϵ-\gamma }\le \frac{f\left(\delta \right)-f\left(\gamma \right)}{\delta -\gamma }\le \frac{f\left(\delta \right)-f\left(ϵ\right)}{\delta -ϵ}$$

(48)

where:

$$ϵ=\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta$$

(49)

(here, we assumed $\gamma \le ϵ\le \delta$ without loss of generality). In fact, the slope, $\frac{f\left(\delta \right)-f\left(\gamma \right)}{\delta -\gamma }$, is an increasing function in both δ and γ. The slopes for the chords connecting from the midpoint to either end point are, respectively:

$$L^{\left(\alpha \right)}(\gamma ,\delta )=\frac{f\left(\delta \right)-f\left(ϵ\right)}{\delta -ϵ}=\frac{1}{\delta -\gamma }\frac{2}{1-\alpha }\left(f\left(\delta \right)-f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\right)$$

(50)

$${\widehat{L}}^{\left(\alpha \right)}(\gamma ,\delta )=\frac{f\left(\gamma \right)-f\left(ϵ\right)}{\gamma -ϵ}=\frac{1}{\delta -\gamma }\frac{2}{1+\alpha }\left(f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)-f\left(\gamma \right)\right)$$

(51)

with skew symmetry:

$$L^{(-\alpha )}(\gamma ,\delta )=-{\widehat{L}}^{\left(\alpha \right)}(\delta ,\gamma ),{\widehat{L}}^{(-\alpha )}(\gamma ,\delta )=-L^{\left(\alpha \right)}(\delta ,\gamma )$$

(52)

As $\alpha :-1\to 1$ (i.e., as point ϵ moves from γ to δ, the two fixed ends), both $L^{\left(\alpha \right)}(\gamma ,\delta )$ and ${\widehat{L}}^{\left(\alpha \right)}(\gamma ,\delta )$, are increasing functions of α, but the chord theorem dictates that the latter is always no greater than the former. In fact, their difference has a non-negative value:

$$\begin{array}{ccc}\hfill 0& \le & L^{\left(\alpha \right)}(\gamma ,\delta )-{\widehat{L}}^{\left(\alpha \right)}(\gamma ,\delta )=L^{(-\alpha )}(\delta ,\gamma )-{\widehat{L}}^{(-\alpha )}(\delta ,\gamma )\hfill \\ \hfill & =& \frac{1}{\delta -\gamma }\frac{4}{1-{\alpha }^2}\left(\frac{1-\alpha }{2}f\left(\gamma \right)+\frac{1+\alpha }{2}f\left(\delta \right)-f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\right)\hfill \end{array}$$

(53)

Though the above is obviously valid for $\alpha \in [-1,1]$, it can be shown that it is also valid for any $\alpha \in \mathbb{R}$.

The fundamental convex inequality applies to any two real numbers, $\gamma ,\delta$. We can treat $\gamma ,\delta$ as the values of two functions, $p,q:\mathcal{X}\to \mathbb{R}$, evaluated at any particular sample point, ζ, that is, $\gamma =p\left(\zeta \right)$, $\delta =q\left(\zeta \right)$. This allows us to define the following family of divergence functionals (see [48]).

Proposition 1 Let $f:\mathbb{R}\to \mathbb{R}$ be smooth and strictly convex, and $\rho :\mathbb{R}\to \mathbb{R}$ be strictly increasing. For any two ζ-functions, $p,q$, and any $\alpha \in \mathbb{R}$:

$${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)=\frac{4}{1-{\alpha }^2}{E}_{\mu }\left\{\frac{1-\alpha }{2}f\left(\rho \left(p\right)\right)+\frac{1+\alpha }{2}f\left(\rho \left(q\right)\right)-f\left(\frac{1-\alpha }{2}\rho \left(p\right)+\frac{1+\alpha }{2}\rho \left(q\right)\right)\right\}$$

(54)

is non-negative and equals zero, if and only

$$p\left(\zeta \right)=q\left(\zeta \right)almosteverywhere$$

(55)

Proof. See Section 4.

Proposition 1 constructed a family (parameterized by α) of divergence functionals, ${\mathcal{D}}^{\left(\alpha \right)}$, for two ζ-functions, in which representational duality is embodied as:

$${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)={\mathcal{D}}_{f,\rho }^{(-\alpha )}(q,p)$$

(56)

Its definition involves a strictly increasing function ρ, which can be taken to be the identity function if necessary. The reason ρ is introduced will be clear in the next subsection, where we introduce the notion of conjugate-scaled representations. Furthermore, in order to ensure that the integrals in Equation (54) are well defined, we require $p,q$ to be elements of the set:

$$\{p\left(\zeta \right):E_{\mu }\left\{f\left(\rho \left(p\right)\right)\right\}<\infty \}$$

(57)

${\mathcal{D}}^{\left(\alpha \right)}$-divergence was first introduced in [48]. It generalized the familiar α-divergence Equation (18): take $f\left(p\right)=e^p$ and $\rho \left(p\right)=logp$; then ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)={\mathcal{A}}^{\left(\alpha \right)}(p,q)$. ${\mathcal{D}}^{\left(\alpha \right)}$-divergence became the U-divergence [15] when $f\left(p\right)=U\left(p\right),\rho \left(p\right)={\left(U^{\prime }\right)}^{-1}\left(p\right),\alpha \to 1$ for any strictly convex and strictly increasing $U:{\mathbb{R}}_{+}\to \mathbb{R}$. It was well known that U-divergence, when taking

$$U\left(t\right)=\frac{1}{\beta }{(1+(\beta -1)t)}^{\frac{\beta }{\beta -1}}(\beta \ne 0,1)$$

(58)

specializes to β-divergence [49], defined as:

$${\mathcal{B}}^{\left(\beta \right)}(p,q)=E_{\mu }\left\{p\frac{p^{\beta -1}-q^{\beta -1}}{\beta -1}-\frac{p^{\beta }-q^{\beta }}{\beta }\right\}$$

(59)

and that both α- and β-divergence specialize to the Kullback-Leibler divergence as $\alpha \to \pm 1$ and $\beta \to 1$, respectively.

2.2.2. Conjugate-Scaled Representations of Measurable Functions

In one-dimension, any strictly convex function, $f:\mathbb{R}\to \mathbb{R}$, can be written as an integral of a strictly increasing function, g, and vice versa: $f\left(\delta \right)={\int }_{\gamma }^{\delta }g\left(t\right)dt+f\left(\gamma \right)$, with $g^{\prime }\left(t\right)>0$. The convex (Legendre–Fenchel) conjugate, $f^{*}:\mathbb{R}\to \mathbb{R}$, defined by:

$$f^{*}\left(t\right)=t{\left(f^{\prime }\right)}^{-1}\left(t\right)-f\left({\left(f^{\prime }\right)}^{-1}\left(t\right)\right)$$

(60)

has the integral expression $f^{*}\left(\lambda \right)={\int }_{g\left(\gamma \right)}^{\lambda }{g}^{-1}\left(t\right)dt+f^{*}\left(g\left(\gamma \right)\right)$, with $g^{-1}$ also strictly monotonic and $\gamma ,\delta ,\lambda \in \mathbb{R}$. (Here, the monotonicity condition replaces the requirement of a positive semi-definite Hessian in the case of a convex function of several variables.) The Legendre–Fenchel inequality:

$$f\left(\delta \right)+f^{*}\left(\lambda \right)-\gamma \lambda \ge 0$$

(61)

can be cast as the Young’s inequality:

$${\int }_{\gamma }^{\delta }g\left(t\right)dt+{\int }_{g\left(\gamma \right)}^{\lambda }{g}^{-1}\left(t\right)dt+\gamma g\left(\gamma \right)\ge \delta \lambda$$

(62)

with equality holding, if and only if $\lambda =g\left(\delta \right)$. The conjugate function, $f^{*}$, which is also strictly convex, satisfies ${\left(f^{*}\right)}^{*}=f$ and ${\left(f^{*}\right)}^{\prime }={\left(f^{\prime }\right)}^{-1}$.

We introduce the notion of ρ-representation of a ζ-function $p(·)$ by defining a mapping, $p\mapsto \rho \left(p\right)$, for a strictly increasing function, $\rho :\mathbb{R}\to \mathbb{R}$. We say that a τ-representation of a ζ-function, $p\mapsto \tau \left(p\right)$, is conjugate to the ρ-representation with respect to a smooth and strictly convex function, $f:\mathbb{R}\to \mathbb{R}$, if:

$$\tau \left(p\right)=f^{\prime }\left(\rho \left(p\right)\right)={\left({\left(f^{*}\right)}^{\prime }\right)}^{-1}\left(\rho \left(p\right)\right)⟷\rho \left(p\right)={\left(f^{\prime }\right)}^{-1}\left(\tau \left(p\right)\right)={\left(f^{*}\right)}^{\prime }\left(\tau \left(p\right)\right)$$

(63)

As an example, we may let $\rho \left(p\right)=l^{\left(\alpha \right)}\left(p\right)$ be the α-representation, where $l^{\left(\alpha \right)}$ is given by Equation (13), and the conjugate representation is the $(-\alpha )$-representation $\tau \left(p\right)=l^{(-\alpha )}\left(p\right)$:

$$\rho \left(t\right)=l^{\left(\alpha \right)}\left(t\right)⟷\tau \left(p\right)=l^{(-\alpha )}\left(p\right)$$

(64)

In this case:

$$f\left(t\right)=\frac{2}{1+\alpha }{\left(\left(\frac{1-\alpha }{2}\right)t\right)}^{\frac{2}{1-\alpha }},f^{*}\left(t\right)=\frac{2}{1-\alpha }{\left(\left(\frac{1+\alpha }{2}\right)t\right)}^{\frac{2}{1+\alpha }}$$

(65)

so that:

$$f\left(\rho \left(p\right)\right)=\frac{2}{1+\alpha }p,f^{*}\left(\tau \left(p\right)\right)=\frac{2}{1-\alpha }p$$

(66)

both linear in p. More generally, strictly increasing functions from $\mathbb{R}\to \mathbb{R}$ form a group, with functional composition as group composition operation and the functional inverse as the group inverse operation. That is, (i) for any two strictly increasing functions, ${\rho }_1,{\rho }_2$, their functional composition ${\rho }_2\circ {\rho }_1$ is strictly increasing; (ii) the functional inverse, ${\rho }^{-1}$, of any strictly increasing function, ρ, is also strictly increasing; (iii) there exists a strictly increasing function, ι, the identity function, such that $\rho \circ {\rho }^{-1}={\rho }^{-1}\circ \rho =\iota$. From this perspective, $f^{\prime }=\tau \circ {\rho }^{-1},{\left(f^{*}\right)}^{\prime }=\rho \circ {\tau }^{-1}$, encountered above, are themselves two mutually inverse strictly increasing functions.

If, in the above discussions, $f^{\prime }=\tau \circ {\rho }^{-1}$ is further assumed to be strictly convex, that is:

$$\frac{1-\alpha }{2}\tau \left({\rho }^{-1}\left(\gamma \right)\right)+\frac{1+\alpha }{2}\tau \left({\rho }^{-1}\left(\delta \right)\right)\ge \tau \left({\rho }^{-1}\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\right)$$

(67)

for any $\gamma ,\delta \in \mathbb{R}$ and $\alpha \in (-1,1)$, then by taking ${\tau }^{-1}$ on both sides of the inequality and renaming ${\rho }^{-1}\left(\gamma \right)$ as γ and ${\rho }^{-1}\left(\delta \right)$ as δ, we obtain:

$${\tau }^{-1}\left(\frac{1-\alpha }{2}\tau \left(\gamma \right)+\frac{1+\alpha }{2}\tau \left(\delta \right)\right)\ge {\rho }^{-1}\left(\frac{1-\alpha }{2}\rho \left(\gamma \right)+\frac{1+\alpha }{2}\rho \left(\delta \right)\right)$$

(68)

This is to say:

$$M_{\tau }^{\left(\alpha \right)}(\gamma ,\delta )\ge M_{\rho }^{\left(\alpha \right)}(\gamma ,\delta )$$

(69)

with equality holding, if and only if $\gamma =\delta$, where:

$$M_{\rho }^{\left(\alpha \right)}(\gamma ,\delta )={\rho }^{-1}\left(\frac{1-\alpha }{2}\rho \left(\gamma \right)+\frac{1+\alpha }{2}\rho \left(\delta \right)\right)$$

(70)

is the quasi-linear mean of two numbers $\gamma ,\delta$. Therefore, the following is also a divergence functional (see more discussions in Section 2.4)

$$\frac{4}{1-{\alpha }^2}{\int }_{\mathcal{X}}\left\{{\tau }^{-1}\left(\frac{1-\alpha }{2}\tau \left(p\left(\zeta \right)\right)+\frac{1+\alpha }{2}\tau \left(q\left(\zeta \right)\right)\right)-{\rho }^{-1}\left(\frac{1-\alpha }{2}\rho \left(p\left(\zeta \right)\right)+\frac{1+\alpha }{2}\rho \left(q\left(\zeta \right)\right)\right)\right\}d\mu$$

(71)

2.2.3. Canonical Divergence

The use of a pair of strictly increasing functions, $f,f^{*}$, allow us to define, in parallel with ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)$ given in Equation (54), the conjugate family, ${\mathcal{D}}_{f^{*},\tau }^{\left(\alpha \right)}(p,q)$. The two families turn out to have the same form when $\alpha =\pm 1$; this is the so-called canonical divergence.

Taking the limit, $\alpha \to -1$, the inequality Equation (53) becomes:

$$\frac{f\left(\delta \right)-f\left(\gamma \right)}{\delta -\gamma }-f^{\prime }\left(\gamma \right)\ge 0$$

(72)

where f is strictly convex. A similar inequality is obtained when $\alpha \to 1$. Hence, the divergence functionals, ${\mathcal{D}}_{f,\rho }^{(\pm 1)}(p,q)$, take the form:

$$\begin{array}{ccc}\hfill {\mathcal{D}}_{f,\rho }^{(-1)}(p,q)& =E_{\mu }\{f\left(\rho \left(q\right)\right)-f\left(\rho \left(p\right)\right)-(\rho \left(q\right)-\rho \left(p\right))f^{\prime }\left(\rho \left(p\right)\right)\}\hfill & \left(73\right)\\ & =E_{\mu }\{f^{*}\left(\tau \left(p\right)\right)-f^{*}\left(\tau \left(q\right)\right)-(\tau \left(p\right)-\tau \left(q\right)){\left(f^{*}\right)}^{\prime }\left(\tau \left(q\right)\right)\}={\mathcal{D}}_{f^{*},\tau }^{(-1)}(q,p)\hfill & \left(74\right)\\ \hfill {\mathcal{D}}_{f,\rho }^{\left(1\right)}(p,q)& =E_{\mu }\{f\left(\rho \left(p\right)\right)-f\left(\rho \left(q\right)\right)-(\rho \left(p\right)-\rho \left(q\right))f^{\prime }\left(\rho \left(q\right)\right)\}\hfill & \left(75\right)\\ & =E_{\mu }\{f^{*}\left(\tau \left(q\right)\right)-f^{*}\left(\tau \left(p\right)\right)-(\tau \left(q\right)-\tau \left(p\right)){\left(f^{*}\right)}^{\prime }\left(\tau \left(p\right)\right)\}={\mathcal{D}}_{f^{*},\tau }^{\left(1\right)}(q,p)\hfill & \left(76\right)\end{array}$$

The canonical divergence functional, $\mathcal{A}:\mathcal{M}\times \mathcal{M}\to {\mathbb{R}}_{+}$, is defined (with the aid of a pair of conjugate representations) as:

$${\mathcal{A}}_f(\rho \left(p\right),\tau \left(q\right))=E_{\mu }\{f\left(\rho \left(p\right)\right)+f^{*}\left(\tau \left(q\right)\right)-\rho \left(p\right)\tau \left(q\right)\}={\mathcal{A}}_{f^{*}}(\tau \left(q\right),\rho \left(p\right))$$

(77)

where ${\int }_{\mathcal{X}}f\left(\rho \left(p\right)\right)d\mu$ can be called the (generalized) cumulant generating functional and ${\int }_{\mathcal{X}}{f}^{*}\left(\tau \left(p\right)\right)d\mu$, the (generalized) entropy functional. Thus, a dualistic relation exists between $\alpha =1⟷\alpha =-1$ and between $(f,\rho )⟷(f^{*},\tau )$:

$$\begin{array}{ccc}\hfill {\mathcal{D}}_{f,\rho }^{\left(1\right)}(p,q)& ={\mathcal{D}}_{f,\rho }^{(-1)}(q,p)={\mathcal{D}}_{f^{*},\tau }^{\left(1\right)}(q,p)\hfill & \left(78\right)\\ & ={\mathcal{A}}_f(\rho \left(p\right),\tau \left(q\right))={\mathcal{A}}_{f^{*}}(\tau \left(q\right),\rho \left(p\right))\hfill & \left(79\right)\end{array}$$

We can see that under conjugate $(\pm \alpha )$-representations Equation (64), ${\mathcal{A}}_f$ is simply the α-divergence proper ${\mathcal{A}}^{\left(\alpha \right)}$:

$${\mathcal{A}}_f(\rho \left(p\right),\tau \left(q\right))={\mathcal{A}}^{\left(\alpha \right)}(p,q)$$

(80)

In fact:

$$\frac{1-{\alpha }^2}{4}{\mathcal{A}}^{\left(\alpha \right)}(u,v)=E_{\mu }\left\{\frac{1-\alpha }{2}{u}^{\frac{2}{1-\alpha }}+\frac{1+\alpha }{2}{v}^{\frac{2}{1+\alpha }}-uv\right\}\ge 0$$

(81)

is an expression of Young’s inequality between two functions $u={\left(l^{\left(\alpha \right)}\right)}^{-1}\left(p\right),v={\left(l^{(-\alpha )}\right)}^{-1}\left(q\right)$ under conjugate exponents, $\frac{2}{1-\alpha }$ and $\frac{2}{1+\alpha }$.

2.3. Geometry Induced by the ${\mathcal{D}}^{\left(\alpha \right)}$-Divergence

The last two sections showed that the divergence functional, ${\mathcal{D}}^{\left(\alpha \right)}$, we constructed on $\mathcal{M}$ according to Equation (54) generalizes the α-divergence in a sensible way. Now, we investigate the metric and conjugate connections that such divergence functionals induce; this is accomplished by invoking Eguchi relations Equations (26)–(28).

Proposition 2. At any given $p\in \mathcal{M}$ and for any vector fields, $u,v\in \Sigma \left(\mathcal{M}\right)$:

(i)

the metric tensor field, $g:\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \mathcal{F}\left(\mathcal{M}\right)$, is given by:

$$g(u,v)=E_{\mu }\left\{g\left(p\left(\zeta \right)\right)u\left(\zeta \right|p\left(\zeta \right))v\left(\zeta \right|p\left(\zeta \right))\right\}$$

(82)

where:

$$g\left(t\right)=f^{\prime \prime }\left(\rho \left(t\right)\right){\left({\rho }^{\prime }\left(t\right)\right)}^2$$

(83)

(ii)

the family of covariant derivatives (connections) ${\mathrm{\nabla }}^{\left(\alpha \right)}:\Sigma \left(\mathcal{M}\right)\times \Sigma \left(\mathcal{M}\right)\to \Sigma \left(\mathcal{M}\right)$ is given as:

$${\mathrm{\nabla }}_w^{\left(\alpha \right)}u=\left(d_{w}u\right)\left(\zeta \right)+b^{\left(\alpha \right)}\left(p\left(\zeta \right)\right)u\left(\zeta \right|p\left(\zeta \right))w\left(\zeta \right|p\left(\zeta \right))$$

(84)

where:

$$b^{\left(\alpha \right)}\left(t\right)=\frac{1-\alpha }{2}\frac{f^{\prime \prime \prime }\left(\rho \left(t\right)\right){\rho }^{\prime }\left(t\right)}{f^{\prime \prime }\left(\rho \left(t\right)\right)}+\frac{{\rho }^{\prime \prime }\left(t\right)}{{\rho }^{\prime }\left(t\right)}$$

(85)

(iii)

the family of conjugate covariant derivatives is:

$${\mathrm{\nabla }}_w^{*\left(\alpha \right)}u=\left(d_{w}u\right)\left(\zeta \right)+b^{(-\alpha )}\left(p\left(\zeta \right)\right)u\left(\zeta \right|p\left(\zeta \right))w\left(\zeta \right|p\left(\zeta \right))$$

(86)

Proof. See Section 4.

Note that the $g(·)$ term in Equation (82) and the $b^{\left(\alpha \right)}(·)$ term in covariant derivatives Equation (84) depend on p, the point on the base manifold, where the metric and covariant derivatives are evaluated. They both depend on the auxiliary “scaling functions”, f and ρ. We may cast them into an equivalent, dually symmetric form as follows.

Corollary 3. The $g(·)$ function in expressing the metric Equation (82) and $b^{\left(\alpha \right)}(·)$ in expressing the covariant derivatives Equation (84) can be expressed in dualistic forms:

$$g\left(t\right)={\rho }^{\prime }\left(t\right){\tau }^{\prime }\left(t\right)$$

(87)

and:

$$b^{\left(\alpha \right)}\left(t\right)=\frac{d}{dt}\left(\frac{1+\alpha }{2}log{\rho }^{\prime }\left(t\right)+\frac{1-\alpha }{2}log{\tau }^{\prime }\left(t\right)\right)$$

(88)

Proof. See Section 4.

Corollary 3 makes it immediately evident that the Riemannian metrics induced by ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)$ and by ${\mathcal{D}}_{f^{*},\tau }^{\left(\alpha \right)}(p,q)$ are identical for all α values, while the connections (covariant derivatives) induced by the two families of divergence are conjugate to each other, expressed as $\alpha ⟷-\alpha$. This implies that the conjugacy embodied by the definition of the pair of connections is related to both referential duality and representational duality.

It can be proven that the covariant derivative of the kind of Equation (84) are both curvature-free and torsion-free.

Proposition 4. For the entire family of covariant derivatives indexed by α ($\alpha \in \mathbb{R}$):

(i)

the Riemann curvature tensor $R^{\left(\alpha \right)}(u,v,w)\equiv 0$;

(ii)

the torsion tensor $T^{\left(\alpha \right)}(u,v)\equiv 0$.

Proof. See Section 4.

In other words, the manifold, $\mathcal{M}$, has zero-curvature and zero-torsion for all α. As such, it can serve as an ambient manifold to embed the manifold, ${\mathcal{M}}_{\mu }$, of non-parametric probability density functions and the manifold, ${\mathcal{M}}_{\theta }$, of parametric density functions, and any curvature on ${\mathcal{M}}_{\mu }$ or ${\mathcal{M}}_{\theta }$ may be interpreted as arising from embedding or restriction to a lower dimensional space. See, also, [50] for a discussion of curvatures of statistical manifolds.

2.4. Homogeneous $(\alpha ,\beta )$-Divergence and the Induced Geometry

Suppose that f is, in addition to being strictly convex, strictly increasing. We may set $\rho \left(t\right)=f^{-1}\left(\epsilon t\right)⟷f\left(t\right)=\epsilon {\rho }^{-1}\left(t\right)$, so that the divergence functional becomes:

$${\mathcal{D}}_{\rho }^{\left(\alpha \right)}(p,q)=\frac{4\epsilon }{1-{\alpha }^2}{\int }_{\mathcal{X}}\left\{\frac{1-\alpha }{2}p\left(\zeta \right)+\frac{1+\alpha }{2}q\left(\zeta \right)-{\rho }^{-1}\left(\frac{1-\alpha }{2}\rho \left(p\left(\zeta \right)\right)+\frac{1+\alpha }{2}\rho \left(q\left(\zeta \right)\right)\right)\right\}d\mu$$

(89)

Now, the second term in the integrand is just the quasi-linear mean, $M_{\rho }^{\left(\alpha \right)}$, introduced in Equation (70), where ρ is strictly increasing and concave here. As an example, take $\rho \left(p\right)=logp$, $ϵ=1$; then $M_{\rho }^{\left(\alpha \right)}(p,q)=p^{\frac{1-\alpha }{2}}{q}^{\frac{1+\alpha }{2}}$, and ${\mathcal{D}}_{\rho }^{\left(\alpha \right)}(p,q)$ is the α-divergence Equation (18), while:

$${\mathcal{D}}_{\rho }^{\left(1\right)}(p,q)={\int }_{\mathcal{X}}(p-q-(\rho \left(p\right)-\rho \left(q\right))){\left({\rho }^{-1}\right)}^{\prime }\left(\rho \left(q\right)\right)d\mu ={\mathcal{D}}_{\rho }^{(-1)}(q,p)$$

(90)

is an immediate generalization of the extended Kullback-Leibler divergence in Equation (16).

If we impose a homogeneous requirement ($\kappa \in {\mathbb{R}}^{+}$) on ${\mathcal{D}}_{\rho }^{\left(\alpha \right)}$:

$${\mathcal{D}}_{\rho }^{\left(\alpha \right)}(\kappa p,\kappa q)=\kappa {\mathcal{D}}_{\rho }^{\left(\alpha \right)}(p,q)$$

(91)

then (see [48]) $\rho \left(p\right)=l^{\left(\beta \right)}\left(p\right)$; so Equation (89) becomes a two-parameter family:

$${\mathcal{D}}^{(\alpha ,\beta )}(p,q)\equiv \frac{4}{1-{\alpha }^2}\frac{2}{1+\beta }{E}_{\mu }\left\{\frac{1-\alpha }{2}p+\frac{1+\alpha }{2}q-{\left(\frac{1-\alpha }{2}{p}^{\frac{1-\beta }{2}}+\frac{1+\alpha }{2}{q}^{\frac{1-\beta }{2}}\right)}^{\frac{2}{1-\beta }}\right\}$$

(92)

Here $(\alpha ,\beta )\in [-1,1]\times [-1,1]$, and $\epsilon =2/(1+\beta )$ in Equation (89) is chosen to make ${\mathcal{D}}^{(\alpha ,\beta )}(p,q)$ well defined for $\beta =-1$. We call this family $(\alpha ,\beta )$-divergence; it belongs to the general class of f-divergence studied by [51]. Note that the α parameter encodes referential duality, and the β parameter encodes representational duality. When either$\alpha =\pm 1$ or $\beta =\pm 1$, the one-parameter version of the generic alpha-connection results. The family, ${\mathcal{D}}^{(\alpha ,\beta )}$, is then a generalization of Amari’s α-divergence Equation (18) with:

$$\begin{array}{cccc}\hfill \underset{\alpha \to -1}{lim}{\mathcal{D}}^{(\alpha ,\beta )}(p,q)& =& {\mathcal{A}}^{(-\beta )}(p,q)\hfill & \left(93\right)\\ \hfill \underset{\alpha \to 1}{lim}{\mathcal{D}}^{(\alpha ,\beta )}(p,q)& =& {\mathcal{A}}^{\left(\beta \right)}(p,q)\hfill & \left(94\right)\\ \hfill \underset{\beta \to 1}{lim}{\mathcal{D}}^{(\alpha ,\beta )}(p,q)& =& {\mathcal{A}}^{\left(\alpha \right)}(p,q)\hfill & \left(95\right)\\ \hfill \underset{\beta \to -1}{lim}{\mathcal{D}}^{(\alpha ,\beta )}(p,q)& =& J^{\left(\alpha \right)}(p,q)\hfill & \left(96\right)\end{array}$$

where $J^{\left(\alpha \right)}$ denotes the Jensen difference discussed by [44]:

$$\begin{array}{cc}{J}^{\left(\alpha \right)}(p,q)\equiv & \frac{4}{1-{\alpha }^2}{E}_{\mu }(\frac{1-\alpha }{2}plogp+\frac{1+\alpha }{2}qlogq\hfill \\ & -\left(\frac{1-\alpha }{2}p+\frac{1+\alpha }{2}q\right)log\left(\frac{1-\alpha }{2}p+\frac{1+\alpha }{2}q\right))\hfill \end{array}$$

(97)

$J^{\left(\alpha \right)}$ reduces to Kullback-Leibler divergence Equation (16) when $\alpha \to \pm 1$. Lastly, we note that in ${\mathcal{D}}^{(\alpha ,\beta )}$, when either α or β equals zero, the Levi-Civita connection results.

Note that the divergence given by Equation (92), which first appeared in [48], was called “$(\alpha ,\beta )$-divergence” in [47]. Cichocki et al. [52], following their review of $\alpha ,\beta ,\gamma$ divergence by [53], introduced the following two-parameter family:

$$D_{AB}^{\alpha ,\beta }=-\frac{1}{\alpha \beta }{E}_{\mu }\left\{p^{\alpha }{q}^{\beta }-\frac{\alpha }{\alpha +\beta }{p}^{\alpha +\beta }-\frac{\beta }{\alpha +\beta }{q}^{\alpha +\beta }\right\}$$

(98)

and called it $(\alpha ,\beta )$-divergence. Essentially, it is α-divergence under β- (power) embedding:

$$D_{AB}^{\alpha ,\beta }={(\alpha +\beta )}^2{\mathcal{A}}^{\frac{\beta -\alpha }{\alpha -\beta }}(p^{\alpha +\beta },q^{\alpha +\beta })$$

(99)

Clearly, by taking $f\left(t\right)=e^t,\rho \left(t\right)=(\alpha +\beta )logt$ and renaming $\frac{\beta -\alpha }{\alpha +\beta }$ as α, $D_{AB}^{\alpha ,\beta }$ is a special case of the ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)$, as is ${\mathcal{D}}^{(\alpha ,\beta )}(p,q)$ by Zhang [47,48]. The two definitions of $(\alpha ,\beta )$-divergences, both special cases of ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)$, only differ by a $l^{\left(\beta \right)}$-embedding in the density functions, leading to a superficial difference in the homogeneity/scaling of the divergence function.

With respect to the geometry induced from the $(\alpha ,\beta )$-divergence of Equation (92), we have the following result.

Proposition 5. The metric g and affine connections (covariant derivatives) ${\mathrm{\nabla }}^{(\alpha ,\beta )}$ corresponding to the $(\alpha ,\beta )$-divergence are given by:

$$g(u,v)={\int }_{\mathcal{X}}\frac{1}{p}uvd\mu$$

(100)

$${\mathrm{\nabla }}_u^{(\alpha ,\beta )}v=d_{u}v-\frac{1+\alpha \beta }{2p}uv$$

(101)

$${\mathrm{\nabla }}_u^{*(\alpha ,\beta )}v=d_{u}v-\frac{1-\alpha \beta }{2p}uv$$

(102)

where $u,v\in \Sigma \left(\mathcal{M}\right)$ and $p=p\left(\zeta \right)$ is the point at which g and ∇ are evaluated.

Proof. The proof is immediate upon substituting Equations (64) and (65) to Equations (83) and (85). ⋄

This is to say, with respect to the $(\alpha ,\beta )$-divergence, the product of the two parameters, $\alpha \beta$, acts as the “alpha” parameter in the family of induced connections, so:

$${\mathrm{\nabla }}^{*(\alpha ,\beta )}={\mathrm{\nabla }}^{(-\alpha ,\beta )}={\mathrm{\nabla }}^{(\alpha ,-\beta )}$$

(103)

Setting ${lim}_{\beta \to 1}{\mathrm{\nabla }}^{(\alpha ,\beta )}$ yields Amari’s one-parameter family of α-connections in the infinite-dimensional setting, taking the very simple form:

$${\mathrm{\nabla }}_u^{\left(\alpha \right)}v=d_{u}v-\frac{1+\alpha }{2p}uv$$

(104)

The same is true when ${lim}_{\alpha \to 1}{\mathrm{\nabla }}^{(\alpha ,\beta )}$ (the connections are indexed by β, of course).

3. Parametric Statistical Manifold As Finite-Dimensional Embedding

3.1. Finite-Dimensional Parametric Models

Now, we restrict attention to a finite-dimensional submanifold of measurable functions whose ρ-representation are parameterized using $\theta =[{\theta }^1,\cdots ,{\theta }^n]\subseteq {\mathbb{R}}^n$. In this case, the divergence functional of the two functions, p and q, assumed to be specified, respectively, by ${\theta }_p$ and ${\theta }_q$ in the parametric model, becomes an implicit function of ${\theta }_p,{\theta }_q$. In other words, through introducing parametric models (i.e.,a finite-dimensional submanifold) of the infinite-dimensional manifold of measurable functions, we arrive at a divergence function defined (“pulled back”) over the vector space. We denote the ρ-representation of a parameterized measurable function as $\rho \left(p\right(\zeta \left|\theta \right))$, and the corresponding divergence function by $D({\theta }_p,{\theta }_q)$. It is important to realize that, while $f(·)$ is strictly convex, $\mathcal{F}\left(p\right)={\int }_{\mathcal{X}}f\left(p\left(\zeta \right|\theta )\right)d\mu$ is not at all convex in θ in general.

3.1.1. Riemannian Geometry of Parametric Models

The parametric family of functions, $p\left(\zeta \right|\theta )$, forms a submanifold of $\mathcal{M}$ defined by:

$${\tilde{\mathcal{M}}}_{\theta }=\{p\left(\zeta \right|\theta )\in \mathcal{M}:\theta \subseteq {\mathbb{R}}^n\}$$

(105)

where $p\left(\zeta \right|\theta )$ is a ζ-function indexed by θ, i.e., θ is treated as a parameter to specify a ζ-function. ${\tilde{\mathcal{M}}}_{\theta }$ is a finite-dimensional submanifold of $\mathcal{M}$. We also denote the manifold of a parametric statistical model as:

$${\mathcal{M}}_{\theta }=\{p\left(\zeta \right|\theta )\in {\mathcal{M}}_{\mu }:\theta \subseteq \Theta \subset {\mathbb{R}}^n\}$$

(106)

The θ values themselves, called the natural parameter of the parametric (statistical) model, $p(·|\theta )$, are coordinates for ${\tilde{\mathcal{M}}}_{\theta }$ (or ${\mathcal{M}}_{\theta }$). The tangent vector fields, $u,v,w$, of $\mathcal{M}$ in the directions that are also tangent for ${\tilde{\mathcal{M}}}_{\theta }$ (or ${\mathcal{M}}_{\theta }$) take the form:

$$u=\frac{\partial p\left(\zeta \right|\theta )}{\partial {\theta }^i},v=\frac{\partial p\left(\zeta \right|\theta )}{\partial {\theta }^k},w=\frac{\partial p\left(\zeta \right|\theta )}{\partial {\theta }^j}$$

(107)

The following proposition gives the metric and the family of α-connections in the parametric case. For convenience, we denote $\rho (\zeta ,\theta )\equiv \rho \left(p\right(\zeta \left|\theta \right))$, $\tau (\zeta ,\theta )\equiv \tau \left(p\right(\zeta \left|\theta \right))$ in this subsection.

Proposition 6. For parametric models $p\left(\zeta \right|\theta )$, the metric tensor takes the form:

$$g_{ij}\left(\theta \right)=E_{\mu }\left\{f^{\prime \prime }\left(\rho (\zeta ,\theta )\right)\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^i}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^j}\right\}$$

(108)

and the α-connections take the form:

$${\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)=E_{\mu }\left\{\frac{1-\alpha }{2}{f}^{\prime \prime \prime }\left(\rho (\zeta ,\theta )\right)A_{ijk}+f^{\prime \prime }\left(\rho (\zeta ,\theta )\right)B_{ijk}\right\}$$

(109)

$${\Gamma }_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)=E_{\mu }\left\{\frac{1+\alpha }{2}{f}^{\prime \prime \prime }\left(\rho (\zeta ,\theta )\right)A_{ijk}+f^{\prime \prime }\left(\rho (\zeta ,\theta )\right)B_{ijk}\right\}$$

(110)

where:

$$A_{ijk}(\zeta ,\theta )=\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^i}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^j}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^k},B_{ijk}(\zeta ,\theta )=\frac{{\partial }^2\rho (\zeta ,\theta )}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^k}$$

(111)

Proof. See Section 4.

Note that strict convexity of f requires that $f^{\prime \prime }>0$; thereby the positive-definiteness of $g_{ij}\left(\theta \right)$ is guaranteed. Clearly, the α-connections form conjugate pairs ${\Gamma }_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)={\Gamma }_{ij,k}^{(-\alpha )}\left(\theta \right)$.

As an example, we take the embedding $f\left(t\right)=e^t$ and $\rho \left(p\right)=logp$, with $\tau \left(p\right)=p$, the identity function; then, the expressions in Proposition 6 reduce to the Fisher information and α-connections of the exponential family in Equations (3) and (4).

Corollary 7. In dualistic form, the metric and α-connections are:

$$\begin{array}{ccc}\hfill g_{ij}\left(\theta \right)& =& E_{\mu }\left\{\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^i}\frac{\partial \tau (\zeta ,\theta )}{\partial {\theta }^j}\right\}\hfill \end{array}$$

(112)

$$\begin{array}{ccc}\hfill {\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)& =& E_{\mu }\left\{\frac{1-\alpha }{2}\frac{{\partial }^2\tau (\zeta ,\theta )}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^k}+\frac{1+\alpha }{2}\frac{{\partial }^2\rho (\zeta ,\theta )}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial \tau (\zeta ,\theta )}{\partial {\theta }^k}\right\}\hfill \end{array}$$

(113)

$$\begin{array}{ccc}\hfill {\Gamma }_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)& =& E_{\mu }\left\{\frac{1+\alpha }{2}\frac{{\partial }^2\tau (\zeta ,\theta )}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial \rho (\zeta ,\theta )}{\partial {\theta }^k}+\frac{1-\alpha }{2}\frac{{\partial }^2\rho (\zeta ,\theta )}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial \tau (\zeta ,\theta )}{\partial {\theta }^k}\right\}\hfill \end{array}$$

(114)

Proof. See Section 4.

An immediate consequence of this corollary is as follows. If we construct the divergence function, ${\mathcal{D}}_{f^{*},\tau }^{\left(\alpha \right)}({\theta }_p,{\theta }_q)$, then the induced metric, ${\tilde{g}}_{ij}$, and the induced conjugate connections, ${\tilde{\Gamma }}_{ij,k}^{\left(\alpha \right)},{\tilde{\Gamma }}_{ij,k}^{*\left(\alpha \right)}$, will be related to those induced from ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}({\theta }_p,{\theta }_q)$ (and denoted without the $\tilde{}$ ) via:

$${\tilde{g}}_{ij}\left(\theta \right)=g_{ij}\left(\theta \right)$$

(115)

with:

$${\tilde{\Gamma }}_{ij,k}^{\left(\alpha \right)}\left(\theta \right)={\Gamma }_{ij,k}^{(-\alpha )}\left(\theta \right),{\tilde{\Gamma }}_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)={\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)$$

(116)

So, the difference between using ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}({\theta }_p,{\theta }_q)$ and ${\mathcal{D}}_{f^{*},\tau }^{\left(\alpha \right)}({\theta }_p,{\theta }_q)$ reflects a conjugacy in the ρ- and τ-scalings of $p\left(\zeta \right|\theta )$. Corollary 7 says that the conjugacy in the connection pair $\Gamma ⟷{\Gamma }^{*}$ reflects, in addition to the referential duality ${\theta }_p⟷{\theta }_q$, the representational duality between ρ-scaling and τ-scaling of a ζ-function:

$${\Gamma }_{ij,k}^{*\left(\alpha \right)}\left(\theta \right)={\tilde{\Gamma }}_{ij,k}^{\left(\alpha \right)}\left(\theta \right)$$

(117)

3.1.2. Example: The Parametric $(\alpha ,\beta )$-Manifold

We have introduced the two-parameter family of divergence functionals ${\mathcal{D}}^{(\alpha ,\beta )}(p,q)$ in Section 2.4. Now, pulling back to ${\tilde{\mathcal{M}}}_{\theta }$ (or to ${\mathcal{M}}_{\theta }$), we have the two-parameter family of divergence functions $D^{(\alpha ,\beta )}({\theta }_p,{\theta }_q)$ defined by:

$$D^{(\alpha ,\beta )}({\theta }_p,{\theta }_q)={\mathcal{D}}_{f,\rho }^{(\alpha ,\beta )}(p(·|{\theta }_p),q(·|{\theta }_q))$$

(118)

There are two ways to reduce to Amari’s alpha-divergence (indexed by β here to avoid confusion): (i) take $\alpha =1$ and $\rho \left(p\right)=l^{\left(\beta \right)}\left(p\right)⟷\tau \left(p\right)=l^{(-\beta )}\left(p\right)$; or (ii) take $\alpha =-1$ and $\rho \left(p\right)=l^{(-\beta )}\left(p\right)⟷\tau \left(p\right)=l^{\left(\beta \right)}\left(p\right)$.

Corollary 8. The metric and affine connections for the parametric $(\alpha ,\beta )$-manifold are:

$$g_{ij}\left(\theta \right)=E_p\left\{\frac{\partial logp}{\partial {\theta }^i}\frac{\partial logp}{\partial {\theta }^j}\right\}$$

(119)

$${\Gamma }_{ij,k}^{(\alpha ,\beta )}\left(\theta \right)=E_p\left\{\frac{{\partial }^{2}logp}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial logp}{\partial {\theta }^k}+\frac{1-\alpha \beta }{2}\frac{\partial logp}{\partial {\theta }^i}\frac{\partial logp}{\partial {\theta }^j}\frac{\partial logp}{\partial {\theta }^k}\right\}$$

(120)

$${\Gamma }_{ij,k}^{*(\alpha ,\beta )}\left(\theta \right)=E_p\left\{\frac{{\partial }^{2}logp}{\partial {\theta }^i\partial {\theta }^j}\frac{\partial logp}{\partial {\theta }^k}+\frac{1+\alpha \beta }{2}\frac{\partial logp}{\partial {\theta }^i}\frac{\partial logp}{\partial {\theta }^j}\frac{\partial logp}{\partial {\theta }^k}\right\}$$

(121)

Proof. Direct substituion of expressions of $\rho \left(p\right)$ and $\tau \left(p\right)$. ⋄

This two-parameter family of affine connections, ${\Gamma }_{ij,k}^{(\alpha ,\beta )}\left(\theta \right)$, indexed now by the numerical product, $\alpha \beta \in [-1,1]$, is actually the alpha-connection proper (i.e., the one-parameter family of its generic form; see [29])

$${\Gamma }_{ij,k}^{(\alpha ,\beta )}\left(\theta \right)={\Gamma }_{ij,k}^{(-\alpha ,-\beta )}\left(\theta \right)$$

(122)

with biduality compactly expressed as

$${\Gamma }_{ij,k}^{*(\alpha ,\beta )}\left(\theta \right)={\Gamma }_{ij,k}^{(-\alpha ,\beta )}\left(\theta \right)={\Gamma }_{ij,k}^{(\alpha ,-\beta )}\left(\theta \right)$$

(123)

3.2. Affine Embedded Submanifold

We now define the notion of ρ-affinity. A parametric model, $p\left(\zeta \right|\theta )$, is said to be ρ-affine if its ρ-representation can be embedded into a finite-dimensional affine space, i.e., if there exists a set of linearly independent functions ${\lambda }_i\left(\zeta \right)$ over the same support, $\mathcal{X}\ni \zeta$, such that:

$$\rho \left(p\left(\zeta \right|\theta )\right)=\sum _i{\theta }^i{\lambda }_i\left(\zeta \right)$$

(124)

As noted in Section 3.1.1, the parameter $\theta =[{\theta }^1,\cdots ,{\theta }^n]\in \Theta$ is its natural parameter.

For any measurable function, $p\left(\zeta \right)$, the projection of its τ-representation onto the functions ${\lambda }_i\left(\zeta \right)$

$${\eta }_i={\int }_{\mathcal{X}}\tau \left(p\left(\zeta \right)\right){\lambda }_i\left(\zeta \right)d\mu$$

(125)

forms a vector $\eta =[{\eta }_1,\cdots ,{\eta }_n]\subseteq {\mathbb{R}}^n$. We call η the expectation parameter of $p\left(\zeta \right)$, and the functions $\lambda \left(\zeta \right)=[{\lambda }_1\left(\zeta \right),\cdots ,{\lambda }_n\left(\zeta \right)]$ the affine basis functions.

The above notion of ρ-affinity is a generalization of α-affine manifolds [1,2], where ρ- and τ-representations are just α- and $(-\alpha )$-representations, respectively. Note that elements of the ρ-affine manifold may not be a probability model; rather, after denormalization, probability models can become ρ-affine. The issue of normalization will be discussed in Section 5.

3.2.1. Biorthogonality of Natural and Expectation Parameters

Proposition 9. When a parametric model is ρ-affine,

(i)

the function:

$$\mathrm{\Phi }\left(\theta \right)={\int }_{\mathcal{X}}f\left(\rho \left(p\left(\zeta \right|\theta )\right)\right)d\mu$$

(126)

is strictly convex;

(ii)

the divergence functional, ${\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)$, takes the form of the divergence function:

$$D_{\mathrm{\Phi }}^{\left(\alpha \right)}({\theta }_p,{\theta }_q)=\frac{4}{1-{\alpha }^2}\left(\frac{1-\alpha }{2}\mathrm{\Phi }\left({\theta }_p\right)+\frac{1+\alpha }{2}\mathrm{\Phi }\left({\theta }_q\right)-\mathrm{\Phi }\left(\frac{1-\alpha }{2}{\theta }_p+\frac{1+\alpha }{2}{\theta }_q\right)\right)$$

(127)

(iii)

the metric tensor, affine connections and the Riemann curvature tensor take the forms:

$$g_{ij}\left(\theta \right)={\mathrm{\Phi }}_{ij};{\Gamma }_{ij,k}^{\left(\alpha \right)}\left(\theta \right)=\frac{1-\alpha }{2}{\mathrm{\Phi }}_{ijk}={\Gamma }_{ij,k}^{*(-\alpha )}\left(\theta \right)$$

(128)

$$R_{ij\mu \nu }^{\left(\alpha \right)}\left(\theta \right)=\frac{1-{\alpha }^2}{4}\sum _{l,k}({\mathrm{\Phi }}_{il\nu }{\mathrm{\Phi }}_{jk\mu }-{\mathrm{\Phi }}_{il\mu }{\mathrm{\Phi }}_{jk\nu }){\mathrm{\Phi }}^{lk}=R_{ij\mu \nu }^{*\left(\alpha \right)}\left(\theta \right)$$

(129)

Here, ${\mathrm{\Phi }}_{ij}$, ${\mathrm{\Phi }}_{ijk}$ denote, respectively, second and third partial derivatives of $\mathrm{\Phi }\left(\theta \right)$:

$${\mathrm{\Phi }}_{ij}=\frac{{\partial }^2\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j},{\mathrm{\Phi }}_{ijk}=\frac{{\partial }^3\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j\partial {\theta }^k}$$

(130)

and ${\mathrm{\Phi }}^{ij}$ is the matrix inverse of ${\mathrm{\Phi }}_{ij}$.

Proof. See Section 4.

Recall that, for a convex function of several variables, $\mathrm{\Phi }:{\mathbb{R}}^n\to \mathbb{R}$, its convex conjugate ${\mathrm{\Phi }}^{*}$ is defined through the Legendre–Fenchel transform:

$${\mathrm{\Phi }}^{*}\left(\eta \right)=\langle \eta ,{(\partial \mathrm{\Phi })}^{-1}\left(\eta \right)\rangle -\mathrm{\Phi }>({(\partial \mathrm{\Phi })}^{-1}\left(\eta \right))$$

(131)

where $\partial \mathrm{\Phi }$ stands for the gradient (sub-differential) of Φ, and $\langle ,\rangle$ denotes the standard inner product. It can be shown that the function, ${\mathrm{\Phi }}^{*}$, is also convex and has Φ as its conjugate ${\left({\mathrm{\Phi }}^{*}\right)}^{*}=\mathrm{\Phi }$. The Hessian (second derivatives) of a strictly convex function (Φ and ${\mathrm{\Phi }}^{*}$) is positive-definite. The Legendre–Fenchel inequality Equation (131) can be expressed using dual variables, $\theta ,\eta$, as:

$$\mathrm{\Phi }\left(\theta \right)+{\mathrm{\Phi }}^{*}\left(\eta \right)-\sum _i{\eta }_i{\theta }^i\ge 0$$

(132)

where equality holds, if and only if:

$$\theta =\left(\partial {\mathrm{\Phi }}^{*}\right)\left(\eta \right)={(\partial \mathrm{\Phi })}^{-1}\left(\eta \right)⟷\eta =\partial \mathrm{\Phi }\left(\theta \right)={\left(\partial {\mathrm{\Phi }}^{*}\right)}^{-1}\left(\theta \right)$$

(133)

Corollary 10. For a ρ-affine manifold:

(i)

define

$$\tilde{\mathrm{\Phi }}\left(\theta \right)={\int }_{\mathcal{X}}{f}^{*}\left(\tau \left(p\left(\zeta \right|\theta )\right)\right)d\mu$$

(134)

then ${\mathrm{\Phi }}^{*}\left(\eta \right)\equiv \tilde{\mathrm{\Phi }}\left({(\partial \mathrm{\Phi })}^{-1}\left(\eta \right)\right)$ is the convex (Legendre–Fenchel) conjugate of $\mathrm{\Phi }\left(\theta \right)$;

(ii)

the pair of convex functions, $\mathrm{\Phi },{\mathrm{\Phi }}^{*}$, form a pair of “potentials” to induce $\eta ,\theta$:

$$\frac{\partial \mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i}={\eta }_i⟷\frac{\partial {\mathrm{\Phi }}^{*}\left(\eta \right)}{\partial {\eta }_i}={\theta }^i$$

(135)

(iii)

the expectation parameter, $\eta \in \Xi$, and the natural parameter, $\theta \in \Theta$, form biorthogonal coordinates:

$$\frac{\partial {\eta }_i}{\partial {\theta }^j}=g_{ij}\left(\theta \right)⟷\frac{\partial {\theta }^i}{\partial {\eta }_j}={\tilde{g}}^{ij}\left(\eta \right)$$

(136)

where ${\tilde{g}}^{ij}\left(\eta \right)$ is the matrix inverse of $g_{ij}\left(\theta \right)$, the metric tensor of the parametric (statistical) manifold.

Proof. See Section 4.

Note that while the function, $\mathrm{\Phi }\left(\theta \right)$, can be viewed as the generalized cumulant generating function (or partition function), the function, ${\mathrm{\Phi }}^{*}\left(\eta \right)$, is the generalized entropy function. For an exponential family, the two are well known to form one-to-one correspondence; either can be used on that index as a density function of the exponential family.

3.2.2. Dually Flat Affine Manifolds

When $\alpha =\pm 1$, part (iii) of Proposition 9 dictates that all components of the curvature tensor vanish, i.e., $R_{ij\mu \nu }^{(\pm 1)}\left(\theta \right)=0$. In this case, there exists a coordinate system under which either ${\Gamma }_{ij,k}^{*(-1)}\left(\theta \right)=0$ or ${\Gamma }_{ij,k}^{\left(1\right)}\left(\theta \right)=0$. This is the well-studied “dually flat” parametric statistical manifold [1,2,20], under which divergence functions have a unique, canonical form.

Corollary 11. When $\alpha \to \pm 1$, $D_{\mathrm{\Phi }}^{\left(\alpha \right)}$ reduces to the Bregman divergence Equation (17):

$$D_{\mathrm{\Phi }}^{(-1)}({\theta }_p,{\theta }_q)=D_{\mathrm{\Phi }}^{\left(1\right)}({\theta }_q,{\theta }_p)=\mathrm{\Phi }\left({\theta }_q\right)-\mathrm{\Phi }\left({\theta }_p\right)-\langle {\theta }_q-{\theta }_p,\partial \mathrm{\Phi }\left({\theta }_p\right)\rangle =B_{\mathrm{\Phi }}({\theta }_q,{\theta }_p)$$

(137)

$$D_{\mathrm{\Phi }}^{\left(1\right)}({\theta }_p,{\theta }_q)=D_{\mathrm{\Phi }}^{(-1)}({\theta }_q,{\theta }_p)=\mathrm{\Phi }\left({\theta }_p\right)-\mathrm{\Phi }\left({\theta }_q\right)-\langle {\theta }_p-{\theta }_q,\partial \mathrm{\Phi }\left({\theta }_q\right)\rangle =B_{\mathrm{\Phi }}({\theta }_p,{\theta }_q)$$

(138)

or equivalently, to the canonical divergence functions:

$$D_{\mathrm{\Phi }}^{\left(1\right)}({\theta }_p,{(\partial \mathrm{\Phi })}^{-1}\left({\eta }_q\right))=\mathrm{\Phi }\left({\theta }_p\right)+{\mathrm{\Phi }}^{*}\left({\eta }_q\right)-\langle {\theta }_p,{\eta }_q\rangle \equiv A_{\mathrm{\Phi }}({\theta }_p,{\eta }_q)$$

(139)

$$D_{\mathrm{\Phi }}^{(-1)}({(\partial \mathrm{\Phi })}^{-1}\left({\theta }_p\right),{\theta }_q)=\mathrm{\Phi }\left({\theta }_q\right)+{\mathrm{\Phi }}^{*}\left({\eta }_p\right)-\langle {\eta }_p,{\theta }_q\rangle \equiv A_{{\mathrm{\Phi }}^{*}}({\eta }_p,{\theta }_q)$$

(140)

Proof. Immediate by substitution using the definition Equation (131). ⋄

The canonical divergence, $A_{\mathrm{\Phi }}({\theta }_p,{\eta }_q)$, based on the Legendre–Fenchel inequality was introduced by [2,20], where the functions, $\mathrm{\Phi },{\mathrm{\Phi }}^{*}$, the cumulant generating functions of an exponential family, were referred to as dual “potential” functions. This form Equation (139) is “canonical”, because it is uniquely specified in a dually flat manifold using a pair of biorthogonal coordinates.

We point out that there are two kinds of duality associated with the divergence defined on dually flat statistical manifold, one between $D_{\mathrm{\Phi }}^{(-1)}⟷D_{\mathrm{\Phi }}^{\left(1\right)}$ and between $D_{{\mathrm{\Phi }}^{*}}^{(-1)}⟷D_{{\mathrm{\Phi }}^{*}}^{\left(1\right)}$; the other between $D_{\mathrm{\Phi }}^{(-1)}⟷D_{{\mathrm{\Phi }}^{*}}^{(-1)}$ and between $D_{\mathrm{\Phi }}^{\left(1\right)}⟷D_{{\mathrm{\Phi }}^{*}}^{\left(1\right)}$. The first kind is related to the duality in the choice of the reference and the comparison status for the two points (θ versus η) for computing the value of the divergence and, hence, called “referential duality”. The second kind is related to the duality in the choice of the representation of the point as a vector in the parameter versus gradient space (θ versus η) in the expression of the divergence function and, hence, called “representational duality”. More concretely:

$$D_{\mathrm{\Phi }}^{(-1)}({\theta }_p,{\theta }_q)=D_{{\mathrm{\Phi }}^{*}}^{(-1)}(\partial \mathrm{\Phi }\left({\theta }_q\right),\partial \mathrm{\Phi }\left({\theta }_p\right))=D_{{\mathrm{\Phi }}^{*}}^{\left(1\right)}(\partial \mathrm{\Phi }\left({\theta }_p\right),\partial \mathrm{\Phi }\left({\theta }_q\right))=D_{\mathrm{\Phi }}^{\left(1\right)}({\theta }_q,{\theta }_p)$$

(141)

The biduality is compactly reflected in the canonical divergence as:

$$A_{\mathrm{\Phi }}({\theta }_p,{\eta }_q)=A_{{\mathrm{\Phi }}^{*}}({\eta }_q,{\theta }_p)$$

(142)

4. Proofs

Proof of Eguchi Relation Equations (26)–(28). Assume the divergence function, $\mathcal{D}$, is Fréchet differentiable up to third order. For any two points, $p,q\in \mathcal{M}$, and two vector fields, $u,v\in \Sigma \left(\mathcal{M}\right)$, let us denote $G(p,q)(u,v)=-{\left(d_u\right)}_p{\left(d_v\right)}_q\mathcal{D}(p,q)$, the mixed second derivative, which is bilinear in $u,v$. Then, ${\langle u,v\rangle }_p=G(p,p)(u,v)$. Suppressing the dependency on $(u,v)$ in G, we take directional derivative with respect to $w\in \Sigma \left(\mathcal{M}\right)$:

$$d_{w}G(p,q)={\left(d_w\right)}_{p}G(p,q)+{\left(d_w\right)}_{q}G(p,q)$$

(143)

and then evaluating at $q=p$ to obtain:

$$d_w\langle u,v\rangle =\langle {\mathrm{\nabla }}_{w}u,v\rangle +\langle u,{\mathrm{\nabla }}_w^{*}v\rangle$$

(144)

which is the defining Equation (22) for conjugate connections. Therefore, what remains to be checked is whether ∇ as defined by:

$$\langle {\mathrm{\nabla }}_{w}u,v\rangle ={\left.-{\left(d_w\right)}_p{\left(d_u\right)}_p\tilde{G}(p,q)\left(v\right)\right|}_{p=q}$$

(145)

indeed transforms as an affine connection (and similarly for ${\mathrm{\nabla }}^{*}$), where $\tilde{G}(p,q)\left(v\right)=-{\left(d_v\right)}_q\mathcal{D}(p,q)$ is linear in v. It is easy to verify that ${\mathrm{\nabla }}_w(u_1+u_2)={\mathrm{\nabla }}_w{u}_1+{\mathrm{\nabla }}_w{u}_2$, ${\mathrm{\nabla }}_{w_1+w_2}u={\mathrm{\nabla }}_{w_1}u+{\mathrm{\nabla }}_{w_2}u$, and ${\mathrm{\nabla }}_{fw}u=f{\mathrm{\nabla }}_{w}u$ for $f\in \mathcal{F}$. We need only to prove:

$$\langle {\mathrm{\nabla }}_w\left(fu\right),v\rangle =f\langle {\mathrm{\nabla }}_{w}u,v\rangle +\langle u,v\rangle d_{w}f$$

(146)

which is immediately obtained by:

$${\left(d_w\right)}_p{\left(f{d}_u\right)}_p\tilde{G}(p,q)=f·\left({\left(d_w\right)}_p{\left(d_u\right)}_p\tilde{G}(p,q)\right)+{\left(d_u\right)}_p\left(\tilde{G}(p,q)\right){\left(d_{w}f\right)}_p$$

(147)

⋄

Proof of Proposition 1. We only need to prove that for a strictly convex function, $f:\mathbb{R}\to \mathbb{R}$ and $\alpha \in \mathbb{R}$, the following quantity:

$$d_f^{\left(\alpha \right)}(\gamma ,\delta )=\frac{4}{1-{\alpha }^2}\left(\frac{1-\alpha }{2}f\left(\gamma \right)+\frac{1+\alpha }{2}f\left(\delta \right)-f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\right)$$

(148)

is non-negative for all real numbers, $\gamma ,\delta \in \mathbb{R}$, with $d_f^{\left(\alpha \right)}(\gamma ,\delta )=0$, if and only if $\gamma =\delta$.

Clearly, for any $\alpha \in (-1,1)$, $1-{\alpha }^2>0$; so, from the fundamental convex inequality Equation (46), the functions $d_f^{\left(\alpha \right)}(\gamma ,\delta )\ge 0$ for all $\gamma ,\delta \in \mathbb{R}$, with equality holding, if and only if $\gamma =\delta$. When $\alpha >1$, we rewrite $\delta =\frac{2}{\alpha +1}\lambda +\frac{\alpha -1}{\alpha +1}\gamma$ as a convex mixture of λ and γ (i.e., $\frac{2}{\alpha +1}=\frac{1-{\alpha }^{\prime }}{2}$, $\frac{\alpha -1}{\alpha +1}=\frac{1+{\alpha }^{\prime }}{2}$ with ${\alpha }^{\prime }\in (-1,1)$). Strict convexity of f guarantees:

$$\frac{2}{\alpha +1}f\left(\lambda \right)+\frac{\alpha -1}{\alpha +1}f\left(\gamma \right)\ge f\left(\delta \right)$$

(149)

or moving left-hand side to right-hand side:

$$\frac{2}{1+\alpha }\left(\frac{1-\alpha }{2}f\left(\gamma \right)+\frac{1+\alpha }{2}f\left(\delta \right)-f\left(\frac{1-\alpha }{2}\gamma +\frac{1+\alpha }{2}\delta \right)\right)\le 0$$

(150)

This, along with $1-{\alpha }^2<0$ proves the non-negativity of $d_f^{\left(\alpha \right)}(\gamma ,\delta )\ge 0$ for $\alpha >1$, with equality holding, if and only if $\lambda =\gamma$, i.e., $\gamma =\delta$. The case of $\alpha <-1$ is similarly proven by applying Equation (47) to the three points, $\gamma ,\lambda$, and their convex mixture $\delta =\frac{2}{1-\alpha }\lambda +\frac{-1-\alpha }{1-\alpha }\gamma$. Finally, the continuity of $d_f^{\left(\alpha \right)}(\gamma ,\delta )$ with respect to α guarantees that the above claim is also valid in the case of $\alpha =\pm 1$. ⋄

Proof of Proposition 2. With respect to Equation (54), note that ${\left(d_u\right)}_p$ means that the functional derivative is with respect to p only (point q is treated as fixed):

$${\left(d_u\right)}_p{\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)=\frac{2}{1+\alpha }{\int }_{\mathcal{X}}\left\{f^{\prime }\left(\rho \left(p\right)\right)-f^{\prime }\left(\frac{1-\alpha }{2}\rho \left(p\right)+\frac{1+\alpha }{2}\rho \left(q\right)\right)\right\}{\rho }^{\prime }\left(p\right)ud\mu$$

(151)

Applying functional derivative ${\left(d_v\right)}_q$, now with respect to q only, to the above equation yields:

$${\left(d_v\right)}_q\left({\left(d_u\right)}_p{\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)\right)=-{\int }_{\mathcal{X}}{f}^{\prime \prime }\left(\frac{1-\alpha }{2}\rho \left(p\right)+\frac{1+\alpha }{2}\rho \left(q\right)\right){\rho }^{\prime }\left(p\right){\rho }^{\prime }\left(q\right)uvd\mu$$

(152)

Setting $p=q$ and invoking Equation (26) yields Equation (82) with Equation (83).

Next, applying ${\left(d_w\right)}_p$ to Equation (152), and realizing that $u,v$ are both vector fields:

$${\left(d_w\right)}_p\left({\left(d_v\right)}_q{\left(d_u\right)}_p{\mathcal{D}}_{f,\rho }^{\left(\alpha \right)}(p,q)\right)=$$

(153)

$$-{\int }_{\mathcal{X}}\frac{1-\alpha }{2}\left(f^{\prime \prime \prime }\left(\frac{1-\alpha }{2}\rho \left(p\right)+\frac{1+\alpha }{2}\rho \left(q\right)\right){\left({\rho }^{\prime }\left(p\right)\right)}^2{\rho }^{\prime }\left(q\right)uvwd\mu \right.$$

(154)

$$-{\int }_{\mathcal{X}}{f}^{\prime \prime }\left(\frac{1-\alpha }{2}\rho \left(p\right)+\frac{1+\alpha }{2}\rho \left(q\right)\right){\rho }^{\prime }\left(q\right)v\left({\rho }^{\prime \prime }\left(p\right)uw+{\rho }^{\prime }\left(p\right)\left(d_{w}u\right)\right)d\mu$$

(155)

Setting $p=q$, invoking Equation (27) and:

$$g({\mathrm{\nabla }}_{w}u,v)=\int f^{\prime \prime }\left(\rho \left(p\right)\right){\left({\rho }^{\prime }\left(p\right)\right)}^2{\mathrm{\nabla }}_{w}uv\left(\zeta \right|p)d\mu$$

(156)

and realizing that $v\left(\zeta \right|p)$ can be arbitrary, we have:

$$f^{\prime \prime }\left(\rho \right){\left({\rho }^{\prime }\right)}^2{\mathrm{\nabla }}_w^{\left(\alpha \right)}u=\frac{1-\alpha }{2}{f}^{\prime \prime \prime }\left(\rho \right){\left({\rho }^{\prime }\right)}^{3}uw+f^{\prime \prime }\left(\rho \right){\rho }^{\prime }({\rho }^{\prime \prime }uw+{\rho }^{\prime }\left(d_{w}u\right))$$

(157)

where we have short-handed ρ for $\rho \left(p\right(\zeta \left)\right)$. Remember that ${\mathrm{\nabla }}_w^{\left(\alpha \right)}u$ is a ζ-function; the above equation yields:

$${\mathrm{\nabla }}_w^{\left(\alpha \right)}u=d_{w}u+\frac{1-\alpha }{2}\frac{f^{\prime \prime \prime }\left(\rho \right)}{f^{\prime \prime }\left(\rho \right)}{\rho }^{\prime }uw+\frac{{\rho }^{\prime \prime }}{{\rho }^{\prime }}uw=d_{w}u+\left(\frac{1-\alpha }{2}\frac{f^{\prime \prime \prime }\left(\rho \right)}{f^{\prime \prime }\left(\rho \right)}{\rho }^{\prime }+\frac{{\rho }^{\prime \prime }}{{\rho }^{\prime }}\right)uw$$

(158)

Thus, we obtain Equation (84) with Equation (85). The expression for ${\mathrm{\nabla }}^{*\left(\alpha \right)}$ is obtained analogously. ⋄

Proof of Corollary 3. From the identities:

$$f^{\prime \prime }\left(\rho \right)=\frac{{\tau }^{\prime }}{{\rho }^{\prime }},f^{\prime \prime \prime }\left(\rho \right)=\frac{{\rho }^{\prime }{\tau }^{\prime \prime }-{\rho }^{\prime \prime }{\tau }^{\prime }}{{\left({\rho }^{\prime }\right)}^3}$$

(159)

we obtain Equations (87) and (88) after substitution. ⋄

Proof of Proposition 4. We first derive a general formula for the Riemann curvature tensor for the infinite-dimensional manifold, since that given by a popular text book ([46], p.226) appears to miss some terms. From Equation (42):

$$d_u\left({\mathrm{\nabla }}_{v}w\right)=d_u\left(d_{v}w\right)+B(d_{u}v,w)+B(v,d_{u}w)+\left(d_{u}B\right)(v,w)$$

(160)

so that:

$$\begin{array}{ccc}\hfill {\mathrm{\nabla }}_u\left({\mathrm{\nabla }}_{v}w\right)& =& d_u\left({\mathrm{\nabla }}_{v}w\right)+B(u,{\mathrm{\nabla }}_{v}w)\hfill \\ & =& (d_u\left(d_{v}w\right)+B(d_{u}v,w)+B(v,d_{u}w)+\left(d_{u}B\right)(v,w))+(B(u,d_{v}w)+B(u,B(v,w)))\hfill \end{array}$$

here $d_{u}B=B^{\prime }u$ refers to the derivative on the B-form itself and not on its $v,w$ arguments. The expression for ${\mathrm{\nabla }}_v\left({\mathrm{\nabla }}_{u}w\right)$ simply exchanges $u\to v$ in the above. Now:

$${\mathrm{\nabla }}_{[u,v]}w=d_{[u,v]}w+B([u,v],w)$$

(161)

where $[u,v]=d_{u}v-d_{v}u$ is a vector field, such that:

$$d_{[u,v]}w=d_u\left(d_{v}w\right)-d_v\left(d_{u}w\right)$$

(162)

Substituting them into Equation (44), we get a general expression of the Riemann curvature tensor in infinite-dimensional setting:

$$R(u,v,w)=B(u,B(v,w))-B(v,B(u,w))+\left(d_{u}B\right)(v,w)-\left(d_{v}B\right)(u,w)$$

(163)

The expression for $T(u,v)$ in Equation (46) becomes:

$$T(u,v)=B(u,v)-B(v,u)$$

(164)

In the current case, B evaluated at $p\left(\zeta \right)$ is the bilinear form:

$$B(u,v)=b^{\left(\alpha \right)}\left(p\left(\zeta \right)\right)u\left(\zeta \right|p)v\left(\zeta \right|p)$$

(165)

Substituting this into the above, and realizing that $\left(d_{u}B\right)(v,w)$ is simply $\left(b^{\left(\alpha \right)}\right)\prime uvw$, we immediately have $R^{\left(\alpha \right)}(u,v,w)=0$, as well as $T^{\left(\alpha \right)}(u,v)=0$. ⋄

Proof of Proposition 6. Given Equation (107) as the tangent vector fields for parametric models with holonomic coordinates θ, we note that:

$$\begin{array}{ccc}\hfill d_u\rho & =& {\rho }^{\prime }\left(p\right)u={\rho }^{\prime }\left(p\right)\frac{\partial p}{\partial {\theta }^i}=\frac{\partial \rho \left(p\right)}{\partial {\theta }^i}\hfill \end{array}$$

(166)

$$\begin{array}{ccc}\hfill d_w\rho & =& {\rho }^{\prime }\left(p\right)w={\rho }^{\prime }\left(p\right)\frac{\partial p}{\partial {\theta }^j}=\frac{\partial \rho \left(p\right)}{\partial {\theta }^j}\hfill \end{array}$$

(167)

so Equation (108) follows. Next, from:

$$d_{w}u=\frac{{\partial }^{2}p}{\partial {\theta }^i\partial {\theta }^j}$$

(168)

we have:

$$\begin{array}{cc}{\rho }^{\prime \prime }\left(p\right)uw+{\rho }^{\prime }\left(p\right)\left(d_{w}u\right)& ={\rho }^{\prime \prime }\left(p\right)\frac{\partial p}{\partial {\theta }^i}\frac{\partial p}{\partial {\theta }^j}+{\rho }^{\prime }\left(p\right)\frac{{\partial }^{2}p}{\partial {\theta }^i\partial {\theta }^j}=\frac{\partial }{\partial {\theta }^i}\left({\rho }^{\prime }\left(p\right)\frac{\partial p}{\partial {\theta }^j}\right)\hfill \\ & =\frac{\partial }{\partial {\theta }^i}\left(\frac{\partial \rho }{\partial {\theta }^j}\right)=\frac{{\partial }^2\rho }{\partial {\theta }^i\partial {\theta }^j}\hfill \end{array}$$

(169)

Observing ${\Gamma }_{ij,k}=\langle {\mathrm{\nabla }}_{w}u,v\rangle$, expression (109) results after substituting the above derived expressions into Equation (84) with Equation (85). ⋄

Proof of Corollary 7. Applying Equations (166) and (167) to Equation (82) with Equation (87) immediately yields Equation (112). Next, from Corollary 3:

$$b^{\left(\alpha \right)}\left(t\right)=\frac{1-\alpha }{2}\frac{{\tau }^{\prime \prime }\left(t\right)}{{\tau }^{\prime }\left(t\right)}+\frac{1+\alpha }{2}\frac{{\rho }^{\prime \prime }\left(t\right)}{{\rho }^{\prime }\left(t\right)}$$

(170)

It follows that:

$${\Gamma }_{ij,k}^{\left(\alpha \right)}=\langle {\mathrm{\nabla }}_w^{\left(\alpha \right)}u,v\rangle =\left(\frac{1-\alpha }{2}\left({\rho }^{\prime }{\tau }^{\prime \prime }uw+{\rho }^{\prime }{\tau }^{\prime }{d}_{w}u\right)+\frac{1+\alpha }{2}\left({\rho }^{\prime \prime }{\tau }^{\prime }uw+{\rho }^{\prime }{\tau }^{\prime }{d}_{w}u\right)\right)v$$

(171)

$$={\rho }^{\prime }v\frac{1-\alpha }{2}{d}_w\left(d_u\tau \right)+{\tau }^{\prime }v\frac{1+\alpha }{2}{d}_w\left(d_u\rho \right)=\frac{1-\alpha }{2}\left(d_v\rho \right)d_w\left(d_u\tau \right)+\frac{1+\alpha }{2}\left(d_v\tau \right)d_w\left(d_u\rho \right)$$

(172)

Note that given holonomic coordinates Equation (107):

$$d_w\left(d_u\rho \right)=\frac{\partial }{\partial {\theta }^j}\left(\frac{\partial \rho \left(p\right)}{\partial {\theta }^i}\right)=\frac{{\partial }^2\rho \left(p\right)}{\partial {\theta }^i\partial {\theta }^j}$$

(173)

Substituting into Equation (84) with Equation (88) yields Equations (113) and (114). ⋄

Proof of Proposition 9. The assumption Equation (124) implies that $\frac{\partial \rho }{\partial {\theta }^i}={\lambda }_i\left(\zeta \right)$, so from Equation (108):

$$\frac{{\partial }^2\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j}={\int }_{\mathcal{X}}{f}^{\prime \prime }(\rho \left(p\left(\zeta \right|\theta )\right){\lambda }_i\left(\zeta \right){\lambda }_j\left(\zeta \right)d\mu$$

(174)

That the above expression is positive definite is seen by observing:

$$\sum _{ij}\frac{{\partial }^2\mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i\partial {\theta }^j}{\xi }^i{\xi }^j={\int }_{\mathcal{X}}{f}^{\prime \prime }(\rho \left(p\left(\zeta \right|\theta )\right){\left(\sum _i{\lambda }_i\left(\zeta \right){\xi }^i\right)}^{2}d\mu >0$$

(175)

for any $\xi =[{\xi }^1,\cdots ,{\xi }^n]\in {\mathbb{R}}^n$, due to the linear independence of the ${\lambda }_i$ components and the strict convexity of f. Hence, $\mathrm{\Phi }\left(\theta \right)$ is strictly convex in θ, proving (i). An immediate consequence is that expression (127) is non-negative and vanishes, if an only if ${\theta }_p={\theta }_q$. This establishes (ii), i.e., $D_{\mathrm{\Phi }}^{\left(\alpha \right)}({\theta }_p,{\theta }_q)$ is a divergence functions. Part (iii) follows from a straight-forward application of Eguchi relations Equations (29)–(31). ⋄

Proof of Corollary 10. First, since $f^{\prime }\left(\rho \left(t\right)\right)=\tau \left(t\right)$, we have the identity:

$$f^{*}(\tau \left(p\left(\zeta \right|\theta )\right)+f\left(\rho \left(p\left(\zeta \right|\theta )\right)\right)=f^{\prime }\left(\rho \left(p\left(\zeta \right|\theta )\right)\right)\rho \left(p\left(\zeta \right|\theta )\right)$$

(176)

From (126), taking a derivative with respect to ${\theta }^i$, while noting that $p\left(\zeta \right|\theta )$ satisfies Equation (124), gives:

$$\frac{\partial \mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i}={\int }_{\mathcal{X}}{f}^{\prime }\left(\sum _j{\theta }^j{\lambda }_j\left(\zeta \right)\right){\lambda }_i\left(\zeta \right)d\mu ={\int }_{\mathcal{X}}\tau \left(p\left(\zeta \right|\theta )\right){\lambda }_i\left(\zeta \right)d\mu ={\eta }_i$$

(177)

and that:

$$\sum _i{\theta }^i\frac{\partial \mathrm{\Phi }\left(\theta \right)}{\partial {\theta }^i}-\mathrm{\Phi }\left(\theta \right)={\int }_{\mathcal{X}}\left\{f^{\prime }\left(\sum _j{\theta }^j{\lambda }_j\left(\zeta \right)\right)\left(\sum _i{\theta }^i{\lambda }_i\left(\zeta \right)\right)-f\left(\sum _j{\theta }^j{\lambda }_j\left(\zeta \right)\right)\right\}d\mu$$

(178)

$$={\int }_{\mathcal{X}}{f}^{*}\left(\tau \left(p\left(\zeta \right|\theta )\right)\right)d\mu =\tilde{\mathrm{\Phi }}\left(\theta \right)$$

(179)

It follows from Equation (131) that ${\mathrm{\Phi }}^{*}$, as defined in (i), is the conjugate of Φ, and that the relation in (ii) is the basic Legendre–Fenchel duality. Finally, the biorthogonality of η and θ as expressed by (iii) also becomes evident on account of (ii). ⋄

5. Discussions

This paper constructs a family of divergence functionals, induced by any smooth and strictly convex function, to measure the non-symmetric “distance” between two measurable functions defined on a sample space. Subject to an arbitrary monotone scaling, the divergence functional induces a Riemannian manifold with a metric tensor generalizing the conventional Fisher information and a pair of conjugate connections generalizing the conventional $(\pm \alpha )$-connections. Such manifolds manifest biduality: referential duality (in choosing a reference point) and representational duality (in choosing a monotone scale). The $(\alpha ,\beta )$-divergence we gave as an example of this bidualistic structure extends the α-divergence, with α and β representing referential duality and representational duality, respectively. It induces the conventional Fisher metric and the conventional α-connection (with $\alpha \beta$ as a single parameter). Finally, for the ρ-affine submanifold, a pair of conjugated potentials exist to induce the natural and expectation parameters as biorthogonal coordinates on the manifold.

Our approach demonstrated an intimate connection between convex analysis and information geometry. The divergence functionals (and the divergence functions in the finite-dimensional case) are associated with the fundamental convex inequality of a convex function, $f:\mathbb{R}\to \mathbb{R}$ (or $\mathrm{\Phi }:{\mathbb{R}}^n\to \mathbb{R}$), with the convex mixture coefficient as the α-parameter in the induced geometry. Referential duality is associated with $\alpha ⟷-\alpha$, and representational duality is associated with the convex conjugacy $f⟷f^{*}$ (or $\mathrm{\Phi }⟷{\mathrm{\Phi }}^{*}$). Thus, our analysis reveals that the $e/m$-duality and $(\pm 1)$-duality that were used almost interchangeably in the current literature are not the same thing!

The kind of referential duality (originating from non-symmetric status for a referent and for a comparison object), while common in psychological and behavioral contexts [54,55], has always been implicitly acknowledged in statistics. Formal investigation of such non-symmetry between a reference probability distribution and comparison probability distribution in constructing divergence functions leads to the framework of preferred point geometry [56,57,58,59,60,61]. Preferred point geometry reformulates Amari’s [20] expected geometry and Barndorff-Nielsen’s [3] observed geometry by studying the product manifold, ${\mathcal{M}}_{\theta }\times {\mathcal{M}}_{\theta }$, formed by an ordered pair of probability densities, $(p,q)$, and defining a family of Riemannian metric defined on the product manifold. The precise relation of the preferred point approach with our approach to referential duality needs future exploration.

With respect to representational duality, it is worth mentioning the field of affine differential geometry which studies hypersurface realization of the dual Riemannian manifold involving a pair of conjugate connections (see [27,28]). [31,32,33,34] investigated affine immersion of statistical manifolds. [62,63,64,65] further illuminated a conformal structure when the (normalized) probability density functions undergo the $l^{\left(\alpha \right)}$ embedding. Such an embedding appears in the context of Tsallis statistics, where Shannon entropy and Kullback-Leibler cross-entropy (divergence) is generalized to a one-parameter family of entropy and cross-entropy (see, e.g., [40]). We demonstrated ([48], and here) that the ρ-affine manifold (Section 3.2) has the structure of an α-Hessian structure [26], a generalization of Hessian manifold [66,67]. It remains to be illuminated whether a conformal structure arises for ρ-affine probability density functions after normalization.

It should be noted that, while any divergence function determines uniquely a statistical manifold (in the broad sense of [29]), the converse is not true. Though a statistical manifold equipped with an arbitrary metric tensor and a pair of conjugate, torsion-free connections always admits a divergence function [68], it is not unique in general, except when the connections are dually flat, in which case the divergence is uniquely determined as the canonical divergence. In this sense, there is nothing special about our use of ${\mathcal{D}}^{\left(\alpha \right)}$-divergence apart from it generalizing familiar divergences (including α-divergence in particular). Rather, ${\mathcal{D}}^{\left(\alpha \right)}$-divergence is merely a vehicle for us to derive the underlying dual Riemannian geometry. It remains to be elucidated why the convex mixture parameter turns out to be the α-parameter in the family of connections of the induced geometry. It seems that our generalizations of the Fisher metric and of conjugate α-connections hinge on this miraculous identification. Generalization from α-affinity/embedding to ρ-affinity/embedding, and the resulting generalized biorthogonality between natural and expectation parameters is akin to generalizing $L_p$ space to $L_{\mathrm{\Phi }}$ (i.e., Orlicz) space, which is an entirely different matter. Future research will further clarify these fundamental relations between convexity, conjugacy, and duality in non-parametric (and parametric) information geometry.

6. Conclusions

We constructed an extension of parametric information geometry to the non-parametric setting by studying the manifold $\mathcal{M}$ of non-parametric functions on sample space (without positivity and normalization constraints). The generalized Fisher information and α-connections on $\mathcal{M}$ are induced by an α-parameterized family of divergence functions, reflecting the fundamental convex inequality associated with any smooth and strictly convex function. Parametric models are recovered as submanifolds of $\mathcal{M}$. We also generalize Amari’s α-embedding to an affine submanifold under arbitrary monotonic embedding, and show that its natural and expectation parameters form biorthogonal coordinates, and such a submanifold is dually flat for $\alpha =\pm 1$. Our analysis illuminates two different types of duality in information geometry, one concerning the referential status of a point (measurable function) expressed in the divergence function (“referential duality”) and the other concerning its representation under an arbitrary monotone scaling (“representational duality”).