Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences

Abstract

This work reviews and extends a family of log-determinant (log-det) divergences for symmetric positive definite (SPD) matrices and discusses their fundamental properties. We show how to use parameterized Alpha-Beta (AB) and Gamma log-det divergences to generate many well-known divergences; in particular, we consider the Stein’s loss, the S-divergence, also called Jensen-Bregman LogDet (JBLD) divergence, Logdet Zero (Bhattacharyya) divergence, Affine Invariant Riemannian Metric (AIRM), and other divergences. Moreover, we establish links and correspondences between log-det divergences and visualise them on an alpha-beta plane for various sets of parameters. We use this unifying framework to interpret and extend existing similarity measures for semidefinite covariance matrices in finite-dimensional Reproducing Kernel Hilbert Spaces (RKHS). This paper also shows how the Alpha-Beta family of log-det divergences relates to the divergences of multivariate and multilinear normal distributions. Closed form formulas are derived for Gamma divergences of two multivariate Gaussian densities; the special cases of the Kullback-Leibler, Bhattacharyya, Rényi, and Cauchy-Schwartz divergences are discussed. Symmetrized versions of log-det divergences are also considered and briefly reviewed. Finally, a class of divergences is extended to multiway divergences for separable covariance (or precision) matrices.

1. Introduction

Divergences or (dis)similarity measures between symmetric positive definite (SPD) matrices underpin many applications, including: Diffusion Tensor Imaging (DTI) segmentation, classification, clustering, pattern recognition, model selection, statistical inference, and data processing problems [1–3]. Furthermore, there is a close connection between divergence and the notions of entropy, information geometry, and statistical mean [2,4–7], while matrix divergences are closely related to the invariant geometrical properties of the manifold of probability distributions [4,8–10]. A wide class of parameterized divergences for positive measures are already well understood and a unification and generalization of their properties can be found in [11–13].

The class of SPD matrices, especially covariance matrices, play a key role in many areas of statistics, signal/image processing, DTI, pattern recognition, and biological and social sciences [14–16]. For example, medical data produced by diffusion tensor magnetic resonance imaging (DTI-MRI) represent the covariance in a Brownian motion model of water diffusion. The diffusion tensors can be represented as SPD matrices, which are used to track the diffusion of water molecules in the human brain, with applications such as the diagnosis of mental disorders [14]. In array processing, covariance matrices capture both the variance and correlation of multidimensional data; this data is often used to estimate (dis)similarity measures, i.e., divergences. This all has led to an increasing interest in divergences for SPD (covariance) matrices [1,5,6,14,17–20].

The main aim of this paper is to review and extend log-determinant (log-det) divergences and to establish a link between log-det divergences and standard divergences, especially the Alpha, Beta, and Gamma divergences. Several forms of the log-det divergence exist in the literature, including the log–determinant α divergence, Riemannian metric, Stein’s loss, S-divergence, also called the Jensen-Bregman LogDet (JBLD) divergence, and the symmetrized Kullback-Leibler Density Metric (KLDM) or Jeffrey’s KL divergence [5,6,14,17–20]. Despite their numerous applications, common theoretical properties and the relationships between these divergences have not been established. To this end, we propose and parameterize a wide class of log-det divergences that provide robust solutions and/or even improve the accuracy for a noisy data. We next review fundamental properties and provide relationships among the members of this class. The advantages of some selected log-det divergences are also discussed; in particular, we consider the efficiency, simplicity, and resilience to noise or outliers, in addition to simplicity of calculations [14]. The log-det divergences between two SPD matrices have also been shown to be robust to biases in composition, which can cause problems for other similarity measures.

The divergences discussed in this paper are flexible enough to facilitate the generation of several established divergences (for specific values of the tuning parameters). In addition, by adjusting the adaptive tuning parameters, we optimize the cost functions of learning algorithms and estimate desired model parameters in the presence of noise and outliers. In other words, the divergences discussed in this paper are robust with respect to outliers and noise if the tuning parameters, α, β, and γ, are chosen properly.

1.1. Preliminaries

We adopt the following notation: SPD matrices will be denoted by P ∈ ℝ^n×n and Q ∈ ℝ^n×n, and have positive eigenvalues λ_i (sorted in descending order); by log(P), det(P) = |P|, tr(P) we denote the logarithm, determinant, and trace of P, respectively.

For any real parameter α ∈ ℝ and for a positive definite matrix P, the matrix P^α is defined using symmetric eigenvalue decomposition as follows:

$${\mathbf{P}}^{\alpha }={(\mathbf{V}\mathbf{\Lambda }{\mathbf{V}}^T)}^{\alpha }=\mathbf{V}({\mathbf{\Lambda }}^{\alpha }){\mathbf{V}}^T,$$

(1)

where Λ is a diagonal matrix of the eigenvalues of P, and V ∈ ℝ^n×n is the orthogonal matrix of the corresponding eigenvectors. Similarly, we define

$$\log {\mathbf{P}}^{\alpha }=\log ({(\mathbf{V}\mathbf{\Lambda }{\mathbf{V}}^T)}^{\alpha })=\mathbf{V}\log ({\mathbf{\Lambda }}^{\alpha }){\mathbf{V}}^T,$$

(2)

where log(Λ) is a diagonal matrix of logarithms of the eigenvalues of P. The basic operations for positive definite matrices are provided in Appendix A.

The dissimilarity between two SPD matrices is called a metric if the following conditions hold:

• D(P ║ Q) ≥ 0, where the equality holds if and only if P = Q (nonnegativity and positive definiteness).
• D(P ║ Q) = D(Q ║ P) (symmetry).
• D(P ║ Z) ≤ D(P ║ Q) + D(Q ║ Z) (subaddivity/triangle inequality).

Dissimilarities that only satisfy condition (1) are not metrics and are referred to as (asymmetric) divergences.

2. Basic Alpha-Beta Log-Determinant Divergence

For SPD matrices P ∈ ℝ^n×n and Q ∈ ℝ^n×n, consider a new dissimilarity measure, namely, the AB log-det divergence, given by

$$\begin{array}{r}{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha \beta }\log \det \frac{\alpha {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+\beta {(\mathbf{P}{\mathbf{Q}}^{-1})}^{-\alpha }}{\alpha +\beta }\\ \text{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0,\end{array}$$

(3)

where the values of the parameters α and β can be chosen so as to guarantee the non-negativity of the divergence and it vanishes to zero if and only if P = Q (this issue is addressed later by Theorems 1 and 2). Observe that this is not a symmetric divergence with respect to P and Q, except when α = β. Note that using the identity log det(P) = tr log(P), the divergence in (3) can be expressed as

$$\begin{array}{r}{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha \beta }\mathrm{tr}\left[\log \left(\frac{\alpha {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+\beta {(\mathbf{P}{\mathbf{Q}}^{-1})}^{-\alpha }}{\alpha +\beta }\right)\right]\\ \text{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0.\end{array}$$

(4)

This divergence is related to the Alpha, Beta, and AB divergences discussed in our previous work, especially Gamma divergences [11–13,21]. Furthermore, the divergence in (4) is related to the AB divergence for SPD matrices [1,12], which is defined by

$$\begin{array}{r}{\overline{D}}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha \beta }\mathrm{tr}\left(\frac{\alpha }{\alpha +\beta }{\mathbf{P}}^{\alpha +\beta }+\frac{\beta }{\alpha +\beta }{\mathbf{Q}}^{\alpha +\beta }-{\mathbf{P}}^{\alpha }{\mathbf{Q}}^{\beta }\right)\\ \text{for}\alpha \ne 0,\beta \ne 0.\alpha +\beta \ne 0.\end{array}$$

(5)

Note that α and β are chosen so that $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})$ is nonnegative and equal to zero if P = Q. Moreover, such divergence functions can be evaluated without computing the inverses of the SPD matrices; instead, they can be evaluated easily by computing (positive) eigenvalues of the matrix PQ⁻¹ or its inverse. Since both matrices P and Q (and their inverses) are SPD matrices, their eigenvalues are positive. In general, it can be shown that even though PQ⁻¹ is nonsymmetric, its eigenvalues are the same as those of the SPD matrix Q^−1/2PQ^−1/2; hence, its eigenvalues are always positive.

Next, consider the eigenvalue decomposition:

$${(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }=\mathbf{V}{\mathbf{\Lambda }}^{\beta }{\mathbf{V}}^{-1},$$

(6)

where V is a nonsingular matrix, and ${\mathbf{\Lambda }}^{\beta }=\mathrm{diag}\{{\mathrm{\lambda }}_1^{\beta },{\mathrm{\lambda }}_2^{\beta },\dots ,{\mathrm{\lambda }}_n^{\beta }\}$ is the diagonal matrix with the positive eigenvalues λ_i > 0, i = 1, 2, …, n, of PQ⁻¹. Then, we can write

$$\begin{array}{l}{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha \beta }\log \det \frac{\alpha \mathbf{V}{\mathbf{\Lambda }}^{\beta }{\mathbf{V}}^{-1}+\beta \mathbf{V}{\mathbf{\Lambda }}^{-\alpha }{\mathbf{V}}^{-1}}{\alpha +\beta }\\ =\frac{1}{\alpha \beta }\log \left[\det \mathbf{V}\det \frac{\alpha {\mathbf{\Lambda }}^{\beta }+\beta {\mathbf{\Lambda }}^{-\alpha }}{\alpha +\beta }\det {\mathbf{V}}^{-1}\right]\\ =\frac{1}{\alpha \beta }\log \det \frac{\alpha {\mathbf{\Lambda }}^{\beta }+\beta {\mathbf{\Lambda }}^{-\alpha }}{\alpha +\beta },\end{array}$$

(7)

which allows us to use simple algebraic manipulations to obtain

$$\begin{array}{l}{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha \beta }\log {\displaystyle \prod _{i=1}^n\frac{\alpha {\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-\alpha }}{\alpha +\beta }}\\ =\frac{1}{\alpha \beta }{\displaystyle \sum _{i=1}^n\log }\left(\frac{\alpha {\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-\alpha }}{\alpha +\beta }\right),\alpha ,\beta ,\alpha +\beta \ne 0.\end{array}$$

(8)

It is straightforward to verify that $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=0$ if P = Q. We will show later that this function is nonnegative for any SPD matrices if the α and β parameters take both positive or negative values.

For the singular values α = 0 and/or β = 0 (also α = −β), the AB log-det divergence in (3) is defined as a limit for α → 0 and/or β → 0. In other words, to avoid indeterminacy or singularity for specific parameter values, the AB log-det divergence can be reformulated or extended by continuity and by applying L’Hôpital’s formula to cover the singular values of α and β. Using L’Hôpital’s rule, the AB log-det divergence can be defined explicitly by

$$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\Vert \mathbf{Q})=\{\begin{array}{ll}\frac{1}{\alpha \beta }\log \det \frac{\alpha {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+\beta {(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha }}{\alpha +\beta }\hfill & \text{for}\alpha ,\beta \ne 0,\alpha +\beta \ne 0\hfill \\ \frac{1}{{\alpha }^2}[\mathrm{tr}\left({(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha }-\mathbf{I}\right)-\alpha \log \det (\mathbf{Q}{\mathbf{P}}^{-1})]\hfill & \text{for}\alpha \ne 0,\beta =0\hfill \\ \frac{1}{{\beta }^2}[\mathrm{tr}\left({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }-\mathbf{I}\right)-\beta \log \det (\mathbf{P}{\mathbf{Q}}^{-1})]\hfill & \text{for}\alpha =0,\beta \ne 0\hfill \\ \frac{1}{{\alpha }^2}\log \frac{\det {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }}{\det (\mathbf{I}+\log {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha })}\hfill & \text{for}\alpha =-\beta \ne 0\hfill \\ \frac{1}{2}\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})=\frac{1}{2}\parallel \log ({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{-1/2}){\parallel }_F^2\hfill & \text{for}\alpha ,\beta =0.\hfill \end{array}$$

(9)

Equivalently, using standard matrix manipulations, the above formula can be expressed in terms of the eigenvalues of PQ⁻¹, i.e., the generalized eigenvalues computed from λ_iQv_i = Pv_i (where v_i (i = 1, 2, …, n) are corresponding generalized eigenvectors) as follows:

$$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\{\begin{array}{ll}\frac{1}{\alpha \beta }{\displaystyle \sum _{i=1}^n\log }\left(\frac{\alpha {\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-\alpha }}{\alpha +\beta }\right)\hfill & \text{for}\alpha ,\beta \ne 0,\alpha +\beta \ne 0\hfill \\ \frac{1}{{\alpha }^2}\left[{\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{-\alpha }-\log ({\mathrm{\lambda }}_i^{-\alpha }))-n}\right]\hfill & \text{for}\alpha \ne 0,\beta =0\hfill \\ \frac{1}{{\beta }^2}\left[{\displaystyle \sum _{i=1}^n\left({\mathrm{\lambda }}_i^{-\beta }-\log ({\mathrm{\lambda }}_i^{-\beta })\right)-n}\right]\hfill & \text{for}\alpha =0,\beta \ne 0\hfill \\ \frac{1}{{\alpha }^2}\left[{\displaystyle \sum _{i=1}^n\log \left(\frac{{\mathrm{\lambda }}_i^{\alpha }}{1+\log {\mathrm{\lambda }}_i^{\alpha }}\right)}\right]\hfill & \text{for}\alpha =-\beta \ne 0\hfill \\ \frac{1}{2}{\displaystyle \sum _{i=1}^n{\log }^2({\mathrm{\lambda }}_i)}\hfill & \text{for}\alpha ,\beta =0.\hfill \end{array}$$

(10)

Theorem 1. The function$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})\ge 0$ given by (3) is nonnegative for any SPD matrices with arbitrary positive eigenvalues if α ≥ 0 and β ≥ 0 or if α < 0 and β < 0. It is equal to zero if and only if P = Q.

Equivalently, if the values of α and β have the same sign, the AB log-det divergence is positive independent of the distribution of the eigenvalues of PQ⁻¹ and goes to zero if and only if all the eigenvalues are equal to one. However, if the eigenvalues are sufficiently close to one, the AB log-det divergence is also positive for different signs of α and β. The conditions for positive definiteness are given by the following theorem.

Theorem 2. The function$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})$ given by (9) is nonnegative if α > 0 and β < 0 or if α < 0 and β > 0 and if all the eigenvalues of PQ⁻¹ satisfy the following conditions:

$${\mathrm{\lambda }}_i>{\left|\frac{\beta }{\alpha }\right|}^{\frac{1}{\alpha +\beta }}\forall i,for\alpha >0and\beta <0,$$

(11)

and

$${\mathrm{\lambda }}_i<{\left|\frac{\beta }{\alpha }\right|}^{\frac{1}{\alpha +\beta }}\forall i,for\alpha <0and\beta >0.$$

(12)

If any of the eigenvalues violate these bounds, the value of the divergence, by definition, is infinite. Moreover, when α → −β these bounds simplify to

$${\mathrm{\lambda }}_i>e^{-1/\alpha }\forall i,\alpha =-\beta >0,$$

(13)

$${\mathrm{\lambda }}_i<e^{-1/\alpha }\forall i,\alpha =-\beta <0.$$

(14)

In the limit, when α → 0 or β → 0, the bounds disappear. A visual presentation of these bounds for different values of α and β is shown in Figure 1.

Additionally, $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=0$ only if λ_i = 1 for all i = 1, …, n, i.e., when P = Q.

The proofs of these theorems are given in Appendices B, C and D.

Figure 2 illustrates the typical shapes of the AB log-det divergence for different values of the eigenvalues for various choices of α and β.

In general, the AB log-det divergence is not a metric distance since the triangle inequality is not satisfied for all parameter values. Therefore, we can define the metric distance as the square root of the AB log-det divergence in the special case when α = β as follows:

$$d_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})=\sqrt{D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q}).}$$

(15)

This follows from the fact that $D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})$ is symmetric with respect to P and Q.

Later, we will show that measures defined in this manner lead to many important and well-known divergences and metric distances such as the Logdet Zero divergence, Affine Invariant Riemannian metric (AIRM), and square root of Stein’s loss [5,6]. Moreover, new divergences can be generated; specifically, generalized Stein’s loss, the Beta-log-det divergence, and extended Hilbert metrics.

From the divergence $D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})$, a Riemannian metric and a pair of dually coupled affine connections are introduced in the manifold of positive definite matrices. Let dP be a small deviation of P, which belongs to the tangent space of the manifold at P. Calculating $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}+d\mathbf{P}\parallel \mathbf{P})$ and neglecting higher-order terms yields (see Appendix E)

$$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}+d\mathbf{P}\parallel \mathbf{P})=\frac{1}{2}\mathrm{tr}[d\mathbf{P}{\mathbf{P}}^{-1}d\mathbf{P}{\mathbf{P}}^{-1}].$$

(16)

This gives a Riemannian metric that is common for all (α, β). Therefore, the Riemannian metric is the same for all AB log-det divergences, although the dual affine connections depend on α and β. The Riemannian metric is also the same as the Fisher information matrix of the manifold of multivariate Gaussian distributions of mean zero and covariance matrix P.

Interestingly, note that the Riemannian metric or geodesic distance is obtained from (3) for α = β = 0:

$$d_R(\mathbf{P}\parallel \mathbf{Q})=\sqrt{2D_{AB}^{(0,0)}(\mathbf{P}\parallel \mathbf{Q})}=\sqrt{\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})}$$

(17)

$$={\Vert \log (\mathbf{P}{\mathbf{Q}}^{-1})\Vert }_F={\Vert \log ({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{-1/2})\Vert }_F$$

(18)

$$=\sqrt{{\displaystyle \sum _{i=1}^n{\log }^2({\mathrm{\lambda }}_i)}},$$

(19)

where λ_i are the eigenvalues of PQ⁻¹.

This is also known as the AIRM. AIRM takes advantage of several important and useful theoretical properties and is probably one of the most widely used (dis)similarity measure for SPD (covariance) matrices [14,15].

For α = β = 0.5 (and for α = β = −0.5), the recently defined and deeply analyzed S-divergence (JBLD) [6,14,15,17] is obtained:

$$\begin{array}{l}{D}_S(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(0.5,0.5)}(\mathbf{P}\parallel \mathbf{Q})=4\log \det \left(\frac{1}{2}\left[{(\mathbf{P}{\mathbf{Q}}^{-1})}^{1/2}+{(\mathbf{P}{\mathbf{Q}}^{-1})}^{1/2}\right]\right)\\ =4\log \frac{\det {(\mathbf{P})}^{1/2}\det \left(\frac{{(\mathbf{P}{\mathbf{Q}}^{-1})}^{1/2}+{(\mathbf{P}{\mathbf{Q}}^{-1})}^{-1/2}}{2}\right)\det {(\mathbf{Q})}^{1/2}}{\det {(\mathbf{P})}^{1/2}\det {(\mathbf{Q})}^{1/2}}\\ =4\log \frac{\det \frac{1}{2}(\mathbf{P}+\mathbf{Q})}{\sqrt{\det (\mathbf{P})\det (\mathbf{Q})}}\\ =4\left(\log \det \left(\frac{(\mathbf{P}+\mathbf{Q})}{2}\right)-\frac{1}{2}\log \det (\mathbf{PQ})\right)=4{\displaystyle \sum _{i=1}^n\left(\frac{{\mathrm{\lambda }}_i+1}{2\sqrt{{\mathrm{\lambda }}_i}}\right).}\end{array}$$

(20)

The S-divergence is not a metric distance. To make it a metric, we take its square root and obtain the LogDet Zero divergence, or Bhattacharyya distance [5,7,18]:

$$\begin{array}{l}{d}_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})=\sqrt{D_{AB}^{(0.5,0.5)}(\mathbf{P}\parallel \mathbf{Q})}\\ =2\sqrt{\log \det \left(\frac{(\mathbf{P}+\mathbf{Q})}{2}\right)-}\frac{1}{2}\log \det (\mathbf{PQ})\\ =2\sqrt{\log \frac{\det \frac{1}{2}(\mathbf{P}+\mathbf{Q})}{\sqrt{\det (\mathbf{P})\det (\mathbf{Q})}}.}\end{array}$$

(21)

Moreover, for α = 0, β ≠ 0 and α ≠ 0, β = 0, we obtain divergences which are generalizations of Stein’s loss (called also Burg matrix divergence or simply LogDet divergence):

$$D_{AB}^{(0,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{{\beta }^2}[\mathrm{tr}\left({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }-\mathbf{I}\right)-\beta \log \det (\mathbf{P}{\mathbf{Q}}^{-1})],\beta \ne 0.$$

(22)

$$D_{AB}^{(\alpha ,0)}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{{\alpha }^2}[\mathrm{tr}\left({(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha }-\mathbf{I}\right)-\alpha \log \det (\mathbf{Q}{\mathbf{P}}^{-1})],\alpha \ne 0$$

(23)

The divergences in (22) and (23) simplify, respectively, to the standard Stein’s loss if β = 1 and to its dual loss if α = 1.

3. Special Cases of the AB Log-Det Divergence

We now illustrate how a suitable choice of the (α, β) parameters simplify the AB log-det divergence into other known divergences such as the Alpha- and Beta-log-det divergences [5,11,18,23] (see Figure 3 and

Table 1. Fundamental Log-det Divergences and Distances

Geodesic Distance (AIRM) (α = β = 0) $\frac{1}{2}{d}_R^2(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})=\frac{1}{2}{\displaystyle \sum _{i-1}^n{\log }^2{\mathrm{\lambda }}_i}$

S-divergence (Squared Bhattacharyya Distance) (α = β = 0.5) $D_S(\mathbf{P}\parallel \mathbf{Q})=d_{\mathrm{Bh}}^2(\mathbf{P}\parallel \mathbf{Q})=4\log \frac{\det \frac{(\mathbf{P}+\mathbf{Q})}{2}}{{(\det \mathbf{PQ})}^{\frac{1}{2}}}=4{\displaystyle \sum _{i-1}^n\log \frac{{\mathrm{\lambda }}_i+1}{2\sqrt{{\mathrm{\lambda }}_i}}}$

Power divergence (α = β ≠ 0) $\frac{1}{{\alpha }^2}\log \det \frac{{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }-{(\mathbf{P}{\mathbf{Q}}^{-1})}^{-\alpha }}{2}=\frac{1}{{\alpha }^2}{\displaystyle \sum \log \frac{{\mathrm{\lambda }}_i^{\alpha }+{\mathrm{\lambda }}_i^{-\alpha }}{2}}$

Generalized Burg divergence (Stein’s Loss) (α =0, β ≠ 0) $\frac{1}{{\beta }^2}\mathrm{tr}\left[{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }-I\right]-\log \det {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }=\frac{1}{{\beta }^2}\left({\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{\beta }-\log {\mathrm{\lambda }}_i^{\beta })-n}\right)$

Generalized Itakura-Saito log-det divergence (α =−β ≠ 0) $\frac{1}{{\alpha }^2}\log \det \frac{\det {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }}{\det \mathbf{I}+\log {(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }}=\frac{1}{{\alpha }^2}{\displaystyle \sum _{i=1}^n\log \frac{{\mathrm{\lambda }}_i^{\alpha }}{1+{\log }^2{\mathrm{\lambda }}_i^{\alpha }}}$

Alpha log-det divergence (0 < α < 1, β = 1 − α) $D_A^{(\alpha )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\alpha (1-\alpha )}\log \frac{\det (\alpha \mathbf{P}+(1-\alpha )\mathbf{Q})}{\det ({\mathbf{P}}^{\alpha }{\mathbf{Q}}^{1-\alpha })}=\frac{1}{\alpha (1-\alpha )}{\displaystyle \sum _{i-1}^n\log \left(\frac{\alpha ({\mathrm{\lambda }}_i+1)+1}{{\mathrm{\lambda }}_i^{\alpha }}\right)}$

Beta log-det divergence (α =1, β ≥ 0) $D_A^{(\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{\beta }\log \det \frac{{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+\beta (\mathbf{P}{\mathbf{Q}}^{-1})}{1+\beta }=\frac{1}{\beta }{\displaystyle \sum _{i-1}^n\log \frac{{\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-1}}{1+\beta }}$ $D_A^{(\infty )}(\mathbf{P}\parallel \mathbf{Q})={\displaystyle \sum _{i\in \mathrm{\Omega }}\log {\mathrm{\lambda }}_i}$, Ω = {i= λi > 1}

Symmetric Jeffrey KL divergence (α =1, β = 0) $D_J(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\mathrm{tr}(\mathbf{P}{\mathbf{Q}}^{-1}+\mathbf{Q}{\mathbf{P}}^{-1}-2\mathbf{I})=\frac{1}{2}{\displaystyle \sum _{i-1}^n{\left(\sqrt{{\mathrm{\lambda }}_i}-\frac{1}{\sqrt{{\mathrm{\lambda }}_i}}\right)}^2}$

Generalized Hilbert metrics $D_{CCA}^{({\gamma }_2,{\gamma }_1)}(\mathbf{P}\parallel \mathbf{Q})=\log \frac{M_{{\gamma }_2}\left\{{\mathrm{\lambda }}_i\right\}}{M_{{\gamma }_2}\left\{{\mathrm{\lambda }}_i\right\}},d_H(\mathbf{P}\parallel \mathbf{Q})=\log \frac{M_{\infty }\left\{{\mathrm{\lambda }}_i\right\}}{M_{-\infty }\left\{{\mathrm{\lambda }}_i\right\}}=\log \frac{{\mathrm{\lambda }}_{max}}{{\mathrm{\lambda }}_{min}}$

).

When α + β = 1, the AB log-det divergence reduces to the Alpha-log-det divergence [5]:

$$D_{AB}^{(\alpha ,1-\alpha )}(\mathbf{P}\parallel \mathbf{Q})=D_A^{(\alpha )}(\mathbf{P}\parallel \mathbf{Q})\doteq \{\begin{array}{ll}\begin{array}{l}\frac{1}{\alpha (1-\alpha )}\log \det [\alpha {({\mathbf{PQ}}^{-1})}^{1-\alpha }+(1-\alpha ){(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha }]=\\ \frac{1}{\alpha (1-\alpha )}\log \frac{\det \left(\alpha \mathbf{P}+(1-\alpha )\mathbf{Q}\right)}{\det ({\mathbf{P}}^{\alpha }{\mathbf{Q}}^{1-\alpha })}=\\ \frac{1}{\alpha (1-\alpha )}{\displaystyle \sum _{i=1}^n\log }\left(\frac{\alpha ({\mathrm{\lambda }}_i-1)+1}{{\mathrm{\lambda }}_i^{\alpha }}\right)\end{array}\hfill & \hfill \text{for}0<\alpha <1,\hfill \\ \mathrm{tr}\left(\mathbf{Q}{\mathbf{P}}^{-1}\right)-\log \det (\mathbf{Q}{\mathbf{P}}^{-1})-n={\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{-1}+\log ({\mathrm{\lambda }}_i))-n}\hfill & \hfill \text{for}\alpha =1,\hfill \\ \mathrm{tr}\left(\mathbf{Q}{\mathbf{P}}^{-1}\right)-\log \det (\mathbf{Q}{\mathbf{P}}^{-1})-n={\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i-\log ({\mathrm{\lambda }}_i))-n}\hfill & \hfill \text{for}\alpha =0.\hfill \end{array}$$

(24)

On the other hand, when α = 1 and β ≥ 0, the AB log-det divergence reduces to the Beta-log-det divergence:

$$D_{AB}^{(1,\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_A^{(\beta )}(\mathbf{P}\parallel \mathbf{Q})\doteq \{\begin{array}{ll}\frac{1}{\beta }\log \det \frac{{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+\beta (\mathbf{Q}{\mathbf{P}}^{-1})}{1+\beta }=\frac{1}{\beta }{\displaystyle \sum _{i=1}^n\log }\left(\frac{{\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-1}}{1+\beta }\right)\hfill & \hfill \text{for}\beta >0,\hfill \\ \mathrm{tr}(\mathbf{Q}{\mathbf{P}}^{-1}-\mathbf{I})-\log \det (\mathbf{Q}{\mathbf{P}}^{-1})={\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{-1}+\log ({\mathrm{\lambda }}_i))-n}\hfill & \hfill \text{for}\beta =0,\hfill \\ \log \frac{\det (\mathbf{P}{\mathbf{Q}}^{-1})}{\det (\mathbf{I}+\log (\mathbf{P}{\mathbf{Q}}^{-1}))}={\displaystyle \sum _{i=1}^n\log }\frac{{\mathrm{\lambda }}_i}{1+\log ({\mathrm{\lambda }}_i)}\hfill & \hfill \text{for}\beta =-1,{\mathrm{\lambda }}_i>e^{-1\forall i.}\hfill \end{array}$$

(25)

Note that $\det (\mathbf{I}+\log (\mathbf{P}{\mathbf{Q}}^{-1})={\Pi }_{i=1}^n[1+\log ({\mathrm{\lambda }}_i)]$, and the Beta-log-det divergence is well defined for β = −1 and if all the eigenvalues are larger than λ_i > e⁻¹ ≈ 0.367 (e ≈ 2.72).

It is interesting to note that the Beta-log-det divergence for β → ∞ leads to a new divergence that is robust with respect to noise. This new divergence is given by

$$\underset{\beta \to \infty }{\lim }{D}_{AB}^{(\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_A^{(\infty )}(\mathbf{P}\parallel \mathbf{Q})=\log ({\displaystyle \prod _{i=1}^k{\mathrm{\lambda }}_i})\text{for all}{\mathrm{\lambda }}_i\ge 1.$$

(26)

This can be easily shown by applying the L’Hôpital’s formula. Assuming that the set Ω = {i : λ_i > 1} gathers the indices of those eigenvalues greater than one, we can more formally express this divergence as

$$D_B^{(\infty )}(\mathbf{P}\parallel \mathbf{Q})=\{\begin{array}{ll}{\displaystyle {\sum }_{i\in \mathrm{\Omega }}\log {\mathrm{\lambda }}_i}& \text{for}\mathrm{\Omega }\ne \varphi ,\\ 0& \text{for}\mathrm{\Omega }=\varphi .\end{array}$$

(27)

The Alpha-log-det divergence gives the standard Stein’s losses (Burg matrix divergences) for α = 1 and α = 0, and the Beta-log-det divergence is equivalent to Stein’s loss for β = 0.

Another important class of divergences is Power log-det divergences for any α = β ∈ ℝ:

$$D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})=D_P^{(\alpha )}(\mathbf{P}\parallel \mathbf{Q})\doteq \{\begin{array}{cc}\frac{1}{{\alpha }^2}\log \det \frac{{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }+{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }}{2}=\frac{1}{{\alpha }^2}{\displaystyle \sum _{i=1}^n\log \frac{{\mathrm{\lambda }}_i^{\alpha }+{\mathrm{\lambda }}_i^{-\alpha }}{2}}& \text{for}\alpha \ne 0,\\ \frac{1}{2}\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})=\frac{1}{2}\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})=\frac{1}{2}{\displaystyle \sum _{i=1}^n{\log }^2({\mathrm{\lambda }}_i)}& \text{for}\alpha \ne 0.\end{array}$$

(28)

4. Properties of the AB Log-Det Divergence

The AB log-det divergence has several important and useful theoretical properties for SPD matrices.

• Nonnegativity; given by

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})\ge 0,\forall \alpha ,\beta \in ℝ.$$

  (29)
• Identity of indiscernibles (see Theorems 1 and 2); given by

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=0\text{if and only if}\mathbf{P}=\mathbf{Q}.$$

  (30)
• Continuity and smoothness of $D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})$ as a function of α ∈ ℝ and β ∈ ℝ, including the singular cases when α = 0 or β = 0, and when α = −β (see Figure 2).
• The divergence can be expressed in terms of the diagonal matrix Λ = diag{λ₁, λ₂, …, λ_n} with the eigenvalues of PQ⁻¹, in the form

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(\alpha ,\beta )}(\mathbf{\Lambda }\parallel \mathbf{I}).$$

  (31)
• Scaling invariance; given by

  $$D_{AB}^{(\alpha ,\beta )}(c\mathbf{P}\parallel c\mathbf{Q})=D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q}),$$

  (32)

  for any c > 0.
• Relative invariance for scale transformation: For given α and β and nonzero scaling factor ω ≠ 0, we have

  $$D_{AB}^{(\omega \alpha ,\omega \beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{{\omega }^2}{D}_{AB}^{(\alpha ,\beta )}({({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{-1/2})}^{\omega }\parallel \mathbf{I}).$$

  (33)
• Dual-invariance under inversion (for ω = −1); given by

  $$D_{AB}^{(-\alpha ,-\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(\alpha ,\beta )}({\mathbf{P}}^{-1}\parallel {\mathbf{Q}}^{-1}).$$

  (34)
• Dual symmetry; given by

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(\beta ,\alpha )}(\mathbf{Q}\parallel \mathbf{P}).$$

  (35)
• Affine invariance (invariance under congruence transformations); given by

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{AP}{\mathbf{A}}^T\parallel \mathbf{AQ}{\mathbf{A}}^T)=D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q}),$$

  (36)

  for any nonsingular matrix A ∈ ℝ^n×n
• Divergence lower-bound; given by

  $$D_{AB}^{(\alpha ,\beta )}({\mathbf{X}}^T\mathbf{PX}\parallel {\mathbf{X}}^T\mathbf{QX})\le D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q}),$$

  (37)

  for any full-column rank matrix X ∈ ℝ^n×m with n ≤ m.
• Scaling invariance under the Kronecker product; given by

  $$D_{AB}^{(\alpha ,\beta )}(\mathbf{Z}\otimes \mathbf{P}\parallel \mathbf{Z}\otimes \mathbf{Q})=n{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q}),$$

  (38)

  for any symmetric and positive definite matrix Z of rank n.
• Double Sided Orthogonal Procrustes property. Consider an orthogonal matrix $\mathrm{\Omega }\in \mathcal{O}(n)$ and two symmetric positive definite matrices P and Q, with respective eigenvalue matrices Λ_P and Λ_Q which elements are sorted in descending order. The AB log-det divergence between Ω^TPΩ and Q is globally minimized when their eigenspaces are aligned, i.e.,

  $$\underset{\mathrm{\Omega }\in \mathcal{O}(n)}{\min }{D}_{AB}^{(\alpha ,\beta )}({\mathrm{\Omega }}^T\mathbf{P}\mathbf{\Omega }\parallel \mathbf{Q})=D_{AB}^{(\alpha ,\beta )}({\mathbf{\Lambda }}_{\mathbf{P}}\parallel {\mathbf{\Lambda }}_{\mathbf{Q}}).$$

  (39)
• Triangle Inequality-Metric Distance Condition, for α = β ∈ ℝ. The previous property implies the validity of the triangle inequality for arbitrary positive definite matrices, i.e.,

  $$\sqrt{D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})}\le \sqrt{D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Z})}+\sqrt{D_{AB}^{(\alpha ,\alpha )}(\mathbf{Z}\parallel \mathbf{Q})}.$$

  (40)

  The proof of this property exploits the metric characterization of the square root of the S-divergence proposed first by S. Sra in [6,17] for arbitrary SPD matrices.

Several of these properties have been already proved for the specific cases of α and β that lead to the S-divergence (α, β = 1/2) [6], the Alpha log-det divergence (0 ≤ α ≤ 1, β = 1 − α) [5] and the Riemannian metric (α, β = 0) [28, Chapter 6]. We refer the reader to Appendix F for their proofs when α, β ∈ ℝ.

5. Symmetrized AB Log-Det Divergences

The basic AB log-det divergence is asymmetric; that is, $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})\ne D_{AB}^{(\alpha ,\beta )}(\mathbf{Q}\parallel \mathbf{P})$, except the spacial case of α = β).

In general, there are several ways to symmetrize a divergence; for example, Type-1,

$$D_{ABS1}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\left[D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})+D_{ABS1}^{(\alpha ,\beta )}(\mathbf{Q}\parallel \mathbf{P})\right],$$

(41)

and Type-2, based on the Jensen-Shannon symmetrization (which is too complex for log-det divergences),

$$D_{ABS2}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\left[D_{AB}^{(\alpha ,\beta )}\left(\mathbf{P}\parallel \frac{\mathbf{P}+\mathbf{Q}}{2}\right)+D_{ABS1}^{(\alpha ,\beta )}\left(\mathbf{Q}\parallel \frac{\mathbf{P}+\mathbf{Q}}{2}\right)\right].$$

(42)

The Type-1 symmetric AB log-det divergence is defined as

$$D_{ABS1}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\{\begin{array}{ll}\frac{1}{2\alpha \beta }\log \det \left[\mathbf{I}+\frac{\alpha \beta }{{(\alpha +\beta )}^2}({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha +\beta }+{(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha +\beta }-\mathbf{2}\mathbf{I})\right]\hfill & \text{for}\alpha \beta >0,\hfill \\ \frac{1}{2{\alpha }^2}[\mathrm{tr}({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha }+{(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha }-2\mathbf{I})]\hfill & \text{for}\alpha \ne 0,\beta =0,\hfill \\ \frac{1}{2{\beta }^2}[\mathrm{tr}({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\beta }+{(\mathbf{Q}{\mathbf{P}}^{-1})}^{\beta }-2\mathbf{I})]\hfill & \text{for}\alpha =0,\beta \ne 0,\hfill \\ \frac{1}{2{\alpha }^2}\mathrm{tr}\log {(\mathbf{I}-{\log }^2{(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha })}^{-1}\hfill & \text{for}\alpha =-\beta \ne 0,\hfill \\ \frac{1}{2}\mathrm{tr}{\log }^2(\mathbf{P}{\mathbf{Q}}^{-1})=\frac{1}{2}\parallel \log ({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{-1/2}){\parallel }_F^2\hfill & \text{for}\alpha ,\beta =0.\hfill \end{array}$$

(43)

Equivalently, this can be expressed by the eigenvalues of PQ⁻¹ in the form

$$D_{ABS1}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\{\begin{array}{ll}\frac{1}{2\alpha \beta }{\displaystyle \sum _{i=1}^n\log }\left(1+\frac{\alpha \beta }{{(\alpha +\beta )}^2}{({\mathrm{\lambda }}_i^{\frac{\alpha +\beta }{2}}-{\mathrm{\lambda }}_i^{-\frac{\alpha +\beta }{2}})}^2\right)\hfill & \text{for}\alpha \beta >0,\hfill \\ \frac{1}{2{\alpha }^2}{\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{\alpha }+{\mathrm{\lambda }}_i^{-\alpha }-2)=\frac{1}{2{\alpha }^2}{\displaystyle \sum _{i=1}^n{({\mathrm{\lambda }}_i^{\frac{\alpha }{2}}-{\mathrm{\lambda }}_i^{-\frac{\alpha }{2}})}^2}}\hfill & \text{for}\alpha \ne 0,\beta =0,\hfill \\ \frac{1}{2{\beta }^2}{\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i^{\beta }+{\mathrm{\lambda }}_i^{-\beta }-2)=\frac{1}{2{\beta }^2}{\displaystyle \sum _{i=1}^n{({\mathrm{\lambda }}_i^{\frac{\beta }{2}}-{\mathrm{\lambda }}_i^{-\frac{\beta }{2}})}^2}}\hfill & \text{for}\alpha =0,\beta \ne 0,\hfill \\ \frac{1}{2{\alpha }^2}{\displaystyle \sum _{i=1}^n\log \frac{1}{1-{\log }^2({\mathrm{\lambda }}_i^{\alpha })}}\hfill & \text{for}\alpha =-\beta \ne 0,\hfill \\ \frac{1}{2}{\displaystyle \sum _{i=1}^n{\log }^2({\mathrm{\lambda }}_i)}\hfill & \text{for}\alpha ,\beta =0.\hfill \end{array}$$

(44)

We consider several well-known symmetric log-det divergences (see Figure 4); in particular, we consider the following:

• For α = β = ±0.5, we obtain the S-divergence or JBLD divergence (20).
• For α = β = 0, we obtain the square of the AIRM (Riemannian metric) (19).
• For α = 0 and β = ±1 or for β = 0 and α = ±1, we obtain the KLDM (symmetrized KL Density Metric), also known as the symmetric Stein’s loss or Jeffreys KL divergence [3]:

  $$\begin{array}{l}{D}_J(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\mathrm{tr}\left(\mathbf{P}{\mathbf{Q}}^{-1}+\mathbf{Q}{\mathbf{P}}^{-1}-2\mathbf{I}\right)\\ =\frac{1}{2}{{\displaystyle \sum _{i=1}^n\left(\sqrt{{\mathrm{\lambda }}_i-\frac{1}{\sqrt{{\mathrm{\lambda }}_i}}}\right)}}^2\end{array}$$

  (45)

One important potential application of the AB log-det divergence is to generate conditionally positive definite kernels, which are widely applied to classification and clustering. For a specific set of parameters, the AB log-det divergence gives rise to a Hilbert space embedding in the form of a Radial Basis Function (RBF) kernel [22]; more specifically, the AB log-det kernel is defined by

$$\begin{array}{l}{K}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=\exp \left(-\gamma D_{ABS1}^{(\alpha ,\beta )}\left(\mathbf{P}\parallel \mathbf{Q}\right)\right)\\ ={\left(\det \left[\mathbf{I}+\frac{\alpha \beta }{{(\alpha +\beta )}^2}({(\mathbf{P}{\mathbf{Q}}^{-1})}^{\alpha +\beta }+{(\mathbf{Q}{\mathbf{P}}^{-1})}^{\alpha +\beta }-2\mathbf{I})\right]\right)}^{-\frac{\gamma }{2\alpha \beta }}\end{array}$$

(46)

for some selected values of γ > 0 and α, β > 0 or α, β < 0 that can make the kernel positive definite.

6. Similarity Measures for Semidefinite Covariance Matrices in Reproducing Kernel Hilbert Spaces

There are many practical applications for which the underlying covariance matrices are symmetric but only positive semidefinite, i.e., their columns do not span the whole space. For instance, in classification problems, assume two classes and a set of observation vectors {x₁, …, x_T} and {y₁, …, y_T} in ℝ^m for each class, then we may wish to find a principled way to evaluate the ensemble similarity of the data from their sample similarity. The problem of the modeling of similarity between two ensembles was studied by Zhou and Chellappa in [32]. For this purpose, they proposed several probabilistic divergence measures between positive semidefinite covariance matrices in a Reproducing kernel Hilbert space (RKHS) of finite dimensionality. Their strategy was later extended for image classification problems [33] and formalized for the Log-Hilbert-Schmidt metric between infinite-dimensional RKHS covariance operators [34].

In this section, we propose the unifying framework of the AB log-det divergences to reinterpret and extend the similarity measures obtained in [32,33] for semidefinite covariance matrices in the finite-dimensional RKHS.

We shall assume that the nonlinear functions Φ_x : ℝ^m → ℝⁿ and Φ_y : ℝ^m → ℝⁿ (where n > m) respectively map the data from each of the classes into their higher dimensional feature spaces. We implicitly define the feature matrices as

$${\mathbf{\Phi }}_x=[{\mathrm{\Phi }}_x(x_1),\dots ,{\mathrm{\Phi }}_x(x_T)],{\mathbf{\Phi }}_y=[{\mathrm{\Phi }}_y(y_1),\dots ,{\mathrm{\Phi }}_y(y_T)],$$

(47)

and the sample covariance matrices of the observations in the feature space as: ${\mathbf{C}}_x={\mathbf{\Phi }}_x\mathbf{J}{\mathbf{\Phi }}_x^T/T\in {ℝ}^{n\times n}$ and ${\mathbf{C}}_y={\mathrm{\Phi }}_y\mathbf{J}{\mathrm{\Phi }}_y^T/T\in {ℝ}^{n\times n}$, where $\mathbf{J}={\mathbf{I}}_T-\frac{1}{T}{11}^T$ denotes the T × T centering matrix.

In practice, it is common to consider low-rank approximations of sample covariance matrices. For a given basis V_x = (v₁, …, v_r) ∈ ℝ^T×r of the principal subspace of $\mathbf{J}{\mathbf{\Phi }}_x^T{\mathbf{\Phi }}_x\mathbf{J}$, we can define the projection matrix ${\prod }_x={\mathbf{V}}_x{\mathbf{V}}_x^T$ and redefine the covariance matrices as

$${\mathbf{C}}_x=\frac{1}{T}{\mathbf{\Phi }}_x{\mathbf{V}}_x{\mathbf{V}}_x^T{\mathbf{\Phi }}_x^T\text{and}{\mathbf{C}}_y=\frac{1}{T}{\mathbf{\Phi }}_y{\mathbf{V}}_y{\mathbf{V}}_y^T{\mathbf{\Phi }}_y^T.$$

(48)

Assuming the Gaussianity of the data in the feature space, the mean vector and covariance matrix are sufficient statistics and a natural measure of dissimilarity between Φ_x and Φ_y should be a function of the first and second order statistics of the features. Furthermore, in most practical problems the mean value should be ignored due to robustness considerations, and then the comparison reduces to the evaluation of a suitable dissimilarity measure between C_x and C_y.

The dimensionality of the feature space n is typically much larger than r, so the rank of the covariance matrices in (48) will be r ≪ n and, therefore, both matrices are positive semidefinite. The AB log-det divergence is infinite when the range spaces of the covariance matrices C_x and C_y differ. This property is useful in applications which require an automatic constraint in the range of the estimates [22], but it will prohibit the practical use of the comparison when the ranges of the covariance matrices differ. The next subsections present two different strategies to address this challenging problem.

6.1. Measuring the Dissimilarity with a Divergence Lower-Bound

One possible strategy is to use dissimilarity measures which ignore the contribution to the divergence caused by the rank deficiency of the covariance matrices. This is useful when performing one comparison of the covariances matrices after applying a congruence transformation that aligns their range spaces, and can be implemented by retaining only the finite and non-zero eigenvalues of the matrix pencil (C_x, C_y).

Let I_r denote the identity matrix of size r and (·)⁺ the Moore-Penrose pseudoinverse operator. Consider the eigenvalue decomposition of the symmetric matrix

$${({\mathbf{C}}_y^{+})}^{\frac{1}{2}}{\mathbf{C}}_x{({\mathbf{C}}_y^{+})}^{\frac{1}{2}}=\mathbf{U}\mathbf{\Lambda }{\mathbf{U}}^T$$

(49)

where U is a semi-orthogonal matrix for which the columns are the eigenvectors associated with the positive eigenvalues of the matrix pencil and

$$\mathbf{\Lambda }=\mathrm{diag}({\mathrm{\lambda }}_1,\dots {\mathrm{\lambda }}_r)\equiv \mathrm{diag}Ei{g}_{+}\{{({\mathbf{C}}_y^{+})}^{\frac{1}{2}}{\mathbf{C}}_x{({\mathbf{C}}_y^{+})}^{\frac{1}{2}}\}.$$

(50)

is a diagonal matrix with the eigenvalues sorted in a descending order.

Note that the tall matrix $\mathbf{W}={({\mathbf{C}}_y^{+})}^{\frac{1}{2}}\mathbf{U}\in {ℝ}^{n\times r}$ diagonalizes the covariance matrices of the two classes

$${\mathbf{W}}^T{\mathbf{C}}_x\mathbf{W}=\mathbf{\Lambda }$$

(51)

$${\mathbf{W}}^T{\mathbf{C}}_y\mathbf{W}={\mathbf{I}}_r$$

(52)

and compress them to a common range space. The compression automatically discards the singular and infinite eigenvalues of the matrix pencil (C_x, C_y), while it retains the finite and positive eigenvalues. In this way, the following dissimilarity measures can be obtained:

$$L_{AB}^{(\alpha ,\beta )}({\mathbf{C}}_x,{\mathbf{C}}_y)\equiv D_{AB}^{(\alpha ,\beta )}({\mathbf{W}}^T{\mathbf{C}}_x\mathbf{W}\parallel {\mathbf{W}}^T{\mathbf{C}}_y\mathbf{W})=D_{AB}^{(\alpha ,\beta )}(\mathbf{\Lambda }\parallel {\mathbf{I}}_r),$$

(53)

$$L_{ABS1}^{(\alpha ,\beta )}({\mathbf{C}}_x,{\mathbf{C}}_y)\equiv D_{ABS1}^{(\alpha ,\beta )}({\mathbf{W}}^T{\mathbf{C}}_x\mathbf{W}\parallel {\mathbf{W}}^T{\mathbf{C}}_y\mathbf{W})=D_{ABS1}^{(\alpha ,\beta )}(\mathbf{\Lambda }\parallel {\mathbf{I}}_r).$$

(54)

Note, however, that these measures should not be understood as a strict comparison of the original covariance matrices, but rather as an indirect comparison through their respective compressed versions W^TC_xW and W^TC_yW.

With the help of the kernel trick, the next lemma shows that the evaluation of the dissimilarity measures $L_{AB}^{(\alpha ,\beta )}({\mathbf{C}}_x,{\mathbf{C}}_y)$ and $L_{ABS1}^{(\alpha ,\beta )}({\mathbf{C}}_x,{\mathbf{C}}_y)$, does not require the explicit computation of the covariance matrices or of the feature vectors.

Lemma 1. Given the Gram matrix or kernel matrix of the input vectors

$$\left(\begin{array}{cc}{\mathbf{K}}_{xx}& {\mathbf{K}}_{xy}\\ {\mathbf{K}}_{yx}& {\mathbf{K}}_{yy}\end{array}\right)=\left(\begin{array}{cc}{\mathbf{\Phi }}_x^T{\mathbf{\Phi }}_x& {\mathbf{\Phi }}_x^T{\mathbf{\Phi }}_y\\ {\mathbf{\Phi }}_y^T{\mathbf{\Phi }}_x& {\mathbf{\Phi }}_y^T{\mathbf{\Phi }}_y\end{array}\right)$$

(55)

and the matrices V_x and V_y which respectively span the principal subspaces of K_xx and K_yy, the positive and finite eigenvalues of the matrix pencil can be expressed by

$$\mathbf{\Lambda }=\mathrm{diag}Ei{g}_{+}\{({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y){({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y)}^T\}.$$

(56)

Proof. The proof of the lemma relies on the property that for any pair of m × n matrices A and B, the non-zero eigenvalues of AB^T and of B^TA are the same (see [30, pag. 11]). Then, there is an equality between the following matrices of positive eigenvalues

$$\mathbf{\Lambda }=\mathrm{diag}Ei{g}_{+}\left\{{({\mathbf{C}}_y^{+})}^{\frac{1}{2}}{\mathbf{C}}_x{({\mathbf{C}}_y^{+})}^{\frac{1}{2}}\right\}=\mathrm{diag}Ei{g}_{+}\{{\mathbf{C}}_x{\mathbf{C}}_y^{+}\}.$$

(57)

Taking into account the structure of the covariance matrices in (48), such eigenvalues can be explicitly obtained in terms of the kernel matrices

$$Ei{g}_{+}\{{\mathbf{C}}_x{\mathbf{C}}_y^{+}\}=Ei{g}_{+}\{({\mathbf{\Phi }}_x{\mathbf{V}}_x{\mathbf{V}}_x^T{\mathbf{\Phi }}_x^T)({({\mathbf{\Phi }}_y^{+})}^T{\mathbf{V}}_y{\mathbf{V}}_y^T{\mathbf{\Phi }}_y^T)\}$$

(58)

$$=Ei{g}_{+}\{({\mathbf{V}}_x^T{\mathbf{\Phi }}_x^T{({\mathbf{\Phi }}_x^T)}^{+}{\mathbf{V}}_y{\mathbf{V}}_y^T{\mathbf{\Phi }}_y^{+})({\mathbf{\Phi }}_x{\mathbf{V}}_x)\}$$

(59)

$$=Ei{g}_{+}\{({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y){({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y)}^T\}.$$

(60)

6.2. Similarity Measures Between Regularized Covariance Descriptors

Several authors consider a completely different strategy, which consists in the regularization of the original covariance matrices [32–34]. This way the null the eigenvalues of the covariances C_x and C_y are replaced by a small positive constant ρ > 0, to obtain the “regularized” positive definite matrices ${\tilde{\mathbf{C}}}_x$ and ${\tilde{\mathbf{C}}}_y$, respectively. The modification can be illustrated by comparing the eigendecompositions

$${\mathbf{C}}_x=({\mathbf{U}}_x|{\mathbf{U}}_x^{\perp })\left(\begin{array}{cc}{\mathbf{\Lambda }}_x& 0\\ 0& 0\end{array}\right){({\mathbf{U}}_x|{\mathbf{U}}_x^{\perp })}^T={\mathbf{U}}_x{\mathbf{\Lambda }}_x{\mathbf{U}}_x^T$$

(61)

$$\downarrow$$

(62)

$${\tilde{\mathbf{C}}}_x=({\mathbf{U}}_x|{\mathbf{U}}_x^{\perp })\left(\begin{array}{cc}{\mathbf{\Lambda }}_x& 0\\ 0& \rho {\mathbf{I}}_{n-r}\end{array}\right){({\mathbf{U}}_x|{\mathbf{U}}_x^{\perp })}^T={\mathbf{C}}_x+\rho {\mathbf{U}}_x^{\perp }{({\mathbf{U}}_x^{\perp })}^T.$$

(63)

Then, the dissimilarity measure of the data in the feature space can be obtained just by measuring a divergence between the SPD matrices ${\tilde{\mathbf{C}}}_x$ and ${\tilde{\mathbf{C}}}_y$. Again, the idea is to compute the value of the divergence without requiring the evaluation of the feature vectors but by using the available kernels.

Using the properties of the trace and the determinants, a practical formula for the log-det Alpha-divergence has been obtained in [32,33] for 0 < α < 1. The resulting expression

$$D_{AB}^{(\alpha ,1-\alpha )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\frac{1}{\alpha (1-\alpha )}\log \det ({\mathbf{I}}_{2r}+{\rho }^{-1}\mathbf{H})-\frac{1}{(1-\alpha )}\log \det ({\rho }^{-1}{\mathbf{\Lambda }}_x)-\frac{1}{\alpha }\log \det ({\rho }^{-1}{\mathbf{\Lambda }}_y)$$

is a function of the principal eigenvalues of the kernels

$${\mathbf{\Lambda }}_x={\mathbf{V}}_x^T{\mathbf{K}}_{xx}{\mathbf{V}}_x,{\mathbf{\Lambda }}_y={\mathbf{V}}_x^T{\mathbf{K}}_{yy}{\mathbf{V}}_y,$$

(64)

and the matrix

$$\mathbf{H}=\left(\begin{array}{cc}{(\alpha )}^{\frac{1}{2}}{\mathbf{W}}_x& 0\\ 0& {(1-\alpha )}^{\frac{1}{2}}{\mathbf{W}}_y\end{array}\right)\left(\begin{array}{cc}{\mathbf{K}}_{xx}& {\mathbf{K}}_{xy}\\ {\mathbf{K}}_{yx}& {\mathbf{K}}_{yy}\end{array}\right){\left(\begin{array}{cc}{(\alpha )}^{\frac{1}{2}}{\mathbf{W}}_x& 0\\ 0& {(1-\alpha )}^{\frac{1}{2}}{\mathbf{W}}_y\end{array}\right)}^T.$$

(65)

where

$${\mathbf{W}}_x={\mathbf{V}}_x{({\mathbf{I}}_r-\rho {\mathbf{\Lambda }}_x^{-1})}^{\frac{1}{2}}\text{and}{\mathbf{W}}_y={\mathbf{V}}_y{({\mathbf{I}}_r-\rho {\mathbf{\Lambda }}_y^{-1})}^{\frac{1}{2}}.$$

(66)

The evaluation of the divergence outside the interval 0 < α < 1, or when β ≠ 1 − α, is not covered by this formula and, in general, requires knowledge of the eigenvalues of the matrix ${\tilde{\mathbf{C}}}_y^{-\frac{1}{2}}{\tilde{\mathbf{C}}}_x{\tilde{\mathbf{C}}}_y^{-\frac{1}{2}}$. However, different analyses are necessary depending on the dimension of the intersection of the range space of both covariance matrices C_x and C_y. In the following, we study the two more general scenarios.

Case (A) The range spaces of C_x and C_y are the same.

In this case ${\mathbf{U}}_y^{\perp }{({\mathbf{U}}_y^{\perp })}^T={\mathbf{U}}_x^{\perp }{({\mathbf{U}}_x^{\perp })}^T$ and the eigenvalues of the matrix

$${\tilde{\mathbf{C}}}_x{\tilde{\mathbf{C}}}_y^{-1}=({\mathbf{C}}_x+\rho {\mathbf{U}}_x^{\perp }{({\mathbf{U}}_x^{\perp })}^T)({\mathbf{C}}_y^{+}+{\rho }^{-1}{\mathbf{U}}_x^{\perp }{({\mathbf{U}}_x^{\perp })}^T)$$

(67)

$$={\mathbf{C}}_x{\mathbf{C}}_y^{+}+{\mathbf{U}}_x^{\perp }{({\mathbf{U}}_x^{\perp })}^T$$

(68)

coincide with the nonzero eigenvalues of ${\mathbf{C}}_x{\mathbf{C}}_y^{+}$ except for (n − r) additional eigenvalues which are equal to 1. Then, using the equivalence between (57) and (60), the divergence reduces to the following form

$$D_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=L_{AB}^{(\alpha ,\beta )}({\mathbf{C}}_x\parallel {\mathbf{C}}_y)$$

(69)

$$=D_{AB}^{(\alpha ,\beta )}(({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y){({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{K}}_{yy}^{-1}{\mathbf{V}}_y)}^T\parallel {\mathbf{I}}_r).$$

(70)

Case (B) The range spaces of C_x and C_y are disjoint.

In practice, for n ≫ r this is the most probable scenario. In such a case, the r largest eigenvalues of the matrix ${\tilde{\mathbf{C}}}_x{\tilde{\mathbf{C}}}_y^{-1}$ diverge as ρ tends to zero. Hence, we can not bound above these eigenvalues and, for this reason, it makes no sense to study the case of sign(α) ≠ sign(β), so in this section we assume that sign(α) = sign(β).

Theorem 3. When range spaces of C_x and C_y are disjoint and for a sufficiently small value of ρ > 0, the AB log-det divergence is closely approximated by the formula

$$D_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)\approx D_{AB}^{(\alpha ,\beta )}({\mathbf{C}}_{x|y}^{(\rho )}\parallel \rho {\mathbf{I}}_r)+D_{AB}^{(\beta ,\alpha )}({\mathbf{C}}_{y|x}^{(\rho )}\parallel \rho {\mathbf{I}}_r),$$

(71)

where${\mathbf{C}}_{x|y}^{(\rho )}$ (and respectively${\mathbf{C}}_{y|x}^{(\rho )}$ by interchanging x and y) denotes the matrix

$${\mathbf{C}}_{x|y}^{(\rho )}={\mathbf{\Lambda }}_x-\rho {\mathbf{I}}_r-{\rho }^2{\mathbf{\Lambda }}_y^{-1}-{\mathbf{W}}_x^T{\mathbf{K}}_{xy}{\mathbf{W}}_y{\mathbf{\Lambda }}_y^{-1}{\mathbf{W}}_y^T{\mathbf{K}}_{yx}{\mathbf{W}}_x.$$

(72)

The proof of the theorem is presented in the Appendix G. The eigenvalues of the matrices ${\mathbf{C}}_{x|y}^{(\rho )}$ and ${\mathbf{C}}_{y|x}^{(\rho )}$, estimate the r largest eigenvalues of ${\tilde{\mathbf{C}}}_y^{-\frac{1}{2}}{\tilde{\mathbf{C}}}_x{\tilde{\mathbf{C}}}_y^{-\frac{1}{2}}$ and of its inverse ${\tilde{\mathbf{C}}}_x^{-\frac{1}{2}}{\tilde{\mathbf{C}}}_y{\tilde{\mathbf{C}}}_x^{-\frac{1}{2}}$, respectively. The relative error in the estimation of these eigenvalues is of order O(ρ), i.e., it gradually improves as ρ tend to zero. The approximation is asymptotically exact, and ${\mathbf{C}}_{x|y}^{(\rho )}$ and ${\mathbf{C}}_{x|y}^{(\rho )}$ converge respectively to the conditional covariance matrices

$${\mathbf{C}}_{x|y}=\underset{\rho \to 0}{\lim }{\mathbf{C}}_{x|y}^{(\rho )}={\mathbf{V}}_x^T{\mathbf{K}}_{xx}{\mathbf{V}}_x-({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{V}}_y){({\mathbf{V}}_x^T{\mathbf{K}}_{yy}{\mathbf{V}}_y)}^{-1}{({\mathbf{V}}_x^T{\mathbf{K}}_{xy}{\mathbf{V}}_y)}^T,$$

(73)

$${\mathbf{C}}_{y|x}=\underset{\rho \to 0}{\lim }{\mathbf{C}}_{y|x}^{(\rho )}={\mathbf{V}}_x^T{\mathbf{K}}_{yy}{\mathbf{V}}_y-({\mathbf{V}}_y^T{\mathbf{K}}_{yx}{\mathbf{V}}_x){({\mathbf{V}}_x^T{\mathbf{K}}_{xx}{\mathbf{V}}_x)}^{-1}{({\mathbf{V}}_y^T{\mathbf{K}}_{yx}{\mathbf{V}}_x)}^T,$$

(74)

while ρ I converges to the zero matrix.

In the limit, the value of the divergence is not very useful because

$$\underset{\rho \to 0}{\lim }{D}_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\infty$$

(75)

though there are some practical ways to circumvent this limitation. For example, when α = 0 or β = 0, the divergence can be scaled by a suitable power of ρ to make it finite (see Section 3.3.1 in [32]). The scaled form of the divergence between the regularized covariance matrices is

$${\mathcal{SD}}_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)\equiv \underset{\rho \to 0}{\lim }{\rho }^{\max \{\alpha ,\beta \}}{D}_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y).$$

(76)

Examples of scaled divergences are the following versions of Stein’s losses

$${\mathcal{SD}}_{AB}^{(0,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\underset{\rho \to 0}{\lim }{\rho }^{\beta }{D}_{AB}^{(0,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\frac{1}{{\beta }^2}\mathrm{tr}({({\mathbf{C}}_{x|y})}^{\beta })\ge 0,\beta >0,$$

(77)

$${\mathcal{SD}}_{AB}^{(\alpha ,0)}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\underset{\rho \to 0}{\lim }{\rho }^{\alpha }{D}_{AB}^{(0,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\frac{1}{{\alpha }^2}\mathrm{tr}({({\mathbf{C}}_{y|x})}^{\alpha })\ge 0,\alpha >0,$$

(78)

as well as the Jeffrey’s KL family of symmetric divergences (cf. Equation (23) in [33])

$${\mathcal{SD}}_{ABS1}^{(\alpha ,0)}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\underset{\rho \to 0}{\lim }{\rho }^{\alpha }{D}_{ABS1}^{(\alpha ,0)}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\frac{1}{2{\alpha }^2}(\mathrm{tr}({({\mathbf{C}}_{x|y})}^{\alpha })+\mathrm{tr}({({\mathbf{C}}_{x|y})}^{\alpha }),\alpha >0.$$

(79)

In other cases, when the scaling is not sufficient to obtain a finite and practical dissimilarity measure, an affine transformation may be used. The idea is to identify the divergent part of $D_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)$ as ρ → 0 and use its value as a reference for the evaluation the dissimilarity. For α, β ≥ 0, the relative AB log-det dissimilarity measure is the limiting value of the affine transformation

$${\mathcal{RD}}_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)\equiv \underset{\rho \to 0}{\lim }\min \{\alpha ,\beta \}(D_{ABS1}^{(\alpha ,0)}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)-\frac{r}{\alpha \beta }\log \frac{\alpha \beta {\rho }^{-(\alpha +\beta )}}{{(\alpha +\beta )}^2}),\alpha ,\beta >0.$$

(80)

After its extension by continuity (including as special cases α = 0 or β = 0), the function

$${\mathcal{RD}}_{AB}^{(\alpha ,\beta )}({\tilde{\mathbf{C}}}_x\parallel {\tilde{\mathbf{C}}}_y)=\{\begin{array}{ll}\log \det ({\mathbf{C}}_{x|y})+\frac{\alpha }{\beta }\log \det ({\mathbf{C}}_{y|x})\hfill & \beta >\alpha \ge 0\hfill \\ \log \det ({\mathbf{C}}_{x|y})+\log \det ({\mathbf{C}}_{y|x})\hfill & \alpha =\beta \ge 0\hfill \\ \log \det ({\mathbf{C}}_{y|x})+\frac{\beta }{\alpha }\log \det ({\mathbf{C}}_{x|y})\hfill & \alpha >\beta \ge 0\hfill \end{array}$$

(81)

provides simple formulas to measure the relative dissimilarity between symmetric positive semidefinite matrices C_x and C_y. However, it should be taken into account that, as a consequence of its relative character, this function is not bounded below and can achieve negative values.

7. Modifications and Generalizations of AB Log-Det Divergences and Gamma Matrix Divergences

The divergence (3) discussed in the previous sections can be extended and modified in several ways. It is interesting to note that the positive eigenvalues of PQ⁻¹ play a similar role as the ratios (p_i/q_i) and (q_i/p_i) when used in the wide class of standard discrete divergences, see for example, [11,12]; hence, we can apply such divergences to formulate a modified log-det divergence as a function of the eigenvalues λ_i.

For example, consider the Itakura-Saito distance defined by

$$D_{IS}(\mathit{p}\parallel \mathit{q})={\displaystyle \sum _i\left(\frac{p_i}{q_i}+\log \frac{q_i}{p_i}-1\right).}$$

(82)

It is worth noting that we can generate the large class of divergences or cost functions using Csiszár f-functions [13,24,25]. By replacing p_i=q_i with λ_i and q_i=p_i with we obtain the log-det divergence for SPD matrices:

$$D_{IS}(\mathbf{P}\parallel \mathbf{Q})={\displaystyle \sum _{i=1}^n({\mathrm{\lambda }}_i-\log ({\mathrm{\lambda }}_i))-n,}$$

(83)

which is consistent with (24) and (26).

As another example, consider the discrete Gamma divergence [11,12] defined by

$$\begin{array}{l}{D}_{AC}^{(\alpha ,\beta )}(\mathit{p}\parallel \mathit{q})=\frac{1}{\beta (\alpha +\beta )}\log \left({\displaystyle \sum _i{p}_i^{\alpha +\beta }}\right)+\frac{1}{\alpha (\alpha +\beta )}\log \left({\displaystyle \sum _i{q}_i^{\alpha +\beta }}\right)-\frac{1}{\alpha \beta }\ln \left({\displaystyle \sum _i{p}_i^{\alpha }{q}_i^{\alpha }}\right)\\ =\frac{1}{\alpha \beta (\alpha +\beta )}\log \frac{{\left({\displaystyle \sum _i{p}_i^{\alpha +\beta }}\right)}^{\alpha }{\left({\displaystyle \sum _i{q}_i^{\alpha +\beta }}\right)}^{\beta }}{{\left({\displaystyle \sum _i{p}_i^{\alpha }{q}_i^{\beta }}\right)}^{\alpha +\beta }},\\ \text{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0,\end{array}$$

(84)

which when α = 1 and β → −1, simplifies to the following form [11]:

$$\underset{\beta \to -1}{\lim }{D}_{AC}^{(1,\beta )}(\mathit{p}\parallel \mathit{q})=\frac{1}{n}{\displaystyle \sum _{i=1}^n\left(\log \frac{q_i}{p_i}\right)+\log }\left({\displaystyle \sum _{i=1}^n\frac{p_i}{q_i}}\right)-\log (n)=\log \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n\frac{p_i}{q_i}}}{{\left({\displaystyle \prod _{i=1}^n\frac{p_i}{q_i}}\right)}^{1/n}}.$$

(85)

Hence, by substituting p_i/q_i with λ_i, we derive a new Gamma matrix divergence for SPD matrices:

$$\begin{array}{l}{D}_{CCA}^{(1,0)}(\mathbf{P}\parallel \mathbf{Q})=D_{AC}^{(1,-1)}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{n}{\displaystyle \sum _{i=1}^n\left(\log {\mathrm{\lambda }}_i^{-1}\right)+\log }\left({\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i}\right)-\log (n)\\ =\log \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i}}{{\left({\displaystyle \prod _{i=1}^n{\mathrm{\lambda }}_i}\right)}^{1/n}}=\log \frac{\left|M_1\right\{{\mathrm{\lambda }}_i\}}{M_0\left\{{\mathrm{\lambda }}_i\right\}},\end{array}$$

(86)

where M₁ denotes the arithmetic mean, and M₀ denotes the geometric mean.

Interestingly, (86) can be expressed equivalently as

$$D_{CCA}^{(1,0)}(\mathbf{P}\parallel \mathbf{Q})=\log (\mathrm{tr}(\mathbf{P}{\mathbf{Q}}^{-1}))-\frac{1}{n}\log \det (\mathbf{P}{\mathbf{Q}}^{-1})-\log (n).$$

(87)

Similarly, using the symmetric Gamma divergence defined in [11,12],

$$\begin{array}{l}{D}_{ACS}^{(\alpha ,\beta )}(\mathit{p}\parallel \mathit{q})=\frac{1}{\alpha \beta }\log \frac{\left({\displaystyle \sum _i{p}_i^{\alpha +\beta }}\right)\left({\displaystyle \sum _i{q}_i^{\alpha +\beta }}\right)}{\left({\displaystyle \sum _i{p}_i^{\alpha }{q}_i^{\beta }}\right)\left({\displaystyle \sum _i{p}_i^{\beta }{q}_i^{\alpha }}\right)},\\ \text{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0,\end{array}$$

(88)

for α = 1 and β → −1 and by substituting the ratios p_i/q_i with λ_i, we obtain a new Gamma matrix divergence as follows:

$$\begin{array}{l}{D}_{ACS}^{(1,-1)}(\mathbf{P}\parallel \mathbf{Q})=\log \left(({\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i})({\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i^{-1}})\right)-\log {(n)}^2\\ =\log \left((\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i})(\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i^{-1}})\right)\\ =\log (M_1\{{\mathrm{\lambda }}_i\left\}{M}_1\right\{{\mathrm{\lambda }}_i^{-1}\})\end{array}$$

(89)

$$=\log \frac{M_1\left\{{\mathrm{\lambda }}_i\right\}}{M_{-1}\left\{{\mathrm{\lambda }}_i\right\}},$$

(90)

where M₋₁ {λ_i} denotes the harmonic mean.

Note that for n → ∞, this formulated divergence can be expressed compactly as

$$D_{ACS}^{(1,-1)}(\mathbf{P}\parallel \mathbf{Q})=\log (E\{\mathit{u}\left\}E\right\{{\mathit{u}}^{-1}\}),$$

(91)

where u_i = {λ_i} and $u_i^{-1}=\left\{{\mathrm{\lambda }}_i^{-1}\right\}$.

The basic means are defined as follows:

$$M_{\gamma }(\mathrm{\lambda })=\{\begin{array}{ll}{M}_{-\infty }=\min \{{\mathrm{\lambda }}_1,\dots ,{\mathrm{\lambda }}_n\},\hfill & \gamma \to -\infty ,\hfill \\ M_{-1}=n{\left({\displaystyle \sum _{i=1}^n\frac{1}{{\mathrm{\lambda }}_i}}\right)}^{-1},\hfill & \gamma =-1,\hfill \\ M_0=\left({\displaystyle \prod _{i=1}^n{\mathrm{\lambda }}_i}\right),\hfill & \gamma =0,\hfill \\ M_1=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i},\hfill & \gamma =1,\hfill \\ M_2={\left(\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{\lambda }}_i^2}\right)}^{1/2},\hfill & \gamma =2,\hfill \\ M_{\infty }=\max \{{\mathrm{\lambda }}_1,\dots ,{\mathrm{\lambda }}_n\},\hfill & \gamma =\infty ,\hfill \end{array}$$

(92)

with

$$M_{-\infty }\le M_{-1}\le M_0\le M_1\le M_2\le M_{\infty },$$

(93)

where equality holds only if all λ_i are equal. By increasing the values of γ, more emphasis is put on large relative errors, i.e., on λ_i whose values are far from one. Depending on the value of γ, we obtain the minimum entry of the vector λ (for γ → −∞), its harmonic mean (γ = −1), the geometric mean (γ = 0), the arithmetic mean (γ = 1), the quadratic mean (γ = 2), and the maximum entry of the vector (γ → ∞).

Exploiting the above inequalities for the means, the divergences in (86) and (90) can be heuristically generalized (defined) as follows:

$$D_{CCA}^{({\gamma }_2,{\gamma }_1)}(\mathbf{P}\parallel \mathbf{Q})=\log \frac{M_{{\gamma }_2}\{{\mathrm{\lambda }}_i\}}{M_{{\gamma }_1}\{{\mathrm{\lambda }}_i\}},$$

(94)

for γ₂ > γ₁.

The new divergence in (94) is quite general and flexible, and in extreme cases, it takes the following form:

$$D_{CCA}^{(\infty ,-\infty )}(\mathbf{P}\parallel \mathbf{Q})=d_H(\mathbf{P}\parallel \mathbf{Q})=\log \frac{M_{\infty }\left\{{\mathrm{\lambda }}_i\right\}}{M_{-\infty }\left\{{\mathrm{\lambda }}_i\right\}}=\log \frac{{\mathrm{\lambda }}_{max}}{{\mathrm{\lambda }}_{min}},$$

(95)

which is, in fact, a well-known Hilbert projective metric [6,26].

The Hilbert projective metric is extremely simple and suitable for big data because it requires only two (minimum and maximum) eigenvalue computations of the matrix PQ⁻¹.

The Hilbert projective metric satisfies the following important properties [6,27]:

• Nonnegativity, d_H(P ║ Q) ≥ 0, and definiteness, d_H(P ║ Q) = 0, if and only if there exists a c > 0 such that Q = cP.
• Invariance to scaling:

  $$d_H(c_1\mathbf{P}\parallel c_2\mathbf{Q})=d_H(\mathbf{P}\parallel \mathbf{Q}),$$

  (96)

  for any c₁, c₂ > 0.
• Symmetry:

  $$d_H(\mathbf{P}\parallel \mathbf{Q})=d_H(\mathbf{Q}\parallel \mathbf{P}).$$

  (97)
• Invariance under inversion:

  $$d_H(\mathbf{P}\parallel \mathbf{Q})=d_H({\mathbf{P}}^{-1}\parallel {\mathbf{Q}}^{-1}).$$

  (98)
• Invariance under congruence transformations:

  $$d_H(\mathbf{AP}{\mathbf{A}}^T\parallel \mathbf{AQ}{\mathbf{A}}^T)=d_H(\mathbf{P}\parallel \mathbf{Q}),$$

  (99)

  for any invertible matrix A.
• Invariance under geodesic (Riemannian) transformations (by taking A = P^−1/2 in (99)):

  $$d_H(\mathbf{I}\parallel {\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})=d_H(\mathbf{P}\parallel \mathbf{Q}).$$

  (100)
• Separability of divergence for the Kronecker product of SPD matrices:

  $$d_H({\mathbf{P}}_1\otimes {\mathbf{P}}_2\parallel {\mathbf{Q}}_1\otimes {\mathbf{Q}}_2)=d_H({\mathbf{P}}_1\parallel {\mathbf{Q}}_1)+d_H({\mathbf{P}}_2\parallel {\mathbf{Q}}_2).$$

  (101)
• Scaling of power of SPD matrices:

  $$d_H({\mathbf{P}}^{\omega }\parallel {\mathbf{Q}}^{\omega })=\left|\omega \right|d_H(\mathbf{P}\parallel \mathbf{Q}),$$

  (102)

  for any ω ≠ 0.
  Hence, for 0 < |ω₁| ≤ 1 ≤ |ω₂| we have

  $$d_H({\mathbf{P}}^{{\omega }_1}\parallel {\mathbf{Q}}^{{\omega }_1})\le d_H(\mathbf{P}\parallel \mathbf{Q})\le d_H({\mathbf{P}}^{{\omega }_2}\parallel {\mathbf{Q}}^{{\omega }_2}).$$

  (103)
• Scaling under the weighted geometric mean:

  $$d_H(\mathbf{P}{\#}_s\mathbf{Q}\parallel \mathbf{P}{\#}_u\mathbf{Q})=\left|s-u\right|d_H(\mathbf{P}\parallel \mathbf{Q}),$$

  (104)

  for any u, s ≠ 0, where

  $$\mathbf{P}{\#}_u\mathbf{Q}={\mathbf{P}}^{1/2}{({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^u{\mathbf{P}}^{1/2}.$$

  (105)
• Triangular inequality: $d_H(\mathbf{P}\parallel \mathbf{Q})\le d_H(\mathbf{P}\parallel \mathbf{Z})+d_H(\mathbf{Z}\parallel \mathbf{Q})$.

These properties can easily be derived and verified. For example, property (9) can easily be derived as follows [6,27]:

$$\begin{array}{l}{d}_H(\mathbf{P}{\#}_s\mathbf{Q}\parallel \mathbf{P}{\#}_u\mathbf{Q})=d_H({\mathbf{P}}^{1/2}{({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^s{\mathbf{P}}^{1/2}\parallel ({\mathbf{P}}^{1/2}{({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^u{\mathbf{P}}^{1/2})\\ =d_H({({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^s\parallel {({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^u)\\ =d_H({({\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2})}^{(s-u)}\parallel \mathbf{I})\\ =\left|s-u\right|d_H(\mathbf{P}\parallel \mathbf{Q}).\end{array}$$

(106)

In

Table 2. Comparison of the fundamental properties of three basic metric distances: the Riemannian (geodesic) metric (19), LogDet Zero (Bhattacharyya) divergence (21), and the Hilbert projective metric (95). Matrices P, Q, P₁, P₂, Q₁, Q₂, Z ∈ ℝ^n×n are SPD matrices, A ∈ ℝ^n×n is nonsingular, and the matrix X ∈ ℝ^n×r with r < n is full column rank. The scalars satisfy the following conditions: c > 0, c₁, c₂ > 0; 0 < ω ≤ 1, s, u ≠ 0, ψ = |s − u|. The geometric means are defined by P#_uQ = P^1/2(P^−1/2QP^−1/2)^u P^1/2 and P#Q = P#_1/2Q = P^1/2(P^−1/2QP^−1/2)^1/2 P^1/2. The Hadamard product of P and Q is denoted by P ○ Q (cf. with [6]).

Riemannian (geodesic) metric	LogDet Zero (Bhattacharyya) div.	Hilbert projective metric
dR(P║Q) = ║log(Q−1/2PQ−1/2)║F	$d_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})=2\sqrt{\log \frac{\det \frac{1}{2}(\mathbf{P}+\mathbf{Q})}{\det (\mathbf{P})\det (\mathbf{Q})}}$	$d_H(\mathbf{P}\parallel \mathbf{Q})=\log \frac{{\mathrm{\lambda }}_{max}\left\{\mathbf{P}{\mathbf{Q}}^{-1}\right\}}{{\mathrm{\lambda }}_{min}\left\{\mathbf{P}{\mathbf{Q}}^{-1}\right\}}$
dR(P║Q) = dR(Q ║ P)	dBh(P║Q) = dBh(Q ║ P)	dH(P║Q) = dH(Q ║ P)
dR(cP ║ cQ) = dR(P ║ Q)	dBh(cP ║ cQ) = dBh(P ║Q)	dH(c1P ║ c2Q) = dH(P ║Q)
dR(APAT║AQAT) = dR(P ║Q)	dBh(APAT║AQAT) = dBh(P ║Q)	dH(APAT║AQAT) = dH(P ║Q)
dR(P−1║Q−1) = dR(P ║ Q)	dBh (P−1║Q−1) dBh (P ║ Q)	dH (P−1║Q−1) = dH (P ║ Q)
dR (Pω ║ Qω) ≤ ω dR(P ║ Q)	$d_{\mathrm{Bh}}({\mathbf{P}}^{\omega }\parallel {\mathbf{Q}}^{\omega })\le \sqrt{\omega }{d}_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})$	dH (Pω ║ Qω) ≤ ω dH (P ║ Q)
dR(P ║ P#ωQ) = ω dR(P ║ Q)	$d_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{P}{\#}_{\omega }\mathbf{Q})\le \sqrt{\omega }{d}_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})$	dH (P ║ P#ωQ) = ω dH (P ║ Q)
dR (Z#ωP ║ Z#ωQ) ≤ ω dR (P ║ Q)	$d_{\mathrm{Bh}}(\mathbf{Z}{\#}_{\omega }\mathbf{P}\parallel \mathbf{Z}{\#}_{\omega }\mathbf{Q})\le \sqrt{\omega }{d}_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})$	dH (Z#ωP ║ Z#ωQ) ≤ ω dH (P ║ Q)
dR (P#sQ ║ P#uQ) = ψ dR (P ║ Q))	$d_{\mathrm{Bh}}(\mathbf{P}{\#}_s\mathbf{Q}\parallel \mathbf{P}{\#}_u\mathbf{Q})\le \sqrt{\psi }{d}_{\mathrm{Bh}}(\mathbf{P}\parallel \mathbf{Q})$	dH (P#sQ ║ P#uQ) = ψ dH (P ║ Q)
dR (P ║ P#Q) = dR (Q ║ P# Q)	dBh (P ║ P#Q) = dR (Q ║ P# Q)	dH (P ║ P#Q) = dH (Q ║ P# Q)
dR (XTPX ║ XTQX) ≤ dR (P ║ Q)	dBh (XTPX ║ XTQX) ≤ dBh (P ║ Q)	dH (XTPX ║ XTQX) ≤ dH (P ║ Q)
$d_R(\mathbf{Z}\otimes \mathbf{P}\parallel \mathbf{Z}\otimes \mathbf{Q})=\sqrt{n}{d}_R(\mathbf{P}\parallel \mathbf{Q})$	$d_{Bh}(\mathbf{Z}\otimes \mathbf{P}\parallel \mathbf{Z}\otimes \mathbf{Q})=\sqrt{n}{d}_{Bh}(\mathbf{P}\parallel \mathbf{Q})$	dH (Z ⊗ P ║ Z ⊗ Q) = dH (P ║ Q)
$\begin{array}{l}{d}_R^2({\mathbf{P}}_1\otimes {\mathbf{P}}_2\parallel {\mathbf{Q}}_1\otimes {\mathbf{Q}}_2)=\\ =n{d}_R^2({\mathbf{P}}_1\parallel {\mathbf{Q}}_1)+n{d}_R^2({\mathbf{P}}_2\parallel {\mathbf{Q}}_2)+\\ 2\log \det ({\mathbf{P}}_1{\mathbf{Q}}_1^{-1})\log \det ({\mathbf{P}}_2{\mathbf{Q}}_2^{-1})\end{array}$	dBh (P1⊗ P2 ║ Q1⊗ Q2) ≥ dBh (P1○ P2 ║ Q1○ Q2)	dH(P1⊗ P2 ║ Q1⊗ Q2) = dH (P1║ Q1) +dH (P2║ Q2)

, we summarize and compare some fundamental properties of three important metric distances: the Hilbert projective metric, Riemannian metric, and LogDet Zero (Bhattacharyya) distance. Since some of these properties are new, we refer to [6,27,28].

7.1. The AB Log-Det Divergence for Noisy and Ill-Conditioned Covariance Matrices

In real-world signal processing and machine learning applications, the SPD sampled matrices can be strongly corrupted by noise and extremely ill conditioned. In such cases, the eigenvalues of the generalized eigenvalue (GEVD) problem Pv_i = λ_iQv_i can be divided into a signal subspace and noise subspace. The signal subspace is usually represented by the largest eigenvalues (and corresponding eigenvectors), and the noise subspace is usually represented by the smallest eigenvalues (and corresponding eigenvectors), which should be rejected; in other words, in the evaluation of log-det divergences, only the eigenvalues that represent the signal subspace should be taken into account. The simplest approach is to find the truncated dominant eigenvalues by applying the suitable threshold τ > 0; equivalently, find an index r ≤ n for which λ_r+1 ≤ τ and perform a summation. For example, truncation reduces the summation in (8) from 1 to r (instead of 1 to n) [22]. The threshold parameter τ can be selected via cross validation.

Recent studies suggest that the real signal subspace covariance matrices can be better represented by truncating the eigenvalues. A popular and relatively simple method applies a thresholding and shrinkage rule to the eigenvalues [35]:

$${\tilde{\mathrm{\lambda }}}_i={\mathrm{\lambda }}_i\max \{(1-\frac{{\tau }^{\gamma }}{{\mathrm{\lambda }}^{\gamma }}),0\},$$

(107)

where any eigenvalue smaller than the specific threshold is set to zero, and the remaining eigenvalues are shrunk. Note that the smallest eigenvalues are shrunk more than the largest one. For γ = 1, we obtain a standard soft thresholding, and for γ → ∞ a standard hard thresholding is obtained [36]. The optimal threshold τ > 0 can be estimated along with the parameter γ > 0 using cross validation. However, a more practical and efficient method is to apply the Generalized Stein Unbiased Risk Estimate (GSURE) method even if the variance of the noise is unknown (for more details, we refer to [35] and the references therein).

In this paper, we propose an alternative approach in which the bias generated by noise is reduced by suitable choices of α and β [12]. Instead of using the eigenvalues λ_i of PQ⁻¹ or its inverse, we use regularized or shrinked eigenvalues [35–37]. For example, in light of (8), we can use the following shrinked eigenvalues:

$${\tilde{\mathrm{\lambda }}}_i={\left(\frac{\alpha {\mathrm{\lambda }}_i^{\beta }+\beta {\mathrm{\lambda }}_i^{-\alpha }}{\alpha +\beta }\right)}^{\frac{1}{\alpha \beta }}\ge 1,\text{for}\alpha ,\beta \ne 0,\alpha ,\beta >0\text{or}\alpha ,\beta <0,$$

(108)

which play a similar role as the ratios (p_i/q_i) (p_i ≥ q_i), which are used in the standard discrete divergences [11,12]. It should be noted that equalities ${\tilde{\mathrm{\lambda }}}_i=1$, ∀i hold only if all λ_i of PQ⁻¹ are equal to one, which occurs only if P = Q. For example, the new Gamma divergence in (94) can be formulated even more generally as

$$D_{CCA}^{({\gamma }_2,{\gamma }_1)}(\mathbf{P}\parallel \mathbf{Q})=\log \frac{M_{{\gamma }_2}\{{\tilde{\mathrm{\lambda }}}_i\}}{M_{{\gamma }_1}\{{\tilde{\mathrm{\lambda }}}_i\}},$$

(109)

where γ₂ > γ₁, and ${\tilde{\mathrm{\lambda }}}_i$ are the regularized or optimally shrinked eigenvalues.

8. Divergences of Multivariate Gaussian Densities and Differential Relative Entropies of Multivariate Normal Distributions

In this section, we show the links or relationships between a family of continuous Gamma divergences and AB log-det divergences for multivariate Gaussian densities.

Consider the two multivariate Gaussian (normal) distributions:

$$p(\mathit{x})=\frac{1}{\sqrt{{(2\pi )}^n\det \mathbf{P}}}\exp \left(-\frac{1}{2}{(\mathit{x}-{\mathit{\mu }}_1)}^T{\mathbf{P}}^{-1}(\mathit{x}-{\mathit{\mu }}_1)\right),$$

(110)

$$q(\mathit{x})=\frac{1}{\sqrt{{(2\pi )}^n\det \mathbf{Q}}}\exp \left(-\frac{1}{2}{(\mathit{x}-{\mathit{\mu }}_2)}^T{\mathbf{Q}}^{-1}(\mathit{x}-{\mathit{\mu }}_2)\right),x\in {ℝ}^n,$$

(111)

where µ₁ ∈ ℝⁿ and µ₂ ∈ ℝⁿ are mean vectors, and P = Σ₁ ∈ ℝ^n×n and Q = Σ₂ ∈ ℝ^n×n are the covariance matrices of p(x) and q(x), respectively.

Furthermore, consider the Gamma divergence for these distributions:

$$\begin{array}{l}{D}_{AC}^{(\alpha ,\beta )}(p(\mathit{x})\parallel q(\mathit{x}))=\frac{1}{\alpha \beta (\alpha +\beta )}\log \frac{{\left({\displaystyle {\int }_{\mathrm{\Omega }}{p}^{\alpha +\beta }(\mathit{x})d\mathit{x}}\right)}^{\alpha }{\left({\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\alpha +\beta }(\mathit{x})d\mathit{x}}\right)}^{\beta }}{{\left({\displaystyle {\int }_{\mathrm{\Omega }}{p}^{\alpha }(\mathit{x})q^{\beta }(\mathit{x})d\mathit{x}}\right)}^{\alpha +\beta }}\\ \text{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0,\end{array}$$

(112)

which generalizes a family of Gamma divergences [11,12].

Theorem 4. The Gamma divergence in (112) for multivariate Gaussian densities (110) and (111) can be expressed in closed form as follows:

$$\begin{array}{l}{D}_{AC}^{(\alpha ,\beta )}(p(\mathit{x})\parallel q(\mathit{x}))=\frac{1}{2}{D}_{AB}^{(\beta ,\alpha )}\left({({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{-1/2})}^{\frac{1}{\alpha +\beta }}\parallel \mathbf{I}\right)+\frac{1}{2}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{(\alpha \mathbf{Q}+\beta \mathbf{P})}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2),\\ =\frac{1}{2\alpha \beta }\log \frac{\det \left(\frac{\alpha }{\alpha +\beta }\mathbf{Q}+\frac{\beta }{\alpha +\beta }\mathbf{P}\right)}{\det {(\mathbf{Q})}^{\frac{\alpha }{\alpha +\beta }}\det {(\mathbf{P})}^{\frac{\beta }{\alpha +\beta }}}\\ +\frac{1}{2(\alpha +\beta )}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{\left(\frac{\alpha }{a+\beta }\mathbf{Q}+\frac{\beta }{\alpha +\beta }\mathbf{P}\right)}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2),\end{array}$$

(113)

for α > 0 and β > 0.

The proof is provided in Appendix H. Note that for α + β = 1, the first term in the right-hand-side of (113) also simplifies as

$${\frac{1}{2}{D}_{AB}^{(\beta ,\alpha )}\left({({\mathbf{Q}}^{-1/2}\mathbf{P}{\mathbf{Q}}^{1/2})}^{\frac{1}{\alpha +\beta }}\parallel \mathbf{I}\right)|}_{{}_{\beta =1-\alpha }}=\frac{1}{2}{D}_{AB}^{(1-\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}{D}_A^{(1-\alpha )}(\mathbf{P}\parallel \mathbf{Q}).$$

(114)

Observe that Formula (113) consists of two terms: the first term is expressed via the AB log-det divergence, which measures the similarity between two covariance or precision matrices and is independent from the mean vectors, while the second term is a quadratic form expressed by the Mahalanobis distance, which represents the distance between the means (weighted by the covariance matrices) of multivariate Gaussian distributions. Note that the second term is zero when the mean values µ₁ and µ₂ coincide.

Theorem 4 is a generalization of the following well-known results:

• For α = 1 and β = 0 and as β → 0, the Kullback-Leibler divergence can be expressed as [5,38]

  $$\begin{array}{l}\underset{\beta \to 0}{\lim }{D}_{AC}^{(1,\beta )}(p(\mathit{x})\parallel q(\mathit{x}))=D_{KL}(p(\mathit{x})\parallel q(\mathit{x}))={\displaystyle {\int }_{\mathrm{\Omega }}p(\mathit{x})\log \frac{p(\mathit{x})}{q(\mathit{x})}d\mathit{x}}\\ =\frac{1}{2}(\mathrm{tr}(\mathbf{P}{\mathbf{Q}}^{-1})-\log \det (\mathbf{P}{\mathbf{Q}}^{-1})-n)+\frac{1}{2}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{\mathbf{Q}}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2),\end{array}$$

  (115)

  where the last term represents the Mahalanobis distance, which becomes zero for zero-mean distributions µ₁ = µ₂ = 0.
• For α = β = 0.5 we have the Bhattacharyya distance [5,39]

  $$\begin{array}{l}{D}_{AC}^{(0.5,0.5)}(p(\mathit{x})\parallel q(\mathit{x}))=\frac{1}{2}{d}_{Bh}^2(p(\mathit{x})\parallel q(\mathit{x}))=-4\log {\displaystyle {\int }_{\mathrm{\Omega }}\sqrt{p(\mathit{x})q(\mathit{x})d\mathit{x}}}\\ =2\log \frac{\det \frac{\mathbf{P}+\mathbf{Q}}{2}}{\sqrt{\det \mathbf{P}\det \mathbf{Q}}}+\frac{1}{2}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{\left[\frac{\mathbf{P}+\mathbf{Q}}{2}\right]}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2),\end{array}$$

  (116)
• For α + β = 1 and 0 < α < 1, the closed form expression for the Rényi divergence is obtained [5,32,40]:

  $$\begin{array}{l}{D}_A(p\parallel q)=-\frac{1}{\alpha (1-\alpha )}\log {\displaystyle {\int }_{\mathrm{\Omega }}{p}^{\alpha }(\mathit{x})q^{1-\alpha }(\mathit{x})d\mathit{x}}\\ =\frac{1}{2\alpha (1-\alpha )}\log \frac{\det (\alpha \mathbf{Q}+(1-\alpha )\mathbf{P})}{\det ({\mathbf{Q}}^{\alpha }{\mathbf{P}}^{1-\alpha })}+\frac{1}{2}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{[\alpha \mathbf{Q}+(1-\alpha )\mathbf{P}]}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2).\end{array}$$

  (117)
• For α = β = 1, the Gamma-divergences reduce to the Cauchy-Schwartz divergence:

  $$\begin{array}{l}{D}_{CS}(p(x)\parallel q(x))=-\log \frac{{\displaystyle \int p(\mathit{x})q(\mathit{x})d\mu (\mathit{x})}}{{\left({\displaystyle \int p^2(\mathit{x})d\mu (\mathit{x})}\right)}^{1/2}{\left({\displaystyle \int q^2(\mathit{x})d\mu (\mathit{x})}\right)}^{1/2}}\\ =\frac{1}{2}\log \frac{\det \frac{\mathbf{P}+\mathbf{Q}}{2}}{\sqrt{\det \mathbf{Q}\det \mathbf{P}}}+\frac{1}{4}{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^T{\left(\frac{\mathbf{P}+\mathbf{Q}}{2}\right)}^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2).\end{array}$$

  (118)

Similar formulas can be derived for the symmetric Gamma divergence for two multivariate Gaussian distributions. Furthermore, analogous expressions can be derived for Elliptical Gamma distributions (EGD) [41], which facilitate more flexible modeling than standard multivariate Gaussian distributions.

8.1. Multiway Divergences for Multivariate Normal Distributions with Separable Covariance Matrices

Recently, there has been growing interest in the analysis of tensors or multiway arrays [42–45]. One of the most important applications of multiway tensor analysis and multilinear distributions, is magnetic resonance imaging (MRI) (we refer to [46] and the references therein). For multiway arrays, we often use multilinear (array or tensor) normal distributions that correspond to the multivariate normal (Gaussian) distributions in (110) and (111) with common means µ₁ = µ₂ and separable (Kronecker structured) covariance matrices:

$$\overline{\mathbf{P}}={\sigma }_P^2({\mathbf{P}}_1\otimes {\mathbf{P}}_2\otimes \cdots \otimes {\mathbf{P}}_K)\in {ℝ}^{N\times N}$$

(119)

$$\overline{\mathbf{Q}}={\sigma }_P^2({\mathbf{Q}}_1\otimes {\mathbf{Q}}_2\otimes \cdots \otimes {\mathbf{Q}}_K)\in {ℝ}^{N\times N},$$

(120)

where ${\mathbf{P}}_k=\in {ℝ}^{n_k\times n_k}$ and ${\mathbf{Q}}_k=\in {ℝ}^{n_k\times n_k}$ for k = 1, 2, …, K are SPD matrices, usually normalized so that det P_k = det Q_k = 1 for each k and $N={\prod }_{k=1}^K{n}_k$ [45].

One of the main advantages of the separable Kronecker model is the significant reduction in the number of variance-covariance parameters [42]. Usually, such separable covariance matrices are sparse and very large-scale. The challenge is to design an efficient and relatively simple dissimilarity measure for big data between two zero-mean multivariate (or multilinear) normal distributions ((110) and (111)). Because of its unique properties, the Hilbert projective metric is a good candidate; in particular, for separable Kronecker structured covariances, it can be expressed very simply as

$$D_H(\overline{\mathbf{P}}\parallel \overline{\mathbf{Q}})={\displaystyle \sum _{k=1}^K{D}_H}({\mathbf{P}}_k\parallel {\mathbf{Q}}_k)={\displaystyle \sum _{k=1}^K\log }\frac{{\tilde{\mathrm{\lambda }}}_{max}^{(k)}}{{\tilde{\mathrm{\lambda }}}_{min}^{(k)}}=\log {\displaystyle \prod _{k=1}^K\left(\frac{{\tilde{\mathrm{\lambda }}}_{max}^{(k)}}{{\tilde{\mathrm{\lambda }}}_{min}^{(k)}}\right)},$$

(121)

where ${\tilde{\mathrm{\lambda }}}_{max}^{(k)}$ and ${\tilde{\mathrm{\lambda }}}_{min}^{(k)}$ are the (shrinked) maximum and minimum eigenvalues of the (relatively small) matrices ${\mathbf{P}}_k{\mathbf{Q}}_k^{-1}$ for k = 1, 2, …, K, respectively. We refer to this divergence as the multiway Hilbert metric. This metric has many attractive properties, especially invariance under multilinear transformations.

Using the fundamental properties of divergence and SPD matrices, we derive other multiway log-det divergences. For example, the multiway Stein’s loss can be obtained:

$$\begin{array}{l}{D}_{MSL}(\overline{\mathbf{P}},\overline{\mathbf{Q}})=2D_{KL}(p(\mathit{x})\parallel q(\mathit{x}))=D_{AB}^{(0,1)}(\overline{\mathbf{P}}\parallel \overline{\mathbf{Q}})\\ =\mathrm{tr}(\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})-\log \det (\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})-N\end{array}$$

(122)

$$=\frac{{\sigma }_P^2}{{\sigma }_Q^2}\left({\displaystyle \prod _{k=1}^K\mathrm{tr}({\mathbf{P}}_k{\mathbf{Q}}_k^{-1})}\right)-{\displaystyle \sum _{k=1}^K\frac{N}{n_k}}\log \det ({\mathbf{P}}_k{\mathbf{Q}}_k^{-1})-N\log \left(\frac{{\sigma }_P^2}{{\sigma }_Q^2}\right)-N.$$

(123)

Note that under the constraint that det P_k = det Q_k = 1, this simplifies to

$$\begin{array}{l}{D}_{MSL}(\overline{\mathbf{P}}\parallel \overline{\mathbf{Q}})=\mathrm{tr}(\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})-\log \det (\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})-N\\ =\frac{{\sigma }_P^2}{{\sigma }_Q^2}\left({\displaystyle \prod _{k=1}^K\mathrm{tr}({\mathbf{P}}_k{\mathbf{Q}}_k^{-1})}\right)-N\log \left(\frac{{\sigma }_P^2}{{\sigma }_Q^2}\right)-N,\end{array}$$

(124)

which is different from the multiway Stein’s loss recently proposed by Gerard and Hoff [45].

Similarly, if det P_k = det Q_k = 1 for each k = 1, 2, …, K, we can derive the multiway Riemannian metric as follows:

$$D_R^2(\overline{\mathbf{P}}\parallel \overline{\mathbf{Q}})=N{\log }^2\frac{{\sigma }_P^2}{{\sigma }_Q^2}+{\displaystyle \sum _{k=1}^K\frac{N}{n_k}}{D}_R^2({\mathbf{P}}_k\parallel {\mathbf{Q}}_k).$$

(125)

The above multiway divergences are derived using the following properties:

$$\begin{array}{l}\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1}=({\mathbf{P}}_1\otimes {\mathbf{P}}_2\otimes \cdots \otimes {\mathbf{P}}_K)({\mathbf{Q}}_1^{-1}\otimes {\mathbf{Q}}_2^{-1}\otimes \cdots \otimes {\mathbf{Q}}_K^{-1})\\ ={\mathbf{P}}_1{\mathbf{Q}}_1^{-1}\otimes {\mathbf{P}}_2{\mathbf{Q}}_2^{-1}\otimes \cdots \otimes {\mathbf{P}}_K{\mathbf{Q}}_K^{-1},\end{array}$$

(126)

$$\mathrm{tr}(\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})=\mathrm{tr}({\mathbf{P}}_1{\mathbf{Q}}_1^{-1}\otimes {\mathbf{P}}_2{\mathbf{Q}}_2^{-1}\otimes \cdots \otimes {\mathbf{P}}_K{\mathbf{Q}}_K^{-1})={\displaystyle \prod _{k=1}^K\mathrm{tr}({\mathbf{P}}_K{\mathbf{Q}}_K^{-1})},$$

(127)

$$\det (\overline{\mathbf{P}}{\overline{\mathbf{Q}}}^{-1})=\det ({\mathbf{P}}_1{\mathbf{Q}}_1^{-1}\otimes {\mathbf{P}}_2{\mathbf{Q}}_2^{-1}\otimes \cdots \otimes {\mathbf{P}}_K{\mathbf{Q}}_K^{-1})={\displaystyle \prod _{k=1}^K{(\det ({\mathbf{P}}_K{\mathbf{Q}}_K^{-1}))}^{N/n_k}}.$$

(128)

and the basic property: If the eigenvalues {λ_i} and {θ_j} are eigenvalues with corresponding eigenvectors {v_i} and {u_j} for SPD matrices A and B, respectively, then A ⊗ B has eigenvalues {λ_iθ_j} with corresponding eigenvectors {v_i ⊗ u_j}.

Other possible extensions of the AB and Gamma matrix divergences to separable multiway divergences for multilinear normal distributions under additional constraints and normalization conditions will be discussed in future works.

9. Conclusions

In this paper, we presented novel (dis)similarity measures; in particular, we considered the Alpha-Beta and Gamma log-det divergences (and/or their square-roots) that smoothly connect or unify a wide class of existing divergences for SPD matrices. We derived numerous results that uncovered or unified theoretic properties and qualitative similarities between well-known divergences and new divergences. The scope of the results presented in this paper is vast, especially since the parameterized Alpha-Beta and Gamma log-det divergence functions include several efficient and useful divergences, including those based on relative entropies, the Riemannian metric (AIRM), S-divergence, generalized Jeffreys KL (KLDM), Stein’s loss, and Hilbert projective metric. Various links and relationships between divergences were also established. Furthermore, we proposed several multiway log-det divergences for tensor (array) normal distributions.