Two Measures of Dependence

Abstract

Two families of dependence measures between random variables are introduced. They are based on the Rényi divergence of order $\alpha$ and the relative $\alpha$-entropy, respectively, and both dependence measures reduce to Shannon’s mutual information when their order $\alpha$ is one. The first measure shares many properties with the mutual information, including the data-processing inequality, and can be related to the optimal error exponents in composite hypothesis testing. The second measure does not satisfy the data-processing inequality, but appears naturally in the context of distributed task encoding.

1. Introduction

The solutions to many information-theoretic problems can be expressed using Shannon’s information measures such as entropy, relative entropy, and mutual information. Other problems require Rényi’s information measures, which generalize Shannon’s. In this paper, we analyze two Rényi measures of dependence, $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$, between random variables X and Y taking values in the finite sets $\mathcal{X}$ and $\mathcal{Y}$, with $\alpha \in [0,\infty ]$ being a parameter. (Our notation is similar to the one used for the mutual information: technically, $J_{\alpha }(·)$ and $K_{\alpha }(·)$ are functions not of X and Y, but of their joint probability mass function (PMF) $P_{XY}$.) For $\alpha \in [0,\infty ]$, we define $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ as

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& \triangleq \underset{(Q_X,Q_Y)\in \mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& \triangleq \underset{(Q_X,Q_Y)\in \mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)}{min}{\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y),\hfill \end{array}$$

(2)

where $\mathcal{P}\left(\mathcal{X}\right)$ and $\mathcal{P}\left(\mathcal{Y}\right)$ denote the set of all PMFs over $\mathcal{X}$ and $\mathcal{Y}$, respectively; $D_{\alpha }(P\parallel Q)$ denotes the Rényi divergence of order $\alpha$ (see (50) ahead); and ${\Delta }_{\alpha }(P\parallel Q)$ denotes the relative $\alpha$-entropy (see (55) ahead). As shown in Proposition 7, $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ are in fact closely related.

The measures $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ have the following operational meanings (see Section 3): $J_{\alpha }(X;Y)$ is related to the optimal error exponents in testing whether the observed independent and identically distributed (IID) samples were generated according to the joint PMF $P_{XY}$ or an unknown product PMF; and $K_{\alpha }(X;Y)$ appears as a penalty term in the sum-rate constraint of distributed task encoding.

The measures $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ share many properties with Shannon’s mutual information [1], and both are equal to the mutual information when $\alpha$ is one. Except for some special cases, we have no closed-form expressions for $J_{\alpha }(X;Y)$ or $K_{\alpha }(X;Y)$. As illustrated in Figure 1, unless $\alpha$ is one, the minimum in the definitions of $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ is typically not achieved by $Q_X=P_X$ and $Q_Y=P_Y$. (When $\alpha$ is one, then the minimum is always achieved by $Q_X=P_X$ and $Q_Y=P_Y$; this follows from Proposition 8 and the fact that $D_1(P_{XY}\parallel Q_X{Q}_Y)={\Delta }_1(P_{XY}\parallel Q_X{Q}_Y)=D(P_{XY}\parallel Q_X{Q}_Y)$.)

The rest of this paper is organized as follows. In Section 2, we review other generalizations of the mutual information. In Section 3, we discuss the operational meanings of $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$. In Section 4, we recall the required Rényi information measures and prove some preparatory results. In Section 5, we state the properties of $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$. In Section 6, we prove these properties.

2. Related Work

The measure $J_{\alpha }(X;Y)$ was discovered independently from the authors of the present paper by Tomamichel and Hayashi [2] (Equation (58)), who, for the case when $\alpha >\frac{1}{2}$, derived some of its properties in [2] (Appendix A-C).

Other Rényi-based measures of dependence appeared in the past. Notable are those by Sibson [3], Arimoto [4], and Csiszár [5], respectively denoted by $I_{\alpha }^{\mathsf{s}}(·)$, $I_{\alpha }^{\mathsf{a}}(·)$, and $I_{\alpha }^{\mathsf{c}}(·)$:

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathsf{s}}(X;Y)& \triangleq \frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P\left(x\right)P{\left(y\right|x)}^{\alpha }\right]}^{\frac{1}{\alpha }}\hfill \end{array}$$

(3)

$$\begin{array}{c}\hfill =\underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel P_X{Q}_Y),\end{array}$$

(4)

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathsf{a}}(X;Y)& \triangleq H_{\alpha }\left(X\right)-H_{\alpha }\left(X\right|Y)\hfill \end{array}$$

(5)

$$\begin{array}{c}=\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _x\frac{P{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}P{\left(x^{\prime }\right)}^{\alpha }}P{\left(y\right|x)}^{\alpha }\right]}^{\frac{1}{\alpha }},\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathsf{c}}(X;Y)& \triangleq \underset{Q_Y}{min}\sum _{x}P\left(x\right)D_{\alpha }(P_{Y|X=x}\parallel Q_Y),\hfill \end{array}$$

(7)

where, throughout the paper, $log(·)$ denotes the base-2 logarithm; $D_{\alpha }(P\parallel Q)$ denotes the Rényi divergence of order $\alpha$ (see (50) ahead); $H_{\alpha }\left(X\right)$ denotes the Rényi entropy of order $\alpha$ (see (45) ahead); and $H_{\alpha }\left(X\right|Y)$ denotes the Arimoto–Rényi conditional entropy [4,6,7], which is defined for positive $\alpha$ other than one as

$$\begin{array}{c}\hfill H_{\alpha }\left(X\right|Y)\triangleq \frac{\alpha }{1-\alpha }log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }\right]}^{\frac{1}{\alpha }}.\end{array}$$

(8)

(Equation (4) follows from Proposition 9 ahead, and (6) follows from (45) and (8).) An overview of $I_{\alpha }^{\mathsf{s}}(·)$, $I_{\alpha }^{\mathsf{a}}(·)$, and $I_{\alpha }^{\mathsf{c}}(·)$ is provided in [8]. Another Rényi-based measure of dependence can be found in [9] (Equation (19)):

$$\begin{array}{c}\hfill I_{\alpha }^{\mathsf{t}}(X;Y)\triangleq D_{\alpha }(P_{XY}\parallel P_X{P}_Y).\end{array}$$

(9)

The relation between $I_{\alpha }^{\mathsf{c}}(X;Y)$, $J_{\alpha }(X;Y)$, and $I_{\alpha }^{\mathsf{s}}(X;Y)$ for $\alpha >1$ was established recently:

Proposition 1

([10] (Theorem IV.1)). For every PMF $P_{XY}$ and every $\alpha >1$,

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathsf{c}}(X;Y)& \le J_{\alpha }(X;Y)\hfill \end{array}$$

(10)

$$\begin{array}{c}\hfill \le I_{\alpha }^{\mathsf{s}}(X;Y).\end{array}$$

(11)

Proof. 

This is proved in [10] for a measure-theoretic setting. Here, we specialize the proof to finite alphabets. We first prove (10):

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\underset{Q_Y}{min}\underset{Q_X}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(12)

$$\begin{array}{c}\hfill =\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _x{\left[\sum _{y}P{(x,y)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\end{array}$$

(13)

$$\begin{array}{c}=\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _{x}P\left(x\right){\left[\sum _{y}P{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\hfill \end{array}$$

(14)

$$\begin{array}{c}\hfill \ge \underset{Q_Y}{min}\frac{\alpha }{\alpha -1}\sum _{x}P\left(x\right)log{\left[\sum _{y}P{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\end{array}$$

(15)

$$\begin{array}{c}=\underset{Q_Y}{min}\sum _{x}P\left(x\right)\frac{1}{\alpha -1}log\sum _{y}P{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\hfill \end{array}$$

(16)

$$\begin{array}{c}\hfill =I_{\alpha }^{\mathsf{c}}(X;Y),\end{array}$$

(17)

where (12) follows from the definition of $J_{\alpha }(X;Y)$ in (1); (13) follows from Proposition 9 ahead with the roles of $Q_X$ and $Q_Y$ swapped; (15) follows from Jensen’s inequality because $log(·)$ is concave and because $\frac{\alpha }{\alpha -1}>0$; and (17) follows from the definition of $I_{\alpha }^{\mathsf{c}}(X;Y)$ in (7).

We next prove (11):

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\underset{Q_X,Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(18)

$$\begin{array}{c}\le \underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel P_X{Q}_Y)\hfill \end{array}$$

(19)

$$\begin{array}{c}\hfill =I_{\alpha }^{\mathsf{s}}(X;Y),\end{array}$$

(20)

where (18) follows from the definition of $J_{\alpha }(X;Y)$ in (1), and (20) follows from (4). □

Many of the above Rényi information measures coincide when they are maximized over $P_X$ with $P_{Y|X}$ held fixed: for every conditional PMF $P_{Y|X}$ and every positive $\alpha$ other than one,

$$\begin{array}{cc}\hfill \underset{P_X}{max}{I}_{\alpha }^{\mathsf{a}}\left(P_X{P}_{Y|X}\right)& =\underset{P_X}{max}{I}_{\alpha }^{\mathsf{s}}\left(P_X{P}_{Y|X}\right)\hfill \end{array}$$

(21)

$$\begin{array}{c}\hfill =\underset{P_X}{max}{I}_{\alpha }^{\mathsf{c}}\left(P_X{P}_{Y|X}\right),\end{array}$$

(22)

where $P_X{P}_{Y|X}$ denotes the joint PMF of X and Y; (21) follows from [4] (Lemma 1); and (22) follows from [5] (Proposition 1). It was recently established that, for $\alpha >1$, this is also true for $J_{\alpha }(X;Y)$:

Proposition 2

([10] (Theorem V.1)). For every conditional PMF $P_{Y|X}$ and every $\alpha >1$,

$$\begin{array}{c}\hfill \underset{P_X}{max}{J}_{\alpha }\left(P_X{P}_{Y|X}\right)=\underset{P_X}{max}{I}_{\alpha }^{\mathsf{s}}\left(P_X{P}_{Y|X}\right).\end{array}$$

(23)

Proof. 

By Proposition 1, we have for all $\alpha >1$

$$\begin{array}{cc}\hfill \underset{P_X}{max}{I}_{\alpha }^{\mathsf{c}}\left(P_X{P}_{Y|X}\right)& \le \underset{P_X}{max}{J}_{\alpha }\left(P_X{P}_{Y|X}\right)\hfill \end{array}$$

(24)

$$\begin{array}{c}\hfill \le \underset{P_X}{max}{I}_{\alpha }^{\mathsf{s}}\left(P_X{P}_{Y|X}\right).\end{array}$$

(25)

By (22), the left-hand side (LHS) of (24) is equal to the right-hand side (RHS) of (25), so (24) and (25) both hold with equality. □

Dependence measures can also be based on the f-divergence $D_f(P\parallel Q)$ [11,12,13]. Every convex function $f:(0,\infty )\to \mathbb{R}$ satisfying $f\left(1\right)=0$ induces a dependence measure, namely

$$\begin{array}{cc}\hfill I_f(X;Y)& \triangleq D_f(P_{XY}\parallel P_X{P}_Y)\hfill \end{array}$$

(26)

$$\begin{array}{c}\hfill =\sum _{x,y}P\left(x\right)P\left(y\right)f\left(\frac{P(x,y)}{P\left(x\right)P\left(y\right)}\right),\end{array}$$

(27)

where (27) follows from the definition of the f-divergence. (For $f\left(t\right)=tlogt$, $I_f(X;Y)$ is the mutual information.) Such dependence measures are used for example in [14], and a construction equivalent to (27) is studied in [15].

3. Operational Meanings

In this section, we discuss the operational meaning of $J_{\alpha }(X;Y)$ in hypothesis testing (Section 3.1) and of $K_{\alpha }(X;Y)$ in distributed task encoding (Section 3.2).

3.1. Testing Against Independence and $J_{\alpha }(X;Y)$

Consider the hypothesis testing problem of guessing whether an observed sequence of pairs was drawn IID from some given joint PMF $P_{XY}$ or IID from some unknown product distribution. Thus, based on a sequence of pairs of random variables ${\left\{(X_i,Y_i)\right\}}_{i=1}^n$, two hypotheses have to be distinguished:

0)

Under the null hypothesis, $(X_1,Y_1),\dots ,(X_n,Y_n)$ are IID according to $P_{XY}$.

1)

Under the alternative hypothesis, $(X_1,Y_1),\dots ,(X_n,Y_n)$ are IID according to some unknown PMF of the form $Q_{XY}=Q_X{Q}_Y$, where $Q_X$ and $Q_Y$ are arbitrary PMFs over $\mathcal{X}$ and $\mathcal{Y}$, respectively.

Associated with every deterministic test $T_n:{\mathcal{X}}^n\times {\mathcal{Y}}^n\to \{0,1\}$ and pair $(Q_X,Q_Y)$ are the type-I error probability $P_{XY}^{\times n}[T_n(X^n,Y^n)=1]$ and the type-II error probability ${\left(Q_X{Q}_Y\right)}^{\times n}[T_n(X^n,Y^n)=0]$, where $R_{XY}^{\times n}\left[\mathcal{A}\right]$ denotes the probability of an event $\mathcal{A}$ when ${\left\{(X_i,Y_i)\right\}}_{i=1}^n$ are IID according to $R_{XY}$. We seek sequences of tests whose worst-case type-II error probability decays exponentially faster than $2^{-n{\mathsf{E}}_{\mathsf{Q}}}$. To be more specific, for a fixed ${\mathsf{E}}_{\mathsf{Q}}\in \mathbb{R}$, denote by $\mathcal{T}\left({\mathsf{E}}_{\mathsf{Q}}\right)$ the set of all sequences of deterministic tests ${\left\{T_n\right\}}_{n=1}^{\infty }$ for which

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\underset{Q_X,Q_Y}{min}-\frac{1}{n}log\left({\left(Q_X{Q}_Y\right)}^{\times n}[T_n(X^n,Y^n)=0]\right)>{\mathsf{E}}_{\mathsf{Q}},\end{array}$$

(28)

where $log(·)$ denotes the base-2 logarithm. Note that (28) implies—but is not equivalent to—that for n sufficiently large, ${\left(Q_X{Q}_Y\right)}^{\times n}[T_n(X^n,Y^n)=0]\le 2^{-n{\mathsf{E}}_{\mathsf{Q}}}$ for all $(Q_X,Q_Y)\in \mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)$. For a fixed ${\mathsf{E}}_{\mathsf{Q}}\in \mathbb{R}$, the optimal type-I error exponent that can be asymptotically achieved under the constraint (28) is given by

$$\begin{array}{c}\hfill {\mathsf{E}}_{\mathsf{P}}\left({\mathsf{E}}_{\mathsf{Q}}\right)\triangleq \underset{{\left\{T_n\right\}}_{n=1}^{\infty }\in \mathcal{T}\left({\mathsf{E}}_{\mathsf{Q}}\right)}{sup}\underset{n\to \infty }{lim\; inf}-\frac{1}{n}log\left(P_{XY}^{\times n}[T_n(X^n,Y^n)=1]\right).\end{array}$$

(29)

The measure $J_{\alpha }(X;Y)$ appears as follows: In [2] (first part of (57)), it is shown that for ${\mathsf{E}}_{\mathsf{Q}}$ sufficiently close to $I(X;Y)$,

$$\begin{array}{c}\hfill {\mathsf{E}}_{\mathsf{P}}\left({\mathsf{E}}_{\mathsf{Q}}\right)=\underset{\alpha \in (\frac{1}{2},1]}{sup}\frac{1-\alpha }{\alpha }(J_{\alpha }(X;Y)-{\mathsf{E}}_{\mathsf{Q}}),\end{array}$$

(30)

and in [16] (Theorem 3), it is shown that for all ${\mathsf{E}}_{\mathsf{Q}}\in \mathbb{R}$,

$$\begin{array}{c}\hfill {\mathsf{E}}_{\mathsf{P}}^{**}\left({\mathsf{E}}_{\mathsf{Q}}\right)=\underset{\alpha \in (0,1]}{sup}\frac{1-\alpha }{\alpha }(J_{\alpha }(X;Y)-{\mathsf{E}}_{\mathsf{Q}}),\end{array}$$

(31)

where ${\mathsf{E}}_{\mathsf{P}}^{**}(·)$ denotes the Fenchel biconjugate of ${\mathsf{E}}_{\mathsf{P}}(·)$. In general, the Fenchel biconjugation cannot be omitted because sometimes [16] (Equation (11) and Example 14)

$$\begin{array}{c}\hfill {\mathsf{E}}_{\mathsf{P}}\left({\mathsf{E}}_{\mathsf{Q}}\right)\ne {\mathsf{E}}_{\mathsf{P}}^{**}\left({\mathsf{E}}_{\mathsf{Q}}\right).\end{array}$$

(32)

For large values of ${\mathsf{E}}_{\mathsf{Q}}$, the optimal type-I error tends to one as n tends to infinity. In this case, the type-I strong-converse exponent [17,18], which is defined for a sequence of tests ${\left\{T_n\right\}}_{n=1}^{\infty }$ as

$$\begin{array}{c}\hfill \mathsf{S}{\mathsf{C}}_{\mathsf{P}}\triangleq \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log\left(1-P_{XY}^{\times n}[T_n(X^n,Y^n)=1]\right),\end{array}$$

(33)

measures how fast the type-I error tends to one as n tends to infinity (smaller values correspond to lower error probabilities). For a fixed ${\mathsf{E}}_{\mathsf{Q}}\in \mathbb{R}$, the optimal type-I strong-converse exponent that can be asymptotically achieved under the constraint (28) is given by

$$\begin{array}{c}\hfill \mathsf{S}{\mathsf{C}}_{\mathsf{P}}\left({\mathsf{E}}_{\mathsf{Q}}\right)\triangleq \underset{{\left\{T_n\right\}}_{n=1}^{\infty }\in \mathcal{T}\left({\mathsf{E}}_{\mathsf{Q}}\right)}{inf}\underset{n\to \infty }{lim\; sup}-\frac{1}{n}log\left(1-P_{XY}^{\times n}[T_n(X^n,Y^n)=1]\right).\end{array}$$

(34)

In [2] (second part of (57)), it is shown that for ${\mathsf{E}}_{\mathsf{Q}}$ sufficiently close to $I(X;Y)$,

$$\begin{array}{c}\hfill \mathsf{S}{\mathsf{C}}_{\mathsf{P}}\left({\mathsf{E}}_{\mathsf{Q}}\right)=\underset{\alpha >1}{sup}\frac{1-\alpha }{\alpha }(J_{\alpha }(X;Y)-{\mathsf{E}}_{\mathsf{Q}}).\end{array}$$

(35)

Here, the same $\frac{1-\alpha }{\alpha }(J_{\alpha }(X;Y)-{\mathsf{E}}_{\mathsf{Q}})$ expression appears as in (30) and (31), but with a different set of $\alpha$’s to optimize over.

3.2. Distributed Task Encoding and $K_{\alpha }(X;Y)$

The task-encoding problem studied in [19] can be extended to a distributed setting as follows [20]: A source ${\left\{(X_i,Y_i)\right\}}_{i=1}^{\infty }$ emits pairs of random variables $(X_i,Y_i)$ taking values in a finite alphabet $\mathcal{X}\times \mathcal{Y}$. For a fixed rate pair $({\mathsf{R}}_{\mathsf{X}},{\mathsf{R}}_{\mathsf{Y}})\in {\mathbb{R}}_{\ge 0}^2$ and a positive integer n, the sequences ${\left\{X_i\right\}}_{i=1}^n$ and ${\left\{Y_i\right\}}_{i=1}^n$ are described separately using $\lfloor 2^{n{\mathsf{R}}_{\mathsf{X}}}\rfloor$ and $\lfloor 2^{n{\mathsf{R}}_{\mathsf{Y}}}\rfloor$ labels, respectively. The decoder produces a list comprising all the pairs $(x^n,y^n)$ whose description matches the given labels, and the goal is to minimize the $\rho$-th moment of the list size as n tends to infinity (for some $\rho >0$).

For a fixed $\rho >0$, a rate pair $({\mathsf{R}}_{\mathsf{X}},{\mathsf{R}}_{\mathsf{Y}})\in {\mathbb{R}}_{\ge 0}^2$ is called achievable if there exists a sequence of encoders ${\left\{(f_n,g_n)\right\}}_{n=1}^{\infty }$,

$$\begin{array}{cc}\hfill f_n:{\mathcal{X}}^n& \to \{1,\dots ,\lfloor 2^{n{\mathsf{R}}_{\mathsf{X}}}\rfloor \},\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill g_n:{\mathcal{Y}}^n& \to \{1,\dots ,\lfloor 2^{n{\mathsf{R}}_{\mathsf{Y}}}\rfloor \},\hfill \end{array}$$

(37)

such that the $\rho$-th moment of the list size tends to one as n tends to infinity, i.e.,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}E\left[{\left|\mathcal{L}(X^n,Y^n)\right|}^{\rho }\right]=1,\end{array}$$

(38)

where

$$\begin{array}{c}\hfill \mathcal{L}(x^n,y^n)\triangleq \{({\tilde{x}}^n,{\tilde{y}}^n)\in {\mathcal{X}}^n\times {\mathcal{Y}}^n:f_n\left({\tilde{x}}^n\right)=f_n\left(x^n\right)\wedge g_n\left({\tilde{y}}^n\right)=g_n\left(y^n\right)\}.\end{array}$$

(39)

For a memoryless source and a fixed $\rho >0$, rate pairs in the interior of the region $\mathcal{R}\left(\rho \right)$ defined next are achievable, while those outside $\mathcal{R}\left(\rho \right)$ are not achievable [20] (Theorem 1). The region $\mathcal{R}\left(\rho \right)$ is defined as the set of all rate pairs $({\mathsf{R}}_{\mathsf{X}},{\mathsf{R}}_{\mathsf{Y}})$ satisfying the following inequalities simultaneously:

$$\begin{array}{cc}\hfill {\mathsf{R}}_{\mathsf{X}}& \ge H_{\frac{1}{1+\rho }}\left(X\right),\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill {\mathsf{R}}_{\mathsf{Y}}& \ge H_{\frac{1}{1+\rho }}\left(Y\right),\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\mathsf{R}}_{\mathsf{X}}+{\mathsf{R}}_{\mathsf{Y}}& \ge H_{\frac{1}{1+\rho }}(X,Y)+K_{\frac{1}{1+\rho }}(X;Y),\hfill \end{array}$$

(42)

where $H_{\alpha }\left(X\right)$ denotes the Rényi entropy of order $\alpha$ (see (45) ahead).

To better understand the role of $K_{\alpha }(X;Y)$, suppose that the sequences ${\left\{X_i\right\}}_{i=1}^n$ and ${\left\{Y_i\right\}}_{i=1}^n$ were allowed to be described jointly using $\lfloor 2^{n{\mathsf{R}}_{\mathsf{X}}}\rfloor ·\lfloor 2^{n{\mathsf{R}}_{\mathsf{Y}}}\rfloor \approx 2^{n({\mathsf{R}}_{\mathsf{X}}+{\mathsf{R}}_{\mathsf{Y}})}$ labels. Then, by [19] (Theorem I.2), all rate pairs $({\mathsf{R}}_{\mathsf{X}},{\mathsf{R}}_{\mathsf{Y}})\in {\mathbb{R}}_{\ge 0}^2$ satisfying the following inequality with strict inequality would be achievable, while those not satisfying the inequality would not:

$$\begin{array}{c}\hfill {\mathsf{R}}_{\mathsf{X}}+{\mathsf{R}}_{\mathsf{Y}}\ge H_{\frac{1}{1+\rho }}(X,Y).\end{array}$$

(43)

Comparing (42) and (43), we see that the measure $K_{\alpha }(X;Y)$ appears as a penalty term on the sum-rate constraint incurred by requiring that the sequences be described separately as opposed to jointly.

4. Preliminaries

Throughout the paper, $log(·)$ denotes the base-2 logarithm, $\mathcal{X}$ and $\mathcal{Y}$ are finite sets, $P_{XY}$ denotes a joint PMF over $\mathcal{X}\times \mathcal{Y}$, $Q_X$ denotes a PMF over $\mathcal{X}$, and $Q_Y$ denotes a PMF over $\mathcal{Y}$. We use P and Q as generic PMFs over a finite set $\mathcal{X}$. We denote by $supp\left(P\right)\triangleq \{x\in \mathcal{X}:P(x)>0\}$ the support of P, and by $\mathcal{P}\left(\mathcal{X}\right)$ the set of all PMFs over $\mathcal{X}$. When clear from the context, we often omit sets and subscripts: for example, we write ${min}_{Q_X,Q_Y}$ for ${min}_{(Q_X,Q_Y)\in \mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)}$, ${\sum }_x$ for ${\sum }_{x\in \mathcal{X}}$, $P\left(x\right)$ for $P_X\left(x\right)$, and $P\left(y\right|x)$ for $P_{Y|X}\left(y\right|x)$. Whenever a conditional probability $P\left(y\right|x)$ is undefined because $P\left(x\right)=0$, we define $P\left(y\right|x)\triangleq 1/|\mathcal{Y}|$. We denote by $𝟙\left\{\mathsf{condition}\right\}$ the indicator function that is one if the condition is satisfied and zero otherwise. In the definitions below, we use the following conventions:

$$\begin{array}{c}\hfill \frac{0}{0}=0,\frac{p}{0}=\infty \forall p>0,0log0=0,\beta log0=-\infty \forall \beta >0.\end{array}$$

(44)

The Rényi entropy of order $\alpha$ [21] is defined for positive $\alpha$ other than one as

$$\begin{array}{c}\hfill H_{\alpha }\left(X\right)\triangleq \frac{1}{1-\alpha }log\sum _{x}P{\left(x\right)}^{\alpha }.\end{array}$$

(45)

For $\alpha$ being zero, one, or infinity, we define by continuous extension of (45)

$$\begin{array}{cc}\hfill H_0\left(X\right)& \triangleq log|supp(P\left)\right|,\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill H_1\left(X\right)& \triangleq H\left(X\right),\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill H_{\infty }\left(X\right)& \triangleq -log\underset{x}{max}P\left(x\right),\hfill \end{array}$$

(48)

where $H\left(X\right)$ is the Shannon entropy. With this extension to $\alpha \in \{0,1,\infty \}$, the Rényi entropy satisfies the following basic properties:

Proposition 3

([5]). Let P be a PMF. Then,

(i) 

For all $\alpha \in [0,\infty ]$, $H_{\alpha }\left(X\right)\le log\left|\mathcal{X}\right|$. If $\alpha \in (0,\infty ]$, then $H_{\alpha }\left(X\right)=log\left|\mathcal{X}\right|$ if and only if X is distributed uniformly over $\mathcal{X}$.

(ii) 

The mapping $\alpha \mapsto H_{\alpha }\left(X\right)$ is nonincreasing on $[0,\infty ]$.

(iii) 

The mapping $\alpha \mapsto H_{\alpha }\left(X\right)$ is continuous on $[0,\infty ]$.

The relative entropy (or Kullback–Leibler divergence) is defined as

$$\begin{array}{c}\hfill D(P\parallel Q)\triangleq \sum _{x}P\left(x\right)log\frac{P\left(x\right)}{Q\left(x\right)}.\end{array}$$

(49)

The Rényi divergence of order $\alpha$ [21,22] is defined for positive $\alpha$ other than one as

$$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)\triangleq \frac{1}{\alpha -1}log\sum _{x}P{\left(x\right)}^{\alpha }Q{\left(x\right)}^{1-\alpha },\end{array}$$

(50)

where we read $P{\left(x\right)}^{\alpha }Q{\left(x\right)}^{1-\alpha }$ as $P{\left(x\right)}^{\alpha }/Q{\left(x\right)}^{\alpha -1}$ if $\alpha >1$. For $\alpha$ being zero, one, or infinity, we define by continuous extension of (50)

$$\begin{array}{cc}\hfill D_0(P\parallel Q)& \triangleq -log\sum _{x\in supp\left(P\right)}Q\left(x\right),\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill D_1(P\parallel Q)& \triangleq D(P\parallel Q),\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill D_{\infty }(P\parallel Q)& \triangleq log\underset{x}{max}\frac{P\left(x\right)}{Q\left(x\right)}.\hfill \end{array}$$

(53)

With this extension to $\alpha \in \{0,1,\infty \}$, the Rényi divergence satisfies the following basic properties:

Proposition 4.

Let P and Q be PMFs. Then,

(i) 

For all $\alpha \in [0,1)$, $D_{\alpha }(P\parallel Q)$ is finite if and only if $|supp(P)\cap supp(Q\left)\right|>0$. For all $\alpha \in [1,\infty ]$, $D_{\alpha }(P\parallel Q)$ is finite if and only if $supp\left(P\right)\subseteq supp\left(Q\right)$.

(ii) 

For all $\alpha \in [0,\infty ]$, $D_{\alpha }(P\parallel Q)\ge 0$. If $\alpha \in (0,\infty ]$, then $D_{\alpha }(P\parallel Q)=0$ if and only if $P=Q$.

(iii) 

For every $\alpha \in [0,\infty ]$, the mapping $Q\mapsto D_{\alpha }(P\parallel Q)$ is continuous.

(iv) 

The mapping $\alpha \mapsto D_{\alpha }(P\parallel Q)$ is nondecreasing on $[0,\infty ]$.

(v) 

The mapping $\alpha \mapsto D_{\alpha }(P\parallel Q)$ is continuous on $[0,\infty ]$.

Proof. 

Part (i) follows from the definition of $D_{\alpha }(P\parallel Q)$ and the conventions (44), and Parts (ii)–(v) are shown in [22]. □

The Rényi divergence for negative $\alpha$ is defined as

$$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)\triangleq \frac{1}{\alpha -1}log\sum _x\frac{Q{\left(x\right)}^{1-\alpha }}{P{\left(x\right)}^{-\alpha }}.\end{array}$$

(54)

(We use negative $\alpha$ only in Lemma 19. More about negative orders can be found in [22] (Section V). For other applications of negative orders, see [23] (Proof of Theorem 1 and Example 1).)

The relative $\alpha$-entropy [24,25] is defined for positive $\alpha$ other than one as

$$\begin{array}{c}\hfill {\Delta }_{\alpha }(P\parallel Q)\triangleq \frac{\alpha }{1-\alpha }log\sum _{x}P\left(x\right)Q{\left(x\right)}^{\alpha -1}+log\sum _{x}Q{\left(x\right)}^{\alpha }-\frac{1}{1-\alpha }log\sum _{x}P{\left(x\right)}^{\alpha },\end{array}$$

(55)

where we read $P\left(x\right)Q{\left(x\right)}^{\alpha -1}$ as $P\left(x\right)/Q{\left(x\right)}^{1-\alpha }$ if $\alpha <1$. The relative $\alpha$-entropy appears in mismatched guessing [26], mismatched source coding [26] (Theorem 8), and mismatched task encoding [19] (Section IV). It also arises in robust parameter estimation and constrained compression settings [25] (Section II). For $\alpha$ being zero, one, or infinity, we define by continuous extension of (55)

$$\begin{array}{cc}\hfill {\Delta }_0(P\parallel Q)& \triangleq \left\{\begin{array}{cc}log\frac{|supp(Q\left)\right|}{|supp(P\left)\right|}\hfill & \mathrm{if}supp\left(P\right)\subseteq supp\left(Q\right),\hfill \\ \infty \hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill {\Delta }_1(P\parallel Q)& \triangleq D(P\parallel Q),\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill {\Delta }_{\infty }(P\parallel Q)& \triangleq log\frac{{max}_{x}P\left(x\right)}{{|argmax\left(Q\right)|}^{-1}{\sum }_{x\in argmax\left(Q\right)}P\left(x\right)},\hfill \end{array}$$

(58)

where $argmax\left(Q\right)\triangleq \{x\in \mathcal{X}:Q\left(x\right)={max}_{x^{\prime }\in \mathcal{X}}Q\left(x^{\prime }\right)\}$ and $|argmax(Q\left)\right|$ is the cardinality of this set. With this extension to $\alpha \in \{0,1,\infty \}$, the relative $\alpha$-entropy satisfies the following basic properties:

Proposition 5.

Let P and Q be PMFs. Then,

(i) 

For all $\alpha \in [0,1]$, ${\Delta }_{\alpha }(P\parallel Q)$ is finite if and only if $supp\left(P\right)\subseteq supp\left(Q\right)$. For all $\alpha \in (1,\infty )$, ${\Delta }_{\alpha }(P\parallel Q)$ is finite if and only if $|supp(P)\cap supp(Q\left)\right|>0$.

(ii) 

For all $\alpha \in [0,\infty ]$, ${\Delta }_{\alpha }(P\parallel Q)\ge 0$. If $\alpha \in (0,\infty )$, then ${\Delta }_{\alpha }(P\parallel Q)=0$ if and only if $P=Q$.

(iii) 

For every $\alpha \in (0,\infty )$, the mapping $Q\mapsto {\Delta }_{\alpha }(P\parallel Q)$ is continuous.

(iv) 

The mapping $\alpha \mapsto {\Delta }_{\alpha }(P\parallel Q)$ is continuous on $[0,\infty ]$.

(Part (i) differs from [19] (Proposition IV.1), where the conventions for $\alpha >1$ differ from ours. Our conventions are compatible with [24,25], and, as stated in Part (iii), they result in the continuity of the mapping $Q\mapsto {\Delta }_{\alpha }(P\parallel Q)$.)

Proof of Proposition 5.

Part (i) follows from the definition of ${\Delta }_{\alpha }(P\parallel Q)$ in (55) and the conventions (44). For $\alpha \in (0,1)\cup (1,\infty )$, Part (ii) follows from [19] (Proposition IV.1); for $\alpha =1$, Part (ii) holds because ${\Delta }_1(P\parallel Q)=D(P\parallel Q)$; and for $\alpha \in \{0,\infty \}$, Part (ii) follows from the definition of ${\Delta }_{\alpha }(P\parallel Q)$. Part (iii) follows from the definition of ${\Delta }_{\alpha }(P\parallel Q)$, and Part (iv) follows from [19] (Proposition IV.1). □

In the rest of this section, we prove some auxiliary results that we need later (Propositions 6–9). We first establish the relation between $D_{\alpha }(P\parallel Q)$ and ${\Delta }_{\alpha }(P\parallel Q)$.

Proposition 6

([26] (Section V, Property 4)). Let P and Q be PMFs, and let $\alpha >0$. Then,

$$\begin{array}{c}\hfill {\Delta }_{\alpha }(P\parallel Q)=D_{\frac{1}{\alpha }}(\tilde{P}\parallel \tilde{Q}),\end{array}$$

(59)

where the PMFs $\tilde{P}$ and $\tilde{Q}$ are given by

$$\begin{array}{cc}\hfill \tilde{P}\left(x\right)& \triangleq \frac{P{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}P{\left(x^{\prime }\right)}^{\alpha }},\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill \tilde{Q}\left(x\right)& \triangleq \frac{Q{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}Q{\left(x^{\prime }\right)}^{\alpha }}.\hfill \end{array}$$

(61)

Proof. 

If $\alpha =1$, then (59) holds because $\tilde{P}=P$, $\tilde{Q}=Q$, and ${\Delta }_1(P\parallel Q)=D_1(P\parallel Q)=D(P\parallel Q)$. Now let $\alpha \ne 1$. Because $\tilde{P}\left(x\right)$ and $\tilde{Q}\left(x\right)$ are zero if and only if $P\left(x\right)$ and $Q\left(x\right)$ are zero, respectively, the LHS of (59) is finite if and only if its RHS is finite. If $D_{1/\alpha }(\tilde{P}\parallel \tilde{Q})$ is finite, then (59) follows from a simple computation. □

In light of Proposition 6, $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ are related as follows:

Proposition 7.

Let $P_{XY}$ be a joint PMF, and let $\alpha >0$. Then,

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)=J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y}),\end{array}$$

(62)

where the joint PMF of $\tilde{X}$ and $\tilde{Y}$ is given by

$$\begin{array}{c}\hfill {\tilde{P}}_{XY}(x,y)\triangleq \frac{P_{XY}{(x,y)}^{\alpha }}{{\sum }_{(x^{\prime },y^{\prime })\in \mathcal{X}\times \mathcal{Y}}{P}_{XY}{(x^{\prime },y^{\prime })}^{\alpha }}.\end{array}$$

(63)

Proof. 

Let $\alpha >0$. For fixed PMFs $Q_X$ and $Q_Y$, define the transformed PMFs $\widetilde{Q_X{Q}_Y}$, ${\tilde{Q}}_X$, and ${\tilde{Q}}_Y$ as

$$\begin{array}{cc}\hfill \widetilde{Q_X{Q}_Y}(x,y)& \triangleq \frac{{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\alpha }}{{\sum }_{(x^{\prime },y^{\prime })\in \mathcal{X}\times \mathcal{Y}}{\left[Q_X\left(x^{\prime }\right)Q_Y\left(y^{\prime }\right)\right]}^{\alpha }},\hfill \end{array}$$

(64)

$$\begin{array}{cc}\hfill {\tilde{Q}}_X\left(x\right)& \triangleq \frac{Q_X{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{Q}_X{\left(x^{\prime }\right)}^{\alpha }},\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill {\tilde{Q}}_Y\left(y\right)& \triangleq \frac{Q_Y{\left(y\right)}^{\alpha }}{{\sum }_{y^{\prime }\in \mathcal{Y}}{Q}_Y{\left(y^{\prime }\right)}^{\alpha }}.\hfill \end{array}$$

(66)

Then,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& =\underset{Q_X,Q_Y}{min}{\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(67)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}{D}_{\frac{1}{\alpha }}({\tilde{P}}_{XY}\parallel \widetilde{Q_X{Q}_Y})\end{array}$$

(68)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}{D}_{\frac{1}{\alpha }}({\tilde{P}}_{XY}\parallel {\tilde{Q}}_X{\tilde{Q}}_Y)\end{array}$$

(69)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}{D}_{\frac{1}{\alpha }}({\tilde{P}}_{XY}\parallel Q_X{Q}_Y)\end{array}$$

(70)

$$\begin{array}{c}\hfill =J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y}),\end{array}$$

(71)

where (67) holds by the definition of $K_{\alpha }(X;Y)$; (68) follows from Proposition 6; (69) holds because $\widetilde{Q_X{Q}_Y}={\tilde{Q}}_X{\tilde{Q}}_Y$; (70) holds because the transformations (65) and (66) are bijective on the set of PMFs over $\mathcal{X}$ and $\mathcal{Y}$, respectively; and (71) holds by the definition of $J_{\alpha }(X;Y)$. □

The next proposition provides a characterization of the mutual information that parallels the definitions of $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$. Because $D_1(P\parallel Q)={\Delta }_1(P\parallel Q)=D(P\parallel Q)$, this also shows that $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ reduce to the mutual information when $\alpha$ is one.

Proposition 8

([27] (Theorem 3.4)). Let $P_{XY}$ be a joint PMF. Then, for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{c}\hfill D(P_{XY}\parallel Q_X{Q}_Y)\ge D(P_{XY}\parallel P_X{P}_Y),\end{array}$$

(72)

with equality if and only if $Q_X=P_X$ and $Q_Y=P_Y$. Thus,

$$\begin{array}{c}\hfill I(X;Y)=\underset{Q_X,Q_Y}{min}D(P_{XY}\parallel Q_X{Q}_Y).\end{array}$$

(73)

Proof. 

A simple computation reveals that

$$\begin{array}{c}\hfill D(P_{XY}\parallel Q_X{Q}_Y)=D(P_{XY}\parallel P_X{P}_Y)+D(P_X\parallel Q_X)+D(P_Y\parallel Q_Y),\end{array}$$

(74)

which implies (72) because $D(P\parallel Q)\ge 0$ with equality if and only if $P=Q$. Thus, (73) holds because $I(X;Y)=D(P_{XY}\parallel P_X{P}_Y)$. □

The last proposition of this section is about a precursor to $J_{\alpha }(X;Y)$, namely, the minimization of $D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ with respect to $Q_Y$ only, which can be carried out explicitly. (This proposition extends [5] (Equation (13)) and [2] (Lemma 29).)

Proposition 9.

Let $P_{XY}$ be a joint PMF and $Q_X$ a PMF. Then, for every $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{c}\hfill \underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)=\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }},\end{array}$$

(75)

with the conventions of (44). If the RHS of (75) is finite, then the minimum is achieved uniquely by

$$\begin{array}{c}\hfill Q_Y^{*}\left(y\right)=\frac{{\left[{\sum }_{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}}{{\sum }_{y^{\prime }\in \mathcal{Y}}{\left[{\sum }_{x}P{(x,y^{\prime })}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}}.\end{array}$$

(76)

For $\alpha =\infty$,

$$\begin{array}{c}\hfill \underset{Q_Y}{min}{D}_{\infty }(P_{XY}\parallel Q_X{Q}_Y)=log\sum _y\underset{x}{max}\frac{P(x,y)}{Q_X\left(x\right)},\end{array}$$

(77)

with the conventions of (44). If the RHS of (77) is finite, then the minimum is achieved uniquely by

$$\begin{array}{c}\hfill Q_Y^{*}\left(y\right)=\frac{{max}_x[P(x,y)/Q_X\left(x\right)]}{{\sum }_{y^{\prime }\in \mathcal{Y}}{max}_x[P(x,y^{\prime })/Q_X\left(x\right)]}.\end{array}$$

(78)

Proof. 

We first treat the case $\alpha \in (0,1)\cup (1,\infty )$. If the RHS of (75) is infinite, then the conventions imply that $D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is infinite for every $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$, so (75) holds. Otherwise, if the RHS of (75) is finite, then the PMF $Q_Y^{*}$ given by (76) is well-defined, and a simple computation shows that for every $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$,

$$\begin{array}{c}\hfill D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)=\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}+D_{\alpha }(Q_Y^{*}\parallel Q_Y).\end{array}$$

(79)

The only term on the RHS of (79) that depends on $Q_Y$ is $D_{\alpha }(Q_Y^{*}\parallel Q_Y)$. Because $D_{\alpha }(Q_Y^{*}\parallel Q_Y)\ge 0$ with equality if and only if $Q_Y=Q_Y^{*}$ (Proposition 4), (79) implies (75) and (76).

The case $\alpha =\infty$ is analogous: if the RHS of (77) is infinite, then the LHS of (77) is infinite, too; and if the RHS of (77) is finite, then the PMF $Q_Y^{*}$ given by (78) is well-defined, and a simple computation shows that for every $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$,

$$\begin{array}{c}\hfill D_{\infty }(P_{XY}\parallel Q_X{Q}_Y)=log\sum _y\underset{x}{max}\frac{P(x,y)}{Q_X\left(x\right)}+D_{\infty }(Q_Y^{*}\parallel Q_Y).\end{array}$$

(80)

The only term on the RHS of (80) that depends on $Q_Y$ is $D_{\infty }(Q_Y^{*}\parallel Q_Y)$. Because $D_{\infty }(Q_Y^{*}\parallel Q_Y)\ge 0$ with equality if and only if $Q_Y=Q_Y^{*}$ (Proposition 4), (80) implies (77) and (78). □

5. Two Measures of Dependence

We state the properties of $J_{\alpha }(X;Y)$ in Theorem 1 and those of $K_{\alpha }(X;Y)$ in Theorem 2. The enumeration labels in the theorems refer to the lemmas in Section 6 where the properties are proved. (The enumeration labels are not consecutive because, in order to avoid forward references in the proofs, the order of the results in Section 6 is not the same as here.)

Theorem 1.

Let X, $X_1$, $X_2$, Y, $Y_1$, $Y_2$, and Z be random variables taking values in finite sets. Then:

(Lemma 1)

For every $\alpha \in [0,\infty ]$, the minimum in the definition of $J_{\alpha }(X;Y)$ exists and is finite.

The following properties of the mutual information $I(X;Y)$ [28] (Chapter 2) are also satisfied by $J_{\alpha }(X;Y)$:

(Lemma 2)

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)\ge 0$. If $\alpha \in (0,\infty ]$, then $J_{\alpha }(X;Y)=0$ if and only if X and Y are independent (nonnegativity).

(Lemma 3)

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)=J_{\alpha }(Y;X)$ (symmetry).

(Lemma 4)

If $X⊸--Y⊸--Z$ form a Markov chain, then $J_{\alpha }(X;Z)\le J_{\alpha }(X;Y)$ for all $\alpha \in [0,\infty ]$ (data-processing inequality).

(Lemma 12)

If the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ are independent, then $J_{\alpha }(X_1,X_2;Y_1,Y_2)=J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2)$ for all $\alpha \in [0,\infty ]$ (additivity).

(Lemma 13)

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$ with equality if and only if $(\alpha \in [\frac{1}{2},\infty ]$, X is distributed uniformly over $\mathcal{X}$, and $H\left(X\right|Y)=0)$.

(Lemma 14)

For every $\alpha \in [1,\infty ]$, $J_{\alpha }(X;Y)$ is concave in $P_X$ for fixed $P_{Y|X}$.

Moreover:

(Lemma 5)

$J_0(X;Y)=0$.

(Lemma 6)

Let $f:\{1,\dots ,|\mathcal{X}\left|\right\}\to \mathcal{X}$ and $g:\{1,\dots ,|\mathcal{Y}\left|\right\}\to \mathcal{Y}$ be bijective functions, and let $\mathsf{A}$ be the $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ matrix whose Row-i Column-j entry ${\mathsf{A}}_{i,j}$ equals $\sqrt{P_{XY}(f\left(i\right),g\left(j\right))}$. Then,

$$\begin{array}{c}\hfill J_{\frac{1}{2}}(X;Y)=-2log{\sigma }_1\left(\mathsf{A}\right),\end{array}$$

(81)

where ${\sigma }_1\left(\mathsf{A}\right)$ denotes the largest singular value of $\mathsf{A}$. (Because the singular values of a matrix are invariant under row and column permutations, the result does not depend on f or g.)

(Lemma 7)

$J_1(X;Y)=I(X;Y)$.

(Lemma 8)

For all $\alpha >0$,

$$\begin{array}{c}\hfill (1-\alpha )J_{\alpha }(X;Y)=\underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{min}\left[(1-\alpha )D(R_{XY}\parallel R_X{R}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right].\end{array}$$

(82)

Thus, being the minimum of concave functions in α, the mapping $\alpha \mapsto (1-\alpha )J_{\alpha }(X;Y)$ is concave on $(0,\infty )$.

(Lemma 9)

The mapping $\alpha \mapsto J_{\alpha }(X;Y)$ is nondecreasing on $[0,\infty ]$.

(Lemma 10)

The mapping $\alpha \mapsto J_{\alpha }(X;Y)$ is continuous on $[0,\infty ]$.

(Lemma 11)

If $X=Y$ with probability one, then

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\left\{\begin{array}{cc}\frac{\alpha }{1-\alpha }{H}_{\infty }\left(X\right)\hfill & if\alpha \in [0,\frac{1}{2}],\hfill \\ H_{\frac{\alpha }{2\alpha -1}}\left(X\right)\hfill & if\alpha >\frac{1}{2},\hfill \\ H_{\frac{1}{2}}\left(X\right)\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(83)

The minimization problem in the definition of $J_{\alpha }(X;Y)$ has the following characteristics:

(Lemma 15)

For every $\alpha \in [\frac{1}{2},\infty ]$, the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is convex, i.e., for all $\lambda ,{\lambda }^{\prime }\in [0,1]$ with $\lambda +{\lambda }^{\prime }=1$, all $Q_X,Q_X^{\prime }\in \mathcal{P}\left(\mathcal{X}\right)$, and all $Q_Y,Q_Y^{\prime }\in \mathcal{P}\left(\mathcal{Y}\right)$,

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{XY}\parallel (\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })(\lambda Q_Y+{\lambda }^{\prime }{Q}_Y^{\prime })\right)\le \lambda D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)+{\lambda }^{\prime }{D}_{\alpha }(P_{XY}\parallel Q_X^{\prime }{Q}_Y^{\prime }).\end{array}$$

(84)

For $\alpha \in [0,\frac{1}{2})$, the mapping need not be convex.

(Lemma 16)

Let $\alpha \in (0,1)\cup (1,\infty )$. If $(Q_X^{*},Q_Y^{*})$ achieves the minimum in the definition of $J_{\alpha }(X;Y)$, then there exist positive normalization constants c and d such that

$$\begin{array}{cc}\hfill Q_X^{*}\left(x\right)& =c{\left[\sum _{y}P{(x,y)}^{\alpha }{Q}_Y^{*}{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\forall x\in \mathcal{X},\hfill \end{array}$$

(85)

$$\begin{array}{cc}\hfill Q_Y^{*}\left(y\right)& =d{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X^{*}{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\forall y\in \mathcal{Y},\hfill \end{array}$$

(86)

with the conventions of (44). The case $\alpha =\infty$ is similar: if $(Q_X^{*},Q_Y^{*})$ achieves the minimum in the definition of $J_{\infty }(X;Y)$, then there exist positive normalization constants c and d such that

$$\begin{array}{cc}\hfill Q_X^{*}\left(x\right)& =c\underset{y}{max}\frac{P(x,y)}{Q_Y^{*}\left(y\right)}\forall x\in \mathcal{X},\hfill \end{array}$$

(87)

$$\begin{array}{cc}\hfill Q_Y^{*}\left(y\right)& =d\underset{x}{max}\frac{P(x,y)}{Q_X^{*}\left(x\right)}\forall y\in \mathcal{Y},\hfill \end{array}$$

(88)

with the conventions of (44). (If $\alpha =1$, then $Q_X^{*}=P_X$ and $Q_Y^{*}=P_Y$ by Proposition 8.) Thus, for all $\alpha \in (0,\infty ]$, both inclusions $supp\left(Q_X^{*}\right)\subseteq supp\left(P_X\right)$ and $supp\left(Q_Y^{*}\right)\subseteq supp\left(P_Y\right)$ hold. $\frac{1}{2}$

(Lemma 20)

For every $\alpha \in (\frac{1}{2},\infty ]$, the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ has a unique minimizer. This need not be the case when $\alpha \in [0,\frac{1}{2}]$.

The measure $J_{\alpha }(X;Y)$ can also be expressed as follows:

(Lemma 17)

For all $\alpha \in (0,\infty ]$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\underset{Q_X}{min}{\varphi }_{\alpha }\left(Q_X\right),\end{array}$$

(89)

where ${\varphi }_{\alpha }\left(Q_X\right)$ is defined as

$$\begin{array}{c}\hfill {\varphi }_{\alpha }\left(Q_X\right)\triangleq \underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\end{array}$$

(90)

and is given explicitly as follows: for $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{c}\hfill {\varphi }_{\alpha }\left(Q_X\right)=\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }},\end{array}$$

(91)

with the conventions of (44); and for $\alpha \in \{1,\infty \}$,

$$\begin{array}{cc}\hfill {\varphi }_1\left(Q_X\right)& =D(P_{XY}\parallel Q_X{P}_Y),\hfill \end{array}$$

(92)

$$\begin{array}{cc}\hfill {\varphi }_{\infty }\left(Q_X\right)& =log\sum _y\underset{x}{max}\frac{P(x,y)}{Q_X\left(x\right)},\hfill \end{array}$$

(93)

with the conventions of (44). For every $\alpha \in [\frac{1}{2},\infty ]$, the mapping $Q_X\mapsto {\varphi }_{\alpha }\left(Q_X\right)$ is convex. For $\alpha \in (0,\frac{1}{2})$, the mapping need not be convex.

(Lemma 18)

For all $\alpha \in (0,1)\cup (1,\infty ]$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\left\{\begin{array}{cc}{\displaystyle \underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{min}{\psi }_{\alpha }\left(R_{XY}\right)}\hfill & if\alpha \in (0,1),\hfill \\ {\displaystyle \underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{max}{\psi }_{\alpha }\left(R_{XY}\right)}\hfill & if\alpha \in (1,\infty ],\hfill \end{array}\right.\end{array}$$

(94)

where

$$\begin{array}{c}\hfill {\psi }_{\alpha }\left(R_{XY}\right)\triangleq \left\{\begin{array}{cc}{\displaystyle D(R_{XY}\parallel R_X{R}_Y)+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY})}\hfill & if\alpha \in (0,1)\cup (1,\infty ),\hfill \\ {\displaystyle D(R_{XY}\parallel R_X{R}_Y)-D(R_{XY}\parallel P_{XY})}\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(95)

For every $\alpha \in (1,\infty ]$, the mapping $R_{XY}\mapsto {\psi }_{\alpha }\left(R_{XY}\right)$ is concave. For all $\alpha \in (1,\infty ]$ and all $R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})$, the statement $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$ is equivalent to ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$.

(Lemma 19)

For all $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\underset{R_X\ll P_X}{min}\frac{1}{\alpha -1}\left[D_{\frac{\alpha }{\alpha -1}}(P_X\parallel R_X)-\alpha E_0\left(\frac{1-\alpha }{\alpha },R_X\right)\right],\end{array}$$

(96)

where the minimization is over all PMFs $R_X$ satisfying $R_X\ll P_X$$\left(\mathrm{i}.\mathrm{e}.,supp\left(R_X\right)\subseteq supp\left(P_X\right)\right)$; $D_{\alpha }(P\parallel Q)$ for negative α is given by (54); and Gallager’s $E_0$ function [29] is defined as

$$\begin{array}{c}\hfill E_0(\rho ,R_X)\triangleq -log\sum _y{\left[\sum _x{R}_X\left(x\right)P{\left(y\right|x)}^{\frac{1}{1+\rho }}\right]}^{1+\rho }.\end{array}$$

(97)

We now move on to the properties of $K_{\alpha }(X;Y)$. Some of these properties are derived from their counterparts of $J_{\alpha }(X;Y)$ using the relation $K_{\alpha }(X;Y)=J_{1/\alpha }(\tilde{X};\tilde{Y})$ described in Proposition 7.

Theorem 2.

Let X, $X_1$, $X_2$, Y, $Y_1$, $Y_2$, and Z be random variables taking values in finite sets. Then:

(Lemma 21)

For every $\alpha \in [0,\infty ]$, the minimum in the definition of $K_{\alpha }(X;Y)$ in (2) exists and is finite.

The following properties of the mutual information $I(X;Y)$ are also satisfied by $K_{\alpha }(X;Y)$:

(Lemma 22)

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)\ge 0$. If $\alpha \in (0,\infty )$, then $K_{\alpha }(X;Y)=0$ if and only if X and Y are independent (nonnegativity).

(Lemma 23)

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)=K_{\alpha }(Y;X)$ (symmetry).

(Lemma 34)

If the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ are independent, then $K_{\alpha }(X_1,X_2;Y_1,Y_2)=K_{\alpha }(X_1;Y_1)+K_{\alpha }(X_2;Y_2)$ for all $\alpha \in [0,\infty ]$ (additivity).

(Lemma 35)

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$.

Unlike the mutual information, $K_{\alpha }(X;Y)$ does not satisfy the data-processing inequality:

(Lemma 36)

There exists a Markov chain $X⊸--Y⊸--Z$ for which $K_2(X;Z)>K_2(X;Y)$.

Moreover:

(Lemma 24)

For all $\alpha \in (0,\infty )$,

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)=\underset{Q_X,Q_Y}{min}-log{M}_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y),\end{array}$$

(98)

where $M_{\beta }(Q_X,Q_Y)$ is the following weighted power mean [30] (Chapter III): For $\beta \in \mathbb{R}\setminus \left\{0\right\}$,

$$\begin{array}{c}\hfill M_{\beta }(Q_X,Q_Y)\triangleq {\left[\sum _{x,y}P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\beta }\right]}^{\frac{1}{\beta }},\end{array}$$

(99)

where for $\beta <0$, we read $P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\beta }$ as $P(x,y)/{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{-\beta }$ and use the conventions (44); and for $\beta =0$, using the convention $0^0=1$,

$$\begin{array}{c}\hfill M_0(Q_X,Q_Y)\triangleq \prod _{x,y}{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{P(x,y)}.\end{array}$$

(100)

(Lemma 25)

For $\alpha =0$,

$$\begin{array}{cc}\hfill K_0(X;Y)& =log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill \end{array}$$

(101)

$$\begin{array}{c}\ge \underset{Q_X,Q_Y}{min}log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\frac{1}{Q_X\left(x\right)Q_Y\left(y\right)}-log|supp\left(P_{XY}\right)|\hfill \end{array}$$

(102)

$$\begin{array}{c}\hfill =\underset{\alpha \downarrow 0}{lim}{K}_{\alpha }(X;Y),\end{array}$$

(103)

where in the RHS of (102), we use the conventions (44). The inequality can be strict, so $\alpha \mapsto K_{\alpha }(X;Y)$ need not be continuous at $\alpha =0$.

(Lemma 26)

$K_1(X;Y)=I(X;Y)$.

(Lemma 27)

Let $f:\{1,\dots ,|\mathcal{X}\left|\right\}\to \mathcal{X}$ and $g:\{1,\dots ,|\mathcal{Y}\left|\right\}\to \mathcal{Y}$ be bijective functions, and let $\mathsf{B}$ be the $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ matrix whose Row-i Column-j entry ${\mathsf{B}}_{i,j}$ equals $P_{XY}(f\left(i\right),g\left(j\right))$. Then,

$$\begin{array}{c}\hfill K_2(X;Y)=-2log{\sigma }_1\left(\mathsf{B}\right)-H_2(X,Y),\end{array}$$

(104)

where ${\sigma }_1\left(\mathsf{B}\right)$ denotes the largest singular value of $\mathsf{B}$. (Because the singular values of a matrix are invariant under row and column permutations, the result does not depend on f or g.)

(Lemma 28)

$K_{\infty }(X;Y)=0$.

(Lemma 29)

The mapping $\alpha \mapsto K_{\alpha }(X;Y)$ need not be monotonic on $[0,\infty ]$.

(Lemma 30)

The mapping $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is nonincreasing on $[0,\infty ]$.

(Lemma 31)

The mapping $\alpha \mapsto K_{\alpha }(X;Y)$ is continuous on $(0,\infty ]$. (See Lemma 25 for the behavior at $\alpha =0$.)

(Lemma 32)

If $X=Y$ with probability one, then

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)=\left\{\begin{array}{cc}2H_{\frac{\alpha }{2-\alpha }}\left(X\right)-H_{\alpha }\left(X\right)\hfill & if\alpha \in [0,2),\hfill \\ \frac{\alpha }{\alpha -1}{H}_{\infty }\left(X\right)-H_{\alpha }\left(X\right)\hfill & if\alpha \ge 2,\hfill \\ 0\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(105)

(Lemma 33)

For every $\alpha \in (0,2)$, the mapping $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ in the definition of $K_{\alpha }(X;Y)$ in (2) has a unique minimizer. This need not be the case when $\alpha \in \left\{0\right\}\cup [2,\infty ]$.

6. Proofs

In this section, we prove the properties of $J_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ stated in Section 5.

Lemma 1.

For every $\alpha \in [0,\infty ]$, the minimum in the definition of $J_{\alpha }(X;Y)$ exists and is finite.

Proof. 

Let $\alpha \in [0,\infty ]$. Then ${inf}_{Q_X,Q_Y}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is finite because $D_{\alpha }(P_{XY}\parallel P_X{P}_Y)$ is finite and because the Rényi divergence is nonnegative. The minimum exists because the set $\mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)$ is compact and the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is continuous. □

Lemma 2.

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)\ge 0$. If $\alpha \in (0,\infty ]$, then $J_{\alpha }(X;Y)=0$ if and only if X and Y are independent (nonnegativity).

Proof. 

The nonnegativity follows from the definition of $J_{\alpha }(X;Y)$ because the Rényi divergence is nonnegative for $\alpha \in [0,\infty ]$. If X and Y are independent, then $P_{XY}=P_X{P}_Y$, and the choice $Q_X=P_X$ and $Q_Y=P_Y$ in the definition of $J_{\alpha }(X;Y)$ achieves $J_{\alpha }(X;Y)=0$. Conversely, if $J_{\alpha }(X;Y)=0$, then there exist PMFs $Q_X^{*}$ and $Q_Y^{*}$ satisfying $D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})=0$. If, in addition, $\alpha \in (0,\infty ]$, then $P_{XY}=Q_X^{*}{Q}_Y^{*}$ by Proposition 4, and hence X and Y are independent. □

Lemma 3.

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)=J_{\alpha }(Y;X)$ (symmetry).

Proof. 

The definition of $J_{\alpha }(X;Y)$ is symmetric in X and Y. □

Lemma 4.

If $X⊸--Y⊸--Z$ form a Markov chain, then $J_{\alpha }(X;Z)\le J_{\alpha }(X;Y)$ for all $\alpha \in [0,\infty ]$ (data-processing inequality).

Proof. 

Let $X⊸--Y⊸--Z$ form a Markov chain, and let $\alpha \in [0,\infty ]$. Let ${\widehat{Q}}_X$ and ${\widehat{Q}}_Y$ be PMFs that achieve the minimum in the definition of $J_{\alpha }(X;Y)$, so

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=D_{\alpha }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y).\end{array}$$

(106)

Define the PMF ${\widehat{Q}}_Z$ as

$$\begin{array}{c}\hfill {\widehat{Q}}_Z\left(z\right)\triangleq \sum _y{\widehat{Q}}_Y\left(y\right)P_{Z|Y}\left(z\right|y).\end{array}$$

(107)

(As noted in the preliminaries, we define $P_{Z|Y}\left(z\right|y)\triangleq 1/\left|\mathcal{Z}\right|$ when $P_Y\left(y\right)=0$.) We show below that

$$\begin{array}{c}\hfill D_{\alpha }(P_{XZ}\parallel {\widehat{Q}}_X{\widehat{Q}}_Z)\le D_{\alpha }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y),\end{array}$$

(108)

which implies the data-processing inequality because

$$\begin{array}{cc}\hfill J_{\alpha }(X;Z)& \le D_{\alpha }(P_{XZ}\parallel {\widehat{Q}}_X{\widehat{Q}}_Z)\hfill \end{array}$$

(109)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y)\end{array}$$

(110)

$$\begin{array}{c}\hfill =J_{\alpha }(X;Y),\end{array}$$

(111)

where (109) holds by the definition of $J_{\alpha }(X;Z)$; (110) follows from (108); and (111) follows from (106).

The proof of (108) is based on the data-processing inequality for the Rényi divergence. Define the conditional PMF $A_{X^{\prime }{Z}^{\prime }|XY}$ as

$$\begin{array}{c}\hfill A_{X^{\prime }{Z}^{\prime }|XY}(x^{\prime },z^{\prime }|x,y)\triangleq 𝟙\{x^{\prime }=x\}{P}_{Z|Y}\left(z^{\prime }\right|y).\end{array}$$

(112)

If $(X,Y)\sim P_{XY}$, then the marginal distribution of $X^{\prime }$ and $Z^{\prime }$ is

$$\begin{array}{cc}\hfill \left(P_{XY}{A}_{X^{\prime }{Z}^{\prime }|XY}\right)(x^{\prime },z^{\prime })& =\sum _{x,y}{P}_{XY}(x,y)A_{X^{\prime }{Z}^{\prime }|XY}(x^{\prime },z^{\prime }|x,y)\hfill \end{array}$$

(113)

$$\begin{array}{c}\hfill =\sum _y{P}_{XY}(x^{\prime },y)P_{Z|Y}\left(z^{\prime }\right|y)\end{array}$$

(114)

$$\begin{array}{c}\hfill =\sum _y{P}_{XY}(x^{\prime },y)P_{Z|XY}\left(z^{\prime }\right|x^{\prime },y)\end{array}$$

(115)

$$\begin{array}{c}\hfill =P_{XZ}(x^{\prime },z^{\prime }),\end{array}$$

(116)

where (114) follows from (112); and (115) holds because X, Y, and Z form a Markov chain. If $(X,Y)\sim {\widehat{Q}}_X{\widehat{Q}}_Y$, then the marginal distribution of $X^{\prime }$ and $Z^{\prime }$ is

$$\begin{array}{cc}\hfill \left({\widehat{Q}}_X{\widehat{Q}}_Y{A}_{X^{\prime }{Z}^{\prime }|XY}\right)(x^{\prime },z^{\prime })& =\sum _{x,y}{\widehat{Q}}_X\left(x\right){\widehat{Q}}_Y\left(y\right)A_{X^{\prime }{Z}^{\prime }|XY}(x^{\prime },z^{\prime }|x,y)\hfill \end{array}$$

(117)

$$\begin{array}{c}\hfill =\sum _y{\widehat{Q}}_X\left(x^{\prime }\right){\widehat{Q}}_Y\left(y\right)P_{Z|Y}\left(z^{\prime }\right|y)\end{array}$$

(118)

$$\begin{array}{c}\hfill ={\widehat{Q}}_X\left(x^{\prime }\right){\widehat{Q}}_Z\left(z^{\prime }\right),\end{array}$$

(119)

where (118) follows from (112), and (119) follows from (107). Finally, we are ready to prove (108):

$$\begin{array}{cc}\hfill D_{\alpha }(P_{XZ}\parallel {\widehat{Q}}_X{\widehat{Q}}_Z)& =D_{\alpha }\left(\left(P_{XY}{A}_{X^{\prime }{Z}^{\prime }|XY}\right)\parallel \left({\widehat{Q}}_X{\widehat{Q}}_Y{A}_{X^{\prime }{Z}^{\prime }|XY}\right)\right)\hfill \end{array}$$

(120)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y),\end{array}$$

(121)

where (120) follows from (116) and (119), and where (121) follows from the data-processing inequality for the Rényi divergence [22] (Theorem 9). □

Lemma 5.

$J_0(X;Y)=0$.

Proof. 

By Lemma 2, $J_0(X;Y)\ge 0$, so it suffices to show that $J_0(X;Y)\le 0$. Let $(\widehat{x},\widehat{y})\in \mathcal{X}\times \mathcal{Y}$ satisfy $P_{XY}(\widehat{x},\widehat{y})>0$. Define the PMF ${\widehat{Q}}_X$ as ${\widehat{Q}}_X\left(x\right)\triangleq 𝟙\{x=\widehat{x}\}$ and the PMF ${\widehat{Q}}_Y$ as ${\widehat{Q}}_Y\left(y\right)\triangleq 𝟙\{y=\widehat{y}\}$. Then, $D_0(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y)=0$, so $J_0(X;Y)\le 0$ by the definition of $J_0(X;Y)$. □

Lemma 6.

Let $f:\{1,\dots ,|\mathcal{X}\left|\right\}\to \mathcal{X}$ and $g:\{1,\dots ,|\mathcal{Y}\left|\right\}\to \mathcal{Y}$ be bijective functions, and let $\mathsf{A}$ be the $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ matrix whose Row-i Column-j entry ${\mathsf{A}}_{i,j}$ equals $\sqrt{P_{XY}(f\left(i\right),g\left(j\right))}$. Then,

$$\begin{array}{c}\hfill J_{\frac{1}{2}}(X;Y)=-2log{\sigma }_1\left(\mathsf{A}\right),\end{array}$$

(122)

where ${\sigma }_1\left(\mathsf{A}\right)$ denotes the largest singular value of $\mathsf{A}$. (Because the singular values of a matrix are invariant under row and column permutations, the result does not depend on f or g.)

Proof. 

By the definitions of $J_{\alpha }(X;Y)$ and the Rényi divergence,

$$\begin{array}{c}\hfill J_{\frac{1}{2}}(X;Y)=-2log\underset{Q_X,Q_Y}{max}\sum _{x,y}\sqrt{Q_X\left(x\right)}\sqrt{P(x,y)}\sqrt{Q_Y\left(y\right)}.\end{array}$$

(123)

The claim follows from (123) because

$$\begin{array}{cc}\hfill \underset{Q_X,Q_Y}{max}\sum _{x,y}\sqrt{Q_X\left(x\right)}\sqrt{P(x,y)}\sqrt{Q_Y\left(y\right)}& =\underset{{\parallel \mathrm{u}\parallel }_2={\parallel \mathrm{v}\parallel }_2=1}{max}{\mathrm{u}}^{\mathsf{T}}\mathsf{A}\mathrm{v}\hfill \end{array}$$

(124)

$$\begin{array}{c}=\underset{{\parallel \mathrm{v}\parallel }_2=1}{max}{\parallel \mathsf{A}\mathrm{v}\parallel }_2\hfill \end{array}$$

(125)

$$\begin{array}{c}={\sigma }_1\left(\mathsf{A}\right),\hfill \end{array}$$

(126)

where $\mathrm{u}$ and $\mathrm{v}$ are column vectors with $\left|\mathcal{X}\right|$ and $\left|\mathcal{Y}\right|$ elements, respectively; (124) is shown below; (125) follows from the Cauchy–Schwarz inequality $|{\mathrm{u}}^{\mathsf{T}}\mathsf{A}\mathrm{v}{|\le \parallel \mathrm{u}\parallel }_2{\parallel \mathsf{A}\mathrm{v}\parallel }_2$, which holds with equality if $\mathrm{u}$ and $\mathsf{A}\mathrm{v}$ are linearly dependent; and (126) holds because the spectral norm of a matrix is equal to its largest singular value [31] (Example 5.6.6).

We now prove (124). Let $\mathrm{u}$ and $\mathrm{v}$ be vectors that satisfy ${\parallel \mathrm{u}\parallel }_2={\parallel \mathrm{v}\parallel }_2=1$, and define the PMFs ${\widehat{Q}}_X$ and ${\widehat{Q}}_Y$ as ${\widehat{Q}}_X\left(x\right)\triangleq {\mathrm{u}}_{f^{-1}\left(x\right)}^2$ and ${\widehat{Q}}_Y\left(y\right)\triangleq {\mathrm{v}}_{g^{-1}\left(y\right)}^2$, where $f^{-1}$ and $g^{-1}$ denote the inverse functions of f and g, respectively. Then,

$$\begin{array}{cc}\hfill {\mathrm{u}}^{\mathsf{T}}\mathsf{A}\mathrm{v}& =\sum _{i,j}{\mathrm{u}}_i{\mathsf{A}}_{i,j}{\mathrm{v}}_j\hfill \end{array}$$

(127)

$$\begin{array}{c}\hfill \le \sum _{i,j}|{\mathrm{u}}_i|{\mathsf{A}}_{i,j}\left|{\mathrm{v}}_j\right|\end{array}$$

(128)

$$\begin{array}{c}\hfill =\sum _{x,y}\sqrt{{\widehat{Q}}_X\left(x\right)}\sqrt{P(x,y)}\sqrt{{\widehat{Q}}_Y\left(y\right)}\end{array}$$

(129)

$$\begin{array}{c}\hfill \le \underset{Q_X,Q_Y}{max}\sum _{x,y}\sqrt{Q_X\left(x\right)}\sqrt{P(x,y)}\sqrt{Q_Y\left(y\right)},\end{array}$$

(130)

where (128) holds because all the entries of $\mathsf{A}$ are nonnegative, and in (129), we changed the summation variables to $x\triangleq f\left(i\right)$ and $y\triangleq g\left(j\right)$. It remains to show that equality can be achieved in (128) and (130). To that end, let $Q_X^{*}$ and $Q_Y^{*}$ be PMFs that achieve the maximum on the RHS of (130), and define the vectors $\mathrm{u}$ and $\mathrm{v}$ as ${\mathrm{u}}_i\triangleq Q_X^{*}{\left(f\left(i\right)\right)}^{1/2}$ and ${\mathrm{v}}_j\triangleq Q_Y^{*}{\left(g\left(j\right)\right)}^{1/2}$. Then, ${\parallel \mathrm{u}\parallel }_2={\parallel \mathrm{v}\parallel }_2=1$, and (128) and (130) hold with equality, which proves (124). □

Lemma 7.

$J_1(X;Y)=I(X;Y)$.

Proof. 

This follows from Proposition 8 because $D_1(P_{XY}\parallel Q_X{Q}_Y)$ in the definition of $J_1(X;Y)$ is equal to $D(P_{XY}\parallel Q_X{Q}_Y)$. □

Lemma 8.

For all $\alpha >0$,

$$\begin{array}{c}\hfill (1-\alpha )J_{\alpha }(X;Y)=\underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{min}\left[(1-\alpha )D(R_{XY}\parallel R_X{R}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right].\end{array}$$

(131)

Thus, being the minimum of concave functions in α, the mapping $\alpha \mapsto (1-\alpha )J_{\alpha }(X;Y)$ is concave on $(0,\infty )$.

Proof. 

For $\alpha =1$, (131) holds because $D(R_{XY}\parallel P_{XY})\ge 0$ with equality if $R_{XY}=P_{XY}$. For $\alpha \in (0,1)$,

$$\begin{array}{cc}\hfill (1-\alpha )J_{\alpha }(X;Y)& =\underset{Q_X,Q_Y}{min}(1-\alpha )D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(132)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}\underset{R_{XY}}{min}\left[(1-\alpha )D(R_{XY}\parallel Q_X{Q}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right]\end{array}$$

(133)

$$\begin{array}{c}\hfill =\underset{R_{XY}}{min}\left[(1-\alpha )D(R_{XY}\parallel R_X{R}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right],\end{array}$$

(134)

where (132) holds by the definition of $J_{\alpha }(X;Y)$; (133) follows from [22] (Theorem 30); and (134) follows from Proposition 8 after swapping the minima.

For $\alpha >1$, define the sets

$$\begin{array}{cc}\hfill \mathcal{Q}& \triangleq \{(Q_X,Q_Y)\in \mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right):supp\left(Q_X{Q}_Y\right)=\mathcal{X}\times \mathcal{Y}\},\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill \mathcal{R}& \triangleq \{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y}):supp\left(R_{XY}\right)\subseteq supp\left(P_{XY}\right)\}.\hfill \end{array}$$

(136)

Then,

$$\begin{array}{cc}\hfill (1-\alpha )J_{\alpha }(X;Y)& =\underset{(Q_X,Q_Y)\in \mathcal{Q}}{sup}(1-\alpha )D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(137)

$$\begin{array}{c}\hfill =\underset{(Q_X,Q_Y)\in \mathcal{Q}}{sup}\underset{R_{XY}\in \mathcal{R}\phantom{(Q_X,Q_Y)\in \mathcal{Q}}}{min}\left[(1-\alpha )D(R_{XY}\parallel Q_X{Q}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right]\end{array}$$

(138)

$$\begin{array}{c}\hfill =\underset{R_{XY}\in \mathcal{R}\phantom{(Q_X,Q_Y)\in \mathcal{Q}}}{min}\underset{(Q_X,Q_Y)\in \mathcal{Q}}{sup}\left[(1-\alpha )D(R_{XY}\parallel Q_X{Q}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right]\end{array}$$

(139)

$$\begin{array}{c}\hfill =\underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{min}\left[(1-\alpha )D(R_{XY}\parallel R_X{R}_Y)+\alpha D(R_{XY}\parallel P_{XY})\right],\end{array}$$

(140)

where (137) follows from the definition of $J_{\alpha }(X;Y)$ because $1-\alpha <0$ and because the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is continuous; (138) follows from [22] (Theorem 30); (139) follows from a minimax theorem and is justified below; and (140) follows from Proposition 8, a continuity argument, and the observation that $D(R_{XY}\parallel P_{XY})$ is infinite if $R_{XY}\notin \mathcal{R}$.

We now verify the conditions of Ky Fan’s minimax theorem [32] (Theorem 2), which will establish (139). (We use Ky Fan’s minimax theorem because it does not require that the set $\mathcal{Q}$ be compact, and having a noncompact set $\mathcal{Q}$ helps to guarantee that the function f defined next takes on finite values only. A brief proof of Ky Fan’s minimax theorem appears in [33].) Let the function $f:\mathcal{R}\times \mathcal{Q}\to \mathbb{R}$ be defined by the expression in square brackets in (139), i.e.,

$$\begin{array}{c}\hfill f(R_{XY},Q_X,Q_Y)\triangleq (1-\alpha )D(R_{XY}\parallel Q_X{Q}_Y)+\alpha D(R_{XY}\parallel P_{XY}).\end{array}$$

(141)

We check that

(i)

the sets $\mathcal{Q}$ and $\mathcal{R}$ are convex;

(ii)

the set $\mathcal{R}$ is compact;

(iii)

the function f is real-valued;

(iv)

for every $(Q_X,Q_Y)\in \mathcal{Q}$, the function f is continuous in $R_{XY}$;

(v)

for every $(Q_X,Q_Y)\in \mathcal{Q}$, the function f is convex in $R_{XY}$; and

(vi)

for every $R_{XY}\in \mathcal{R}$, the function f is concave in the pair $(Q_X,Q_Y)$.

Indeed, Parts (i) and (ii) are easy to see; Part (iii) holds because both relative entropies on the RHS of (141) are finite by our definitions of $\mathcal{Q}$ and $\mathcal{R}$; and to show Parts (iv)–(vi), we rewrite f as:

$$\begin{array}{cc}\hfill f(R_{XY},Q_X,Q_Y)& =-H\left(R_{XY}\right)-\alpha \sum _{x,y}{R}_{XY}(x,y)logP(x,y)\hfill \\ & \phantom{=}+(\alpha -1)\sum _x{R}_X\left(x\right)log{Q}_X\left(x\right)+(\alpha -1)\sum _y{R}_Y\left(y\right)log{Q}_Y\left(y\right).\hfill \end{array}$$

(142)

From (142), we see that Part (iv) holds by our definitions of $\mathcal{Q}$ and $\mathcal{R}$; Part (v) holds because the entropy is a concave function (so $-H\left(R_{XY}\right)$ is convex), because linear functionals of $R_{XY}$ are convex, and because the sum of convex functions is convex; and Part (vi) holds because the logarithm is a concave function and because a nonnegative weighted sum of concave functions is concave. (In Ky Fan’s theorem, weaker conditions than Parts (i)–(vi) are required, but it is not difficult to see that Parts (i)–(vi) are sufficient.)

The last claim, namely, that the mapping $\alpha \mapsto (1-\alpha )J_{\alpha }(X;Y)$ is concave on $(0,\infty )$, is true because the expression in square brackets on the RHS of (131) is concave in $\alpha$ for every $R_{XY}$ and because the pointwise minimum preserves the concavity. □

Lemma 9.

The mapping $\alpha \mapsto J_{\alpha }(X;Y)$ is nondecreasing on $[0,\infty ]$.

Proof. 

This is true because for every $\alpha ,{\alpha }^{\prime }\in [0,\infty ]$ with $\alpha \le {\alpha }^{\prime }$,

$$\begin{array}{c}\hfill \underset{Q_X,Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\le \underset{Q_X,Q_Y}{min}{D}_{{\alpha }^{\prime }}(P_{XY}\parallel Q_X{Q}_Y),\end{array}$$

(143)

which holds because the Rényi divergence is nondecreasing in $\alpha$ (Proposition 4). □

Lemma 10.

The mapping $\alpha \mapsto J_{\alpha }(X;Y)$ is continuous on $[0,\infty ]$.

Proof. 

By Lemma 8, the mapping $\alpha \mapsto (1-\alpha )J_{\alpha }(X;Y)$ is concave on $(0,\infty )$, thus it is continuous on $(0,\infty )$, which implies that $\alpha \mapsto J_{\alpha }(X;Y)$ is continuous on $(0,1)\cup (1,\infty )$.

We next prove the continuity at $\alpha =0$. Let $Q_X^{*}$ and $Q_Y^{*}$ be PMFs that achieve the minimum in the definition of $J_0(X;Y)$. Then, for all $\alpha \ge 0$,

$$\begin{array}{cc}\hfill D_0(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})& =J_0(X;Y)\hfill \end{array}$$

(144)

$$\begin{array}{c}\hfill \le J_{\alpha }(X;Y)\end{array}$$

(145)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*}),\end{array}$$

(146)

where (145) holds because $\alpha \mapsto J_{\alpha }(X;Y)$ is nondecreasing (Lemma 9), and (146) holds by the definition of $J_{\alpha }(X;Y)$. The Rényi divergence is continuous in $\alpha$ (Proposition 4), so (144)–(146) and the sandwich theorem imply that $J_{\alpha }(X;Y)$ is continuous at $\alpha =0$.

We continue with the continuity at $\alpha =\infty$. Define

$$\begin{array}{c}\hfill \tau \triangleq \underset{(x,y)\in supp\left(P_{XY}\right)}{min}P(x,y).\end{array}$$

(147)

Then, for all $\alpha >1$,

$$\begin{array}{cc}\hfill J_{\infty }(X;Y)& \ge J_{\alpha }(X;Y)\hfill \end{array}$$

(148)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}\frac{1}{\alpha -1}log\sum _{x,y}P(x,y)\frac{P{(x,y)}^{\alpha -1}}{{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\alpha -1}}\end{array}$$

(149)

$$\begin{array}{c}\ge \underset{Q_X,Q_Y}{min}\frac{1}{\alpha -1}log\underset{x,y}{max}\frac{\tau P{(x,y)}^{\alpha -1}}{{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\alpha -1}}\hfill \end{array}$$

(150)

$$\begin{array}{c}=\frac{1}{\alpha -1}log\tau +\underset{Q_X,Q_Y}{min}log\underset{x,y}{max}\frac{P(x,y)}{Q_X\left(x\right)Q_Y\left(y\right)}\hfill \end{array}$$

(151)

$$\begin{array}{c}\hfill =\frac{1}{\alpha -1}log\tau +J_{\infty }(X;Y),\end{array}$$

(152)

where (148) holds because $\alpha \mapsto J_{\alpha }(X;Y)$ is nondecreasing (Lemma 9), and (149) and (152) hold by the definitions of $J_{\alpha }(X;Y)$ and the Rényi divergence. The RHS of (152) tends to $J_{\infty }(X;Y)$ as $\alpha$ tends to infinity, so $J_{\alpha }(X;Y)$ is continuous at $\alpha =\infty$ by the sandwich theorem.

It remains to show the continuity at $\alpha =1$. Let $\alpha \in (\frac{3}{4},1)\cup (1,\frac{5}{4})$, and let $\delta \triangleq \left|1-\alpha \right|\in (0,\frac{1}{4})$. Then, for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{cc}\hfill 2^{-\delta D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)}& \le 2^{-\delta D_{1-\delta }(P_{XY}\parallel Q_X{Q}_Y)}\hfill \end{array}$$

(153)

$$\begin{array}{c}=\sum _{x,y}P(x,y){\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P(x,y)}\right]}^{\delta }\hfill \end{array}$$

(154)

$$\begin{array}{c}\hfill =\sum _{x,y}P(x,y){\left[\frac{P_X\left(x\right)P_Y\left(y\right)}{P(x,y)}\right]}^{\delta }{\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P_X\left(x\right)P_Y\left(y\right)}\right]}^{\delta }\end{array}$$

(155)

$$\begin{array}{c}\hfill \le {\left\{\sum _{x,y}P(x,y){\left[\frac{P_X\left(x\right)P_Y\left(y\right)}{P(x,y)}\right]}^{2\delta }\right\}}^{\frac{1}{2}}·{\left\{\sum _{x,y}P(x,y){\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P_X\left(x\right)P_Y\left(y\right)}\right]}^{2\delta }\right\}}^{\frac{1}{2}}\end{array}$$

(156)

$$\begin{array}{c}\hfill \le {\left\{\sum _{x,y}P(x,y){\left[\frac{P_X\left(x\right)P_Y\left(y\right)}{P(x,y)}\right]}^{2\delta }\right\}}^{\frac{1}{2}}\end{array}$$

(157)

$$\begin{array}{c}\hfill =2^{-\delta D_{1-2\delta }(P_{XY}\parallel P_X{P}_Y)},\end{array}$$

(158)

where (153) holds because $1-\delta \le \alpha$ and because the Rényi divergence is nondecreasing in $\alpha$ (Proposition 4); (156) follows from the Cauchy–Schwarz inequality; and (157) holds because

$$\begin{array}{cc}\hfill {\left\{\sum _{x,y}P(x,y){\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P_X\left(x\right)P_Y\left(y\right)}\right]}^{2\delta }\right\}}^{\frac{1}{2}}& \le {\left\{\sum _x{P}_X\left(x\right){\left[\frac{Q_X\left(x\right)}{P_X\left(x\right)}\right]}^{4\delta }\right\}}^{\frac{1}{4}}·{\left\{\sum _y{P}_Y\left(y\right){\left[\frac{Q_Y\left(y\right)}{P_Y\left(y\right)}\right]}^{4\delta }\right\}}^{\frac{1}{4}}\hfill \end{array}$$

(159)

$$\begin{array}{c}\hfill =2^{-\delta D_{1-4\delta }(P_X\parallel Q_X)}·2^{-\delta D_{1-4\delta }(P_Y\parallel Q_Y)}\end{array}$$

(160)

$$\begin{array}{c}\hfill \le 1,\end{array}$$

(161)

where (159) follows from the Cauchy–Schwarz inequality, and (161) holds because $1-4\delta >0$ and because the Rényi divergence is nonnegative for positive orders (Proposition 4). Thus, for all $\alpha \in (\frac{3}{4},\frac{5}{4})$,

$$\begin{array}{cc}\hfill D_{1-2\left|1-\alpha \right|}(P_{XY}\parallel P_X{P}_Y)& \le \underset{Q_X,Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(162)

$$\begin{array}{c}\hfill =J_{\alpha }(X;Y)\end{array}$$

(163)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel P_X{P}_Y),\end{array}$$

(164)

where (162) follows from (158) if $\alpha \ne 1$ and from Proposition 8 if $\alpha =1$; and (164) holds by the definition of $J_{\alpha }(X;Y)$. The Rényi divergence is continuous in $\alpha$ (Proposition 4), thus (162)–(164) and the sandwich theorem imply that $J_{\alpha }(X;Y)$ is continuous at $\alpha =1$. □

Lemma 11.

If $X=Y$ with probability one, then

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\left\{\begin{array}{cc}\frac{\alpha }{1-\alpha }{H}_{\infty }\left(X\right)\hfill & if\alpha \in [0,\frac{1}{2}],\hfill \\ H_{\frac{\alpha }{2\alpha -1}}\left(X\right)\hfill & if\alpha >\frac{1}{2},\hfill \\ H_{\frac{1}{2}}\left(X\right)\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(165)

Proof. 

We show below that (165) holds for $\alpha \in (0,1)\cup (1,\infty )$. Thus, (165) holds also for $\alpha \in \{0,1,\infty \}$ because both its sides are continuous in $\alpha$: its LHS by Lemma 10, and its RHS by the continuity of the Rényi entropy (Proposition 3).

Fix $\alpha \in (0,1)\cup (1,\infty )$. Then,

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\underset{Q_X}{min}\underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\hfill \end{array}$$

(166)

$$\begin{array}{c}=\underset{Q_X}{min}\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\hfill \end{array}$$

(167)

$$\begin{array}{c}\hfill =\underset{Q_X}{min}\frac{\alpha }{\alpha -1}log\sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }},\end{array}$$

(168)

where (166) follows from Proposition 9, and (168) holds because

$$\begin{array}{c}\hfill P_{XY}(x,y)=\left\{\begin{array}{cc}{P}_X\left(x\right)\hfill & \mathrm{if}x=y,\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(169)

First consider the case $\alpha >\frac{1}{2}$. Define $\gamma \triangleq {\sum }_x{P}_X{\left(x\right)}^{\frac{\alpha }{2\alpha -1}}$. Then, for all $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$,

$$\begin{array}{cc}\hfill \frac{\alpha }{\alpha -1}log\sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}& =\frac{\alpha }{\alpha -1}log\sum _x{\left[\gamma {\gamma }^{-1}{P}_X{\left(x\right)}^{\frac{\alpha }{2\alpha -1}}\right]}^{\frac{2\alpha -1}{\alpha }}{Q}_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}\hfill \end{array}$$

(170)

$$\begin{array}{c}\hfill =\frac{2\alpha -1}{\alpha -1}log\gamma +D_{\frac{2\alpha -1}{\alpha }}({\gamma }^{-1}{P}_X{}^{\frac{\alpha }{2\alpha -1}}\parallel Q_X)\end{array}$$

(171)

$$\begin{array}{c}\hfill =H_{\frac{\alpha }{2\alpha -1}}\left(X\right)+D_{\frac{2\alpha -1}{\alpha }}({\gamma }^{-1}{P}_X{}^{\frac{\alpha }{2\alpha -1}}\parallel Q_X),\end{array}$$

(172)

where (171) holds because $x\mapsto {\gamma }^{-1}{P}_X{\left(x\right)}^{\frac{\alpha }{2\alpha -1}}$ is a PMF. Because $\frac{2\alpha -1}{\alpha }>0$, Proposition 4 implies that $D_{(2\alpha -1)/\alpha }(P\parallel Q)\ge 0$ with equality if $Q=P$. This together with (168) and (172) establishes (165).

Now consider the case $\alpha \in (0,\frac{1}{2}]$. For all $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$,

$$\begin{array}{cc}\hfill \sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}& \le \sum _x{P}_X\left(x\right)Q_X\left(x\right)\hfill \end{array}$$

(173)

$$\begin{array}{c}\hfill \le \sum _x\left[\underset{x^{\prime }}{max}{P}_X\left(x^{\prime }\right)\right]Q_X\left(x\right)\end{array}$$

(174)

$$\begin{array}{c}\hfill =\underset{x}{max}{P}_X\left(x\right),\end{array}$$

(175)

where (173) holds because $Q_X\left(x\right)\in [0,1]$ for all $x\in \mathcal{X}$ and because $\frac{1-\alpha }{\alpha }\ge 1$. The inequalities (173) and (174) both hold with equality when $Q_X\left(x\right)=𝟙\{x=x^{*}\}$, where $x^{*}\in \mathcal{X}$ is such that $P_X\left(x^{*}\right)={max}_x{P}_X\left(x\right)$. Thus,

$$\begin{array}{c}\hfill \underset{Q_X}{max}\sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}=\underset{x}{max}{P}_X\left(x\right).\end{array}$$

(176)

Now (165) follows:

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\underset{Q_X}{min}\frac{\alpha }{\alpha -1}log\sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}\hfill \end{array}$$

(177)

$$\begin{array}{c}=\frac{\alpha }{\alpha -1}log\underset{Q_X}{max}\sum _x{P}_X\left(x\right)Q_X{\left(x\right)}^{\frac{1-\alpha }{\alpha }}\hfill \end{array}$$

(178)

$$\begin{array}{c}\hfill =\frac{\alpha }{\alpha -1}log\underset{x}{max}{P}_X\left(x\right)\end{array}$$

(179)

$$\begin{array}{c}\hfill =\frac{\alpha }{1-\alpha }{H}_{\infty }\left(X\right),\end{array}$$

(180)

where (177) follows from (168); (178) holds because $\frac{\alpha }{\alpha -1}<0$; (179) follows from (176); and (180) follows from the definition of $H_{\infty }\left(X\right)$. □

Lemma 12.

If the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ are independent, then $J_{\alpha }(X_1,X_2;Y_1,Y_2)=J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2)$ for all $\alpha \in [0,\infty ]$ (additivity).

Proof. 

Let the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ be independent. For $\alpha \in (0,1)\cup (1,\infty )$, we establish the lemma by showing the following two inequalities:

$$\begin{array}{cc}\hfill J_{\alpha }(X_1,X_2;Y_1,Y_2)& \le J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2),\hfill \end{array}$$

(181)

$$\begin{array}{cc}\hfill J_{\alpha }(X_1,X_2;Y_1,Y_2)& \ge J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2).\hfill \end{array}$$

(182)

Because $J_{\alpha }(X;Y)$ is continuous in $\alpha$ (Lemma 10), this will also establish the lemma for $\alpha \in \{0,1,\infty \}$.

To show (181), let $Q_{X_1}^{*}$ and $Q_{Y_1}^{*}$ be PMFs that achieve the minimum in the definition of $J_{\alpha }(X_1;Y_1)$, and let $Q_{X_2}^{*}$ and $Q_{Y_2}^{*}$ be PMFs that achieve the minimum in the definition of $J_{\alpha }(X_2;Y_2)$, so

$$\begin{array}{cc}\hfill J_{\alpha }(X_1;Y_1)& =D_{\alpha }(P_{X_1{Y}_1}\parallel Q_{X_1}^{*}{Q}_{Y_1}^{*}),\hfill \end{array}$$

(183)

$$\begin{array}{cc}\hfill J_{\alpha }(X_2;Y_2)& =D_{\alpha }(P_{X_2{Y}_2}\parallel Q_{X_2}^{*}{Q}_{Y_2}^{*}).\hfill \end{array}$$

(184)

Then, (181) holds because

$$\begin{array}{cc}\hfill J_{\alpha }(X_1,X_2;Y_1,Y_2)& \le D_{\alpha }(P_{X_1{X}_2{Y}_1{Y}_2}\parallel Q_{X_1}^{*}{Q}_{X_2}^{*}{Q}_{Y_1}^{*}{Q}_{Y_2}^{*})\hfill \end{array}$$

(185)

$$\begin{array}{c}=D_{\alpha }(P_{X_1{Y}_1}\parallel Q_{X_1}^{*}{Q}_{Y_1}^{*})+D_{\alpha }(P_{X_2{Y}_2}\parallel Q_{X_2}^{*}{Q}_{Y_2}^{*})\hfill \end{array}$$

(186)

$$\begin{array}{c}\hfill =J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2),\end{array}$$

(187)

where (185) holds by the definition of $J_{\alpha }(X_1,X_2;Y_1,Y_2)$ as a minimum; (186) follows from a simple computation using the independence hypothesis $P_{X_1{X}_2{Y}_1{Y}_2}=P_{X_1{Y}_1}{P}_{X_2{Y}_2}$; and (187) follows from (183) and (184).

To establish (182), we consider the cases $\alpha >1$ and $\alpha <1$ separately, starting with $\alpha >1$. Let ${\widehat{Q}}_{X_1{X}_2}$ and ${\widehat{Q}}_{Y_1{Y}_2}$ be PMFs that achieve the minimum in the definition of $J_{\alpha }(X_1,X_2;Y_1,Y_2)$, so

$$\begin{array}{c}\hfill J_{\alpha }(X_1,X_2;Y_1,Y_2)=D_{\alpha }(P_{X_1{X}_2{Y}_1{Y}_2}\parallel {\widehat{Q}}_{X_1{X}_2}{\widehat{Q}}_{Y_1{Y}_2}).\end{array}$$

(188)

Define the function $f:{\mathcal{X}}_1\times {\mathcal{Y}}_1\to \mathbb{R}\cup \{\infty \}$ as

$$\begin{array}{c}\hfill f(x_1,y_1)\triangleq \sum _{x_2,y_2}{P}_{X_2{Y}_2}{(x_2,y_2)}^{\alpha }{\left[{\widehat{Q}}_{X_2|X_1}\left(x_2\right|x_1){\widehat{Q}}_{Y_2|Y_1}\left(y_2\right|y_1)\right]}^{1-\alpha },\end{array}$$

(189)

and let $(x_1^{\prime },y_1^{\prime })\in {\mathcal{X}}_1\times {\mathcal{Y}}_1$ be such that

$$\begin{array}{c}\hfill f(x_1^{\prime },y_1^{\prime })=\underset{x_1,y_1}{min}f(x_1,y_1).\end{array}$$

(190)

Define the PMFs $Q_{X_2}^{\prime }$ and $Q_{Y_2}^{\prime }$ as

$$\begin{array}{cc}\hfill Q_{X_2}^{\prime }\left(x_2\right)& \triangleq {\widehat{Q}}_{X_2|X_1}\left(x_2\right|x_1^{\prime }),\hfill \end{array}$$

(191)

$$\begin{array}{cc}\hfill Q_{Y_2}^{\prime }\left(y_2\right)& \triangleq {\widehat{Q}}_{Y_2|Y_1}\left(y_2\right|y_1^{\prime }).\hfill \end{array}$$

(192)

Then,

$$\begin{array}{cc}\hfill 2^{(\alpha -1)J_{\alpha }(X_1,X_2;Y_1,Y_2)}& =2^{(\alpha -1)D_{\alpha }(P_{X_1{X}_2{Y}_1{Y}_2}\parallel {\widehat{Q}}_{X_1{X}_2}{\widehat{Q}}_{Y_1{Y}_2})}\hfill \end{array}$$

(193)

$$\begin{array}{c}=\sum _{x_1,x_2,y_1,y_2}{\left[P_{X_1{Y}_1}(x_1,y_1)P_{X_2{Y}_2}(x_2,y_2)\right]}^{\alpha }{\left[{\widehat{Q}}_{X_1{X}_2}(x_1,x_2){\widehat{Q}}_{Y_1{Y}_2}(y_1,y_2)\right]}^{1-\alpha }\hfill \end{array}$$

(194)

$$\begin{array}{c}\hfill =\sum _{x_1,y_1}{P}_{X_1{Y}_1}{(x_1,y_1)}^{\alpha }{\left[{\widehat{Q}}_{X_1}\left(x_1\right){\widehat{Q}}_{Y_1}\left(y_1\right)\right]}^{1-\alpha }f(x_1,y_1)\end{array}$$

(195)

$$\begin{array}{c}\hfill \ge \sum _{x_1,y_1}{P}_{X_1{Y}_1}{(x_1,y_1)}^{\alpha }{\left[{\widehat{Q}}_{X_1}\left(x_1\right){\widehat{Q}}_{Y_1}\left(y_1\right)\right]}^{1-\alpha }f(x_1^{\prime },y_1^{\prime })\end{array}$$

(196)

$$\begin{array}{c}\hfill =2^{(\alpha -1)D_{\alpha }(P_{X_1{Y}_1}\parallel {\widehat{Q}}_{X_1}{\widehat{Q}}_{Y_1})+(\alpha -1)D_{\alpha }(P_{X_2{Y}_2}\parallel Q_{X_2}^{\prime }{Q}_{Y_2}^{\prime })},\end{array}$$

(197)

where (193) follows from (188); (194) holds by the independence hypothesis $P_{X_1{X}_2{Y}_1{Y}_2}=P_{X_1{Y}_1}{P}_{X_2{Y}_2}$; (195) follows from (189); (196) follows from (190); and (197) follows from (191) and (192). Taking the logarithm and multiplying by $\frac{1}{\alpha -1}>0$ establishes (182):

$$\begin{array}{cc}\hfill J_{\alpha }(X_1,X_2;Y_1,Y_2)& \ge D_{\alpha }(P_{X_1{Y}_1}\parallel {\widehat{Q}}_{X_1}{\widehat{Q}}_{Y_1})+D_{\alpha }(P_{X_2{Y}_2}\parallel Q_{X_2}^{\prime }{Q}_{Y_2}^{\prime })\hfill \end{array}$$

(198)

$$\begin{array}{c}\hfill \ge J_{\alpha }(X_1;Y_1)+J_{\alpha }(X_2;Y_2),\end{array}$$

(199)

where (199) holds by the definition of $J_{\alpha }(X_1;Y_1)$ and $J_{\alpha }(X_2;Y_2)$.

The proof of (182) for $\alpha \in (0,1)$ is essentially the same as for $\alpha >1$: Replace the minimum in (190) by a maximum. Inequality (196) is then reversed, but (198) continues to hold because $\frac{1}{\alpha -1}<0$. Inequality (199) also continues to hold, and (198) and (199) together imply (182).

Lemma 13.

For all $\alpha \in [0,\infty ]$, $J_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$ with equality if and only if $(\alpha \in [\frac{1}{2},\infty ]$, X is distributed uniformly over $\mathcal{X}$, and $H\left(X\right|Y)=0)$.

Proof. 

Throughout the proof, define $X^{\prime }\triangleq X$. We first show that $J_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$ for all $\alpha \in [0,\infty ]$:

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& \le J_{\alpha }(X;X^{\prime })\hfill \end{array}$$

(200)

$$\begin{array}{c}\le J_{\infty }(X;X^{\prime })\hfill \end{array}$$

(201)

$$\begin{array}{c}\hfill =H_{\frac{1}{2}}\left(X\right)\end{array}$$

(202)

$$\begin{array}{c}\hfill \le log\left|\mathcal{X}\right|,\end{array}$$

(203)

where (200) follows from the data-processing inequality (Lemma 4) because $X⊸--X^{\prime }⊸--Y$ form a Markov chain; (201) holds because $J_{\alpha }(X;X^{\prime })$ is nondecreasing in $\alpha$ (Lemma 9); (202) follows from Lemma 11; and (203) follows from Proposition 3.

We now show that (200)–(203) can hold with equality only if the following conditions all hold:

(1)

$\alpha \in [\frac{1}{2},\infty ]$;

(ii)

X is distributed uniformly over $\mathcal{X}$; and

(iii)

$H\left(X\right|Y)=0$, i.e., for every $y\in supp\left(P_Y\right)$, there exists an $x\in \mathcal{X}$ for which $P\left(x\right|y)=1$.

Indeed, if $\alpha <\frac{1}{2}$, then Lemma 11 implies that

$$\begin{array}{c}\hfill J_{\alpha }(X;X^{\prime })=\frac{\alpha }{1-\alpha }{H}_{\infty }\left(X\right).\end{array}$$

(204)

Because $\frac{\alpha }{1-\alpha }<1$ for such $\alpha$’s and because $H_{\infty }\left(X\right)\le log\left|\mathcal{X}\right|$ (Proposition 3), the RHS of (204) is strictly smaller than $log\left|\mathcal{X}\right|$. This, together with (200), shows that Part (i) is a necessary condition. The necessity of Part (ii) follows from (203): if X is not distributed uniformly over $\mathcal{X}$, then (203) holds with strict inequality (Proposition 3). As to the necessity of Part (iii),

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& \le J_{\infty }(X;Y)\hfill \end{array}$$

(205)

$$\begin{array}{c}\hfill =\underset{Q_X}{min}\underset{Q_Y}{min}{D}_{\infty }(P_{XY}\parallel Q_X{Q}_Y)\end{array}$$

(206)

$$\begin{array}{c}\hfill =\underset{Q_X}{min}log\sum _y\underset{x}{max}\frac{P(x,y)}{Q_X\left(x\right)}\end{array}$$

(207)

$$\begin{array}{c}\hfill \le log\sum _y\underset{x}{max}\frac{P\left(y\right)P\left(x\right|y)}{1/\left|\mathcal{X}\right|}\end{array}$$

(208)

$$\begin{array}{c}\hfill =log\left|\mathcal{X}\right|+log\sum _{y}P\left(y\right)\underset{x}{max}P\left(x\right|y)\end{array}$$

(209)

$$\begin{array}{c}\hfill \le log\left|\mathcal{X}\right|,\end{array}$$

(210)

where (205) holds because $J_{\alpha }(X;Y)$ is nondecreasing in $\alpha$ (Lemma 9); (207) follows from Proposition 9; and (208) follows from choosing $Q_X$ to be the uniform distribution. The inequality (210) is strict when Part (iii) does not hold, so Part (iii) is a necessary condition.

It remains to show that when Parts (i)–(iii) all hold, $J_{\alpha }(X;Y)=log\left|\mathcal{X}\right|$. By (203), $J_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$ always holds, so it suffices to show that Parts (i)–(iii) together imply $J_{\alpha }(X;Y)\ge log\left|\mathcal{X}\right|$. Indeed,

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& \ge J_{\frac{1}{2}}(X;Y)\hfill \end{array}$$

(211)

$$\begin{array}{c}\hfill \ge J_{\frac{1}{2}}(X;X^{\prime })\end{array}$$

(212)

$$\begin{array}{c}\hfill =H_{\infty }\left(X\right)\end{array}$$

(213)

$$\begin{array}{c}\hfill =log\left|\mathcal{X}\right|,\end{array}$$

(214)

where (211) holds because Part (i) implies that $\alpha \ge \frac{1}{2}$ and because $J_{\alpha }(X;Y)$ is nondecreasing in $\alpha$ (Lemma 9); (212) follows from the data-processing inequality (Lemma 4) because Part (iii) implies that $X⊸--Y⊸--X^{\prime }$ form a Markov chain; (213) follows from Lemma 11; and (214) follows from Part (ii). □

Lemma 14.

For every $\alpha \in [1,\infty ]$, $J_{\alpha }(X;Y)$ is concave in $P_X$ for fixed $P_{Y|X}$.

Proof. 

We prove the claim for $\alpha \in (1,\infty )$; for $\alpha \in \{1,\infty \}$ the claim will then hold because $J_{\alpha }(X;Y)$ is continuous in $\alpha$ (Lemma 10).

Fix $\alpha \in (1,\infty )$. Let $\lambda ,{\lambda }^{\prime }\in [0,1]$ with $\lambda +{\lambda }^{\prime }=1$, let $P_X$ and $P_X^{\prime }$ be PMFs, let $P_{Y|X}$ be a conditional PMF, and define $f:\mathcal{X}\times \mathcal{P}\left(\mathcal{Y}\right)\to \mathbb{R}\cup \{\infty \}$ as

$$\begin{array}{c}\hfill f(x,Q_Y)\triangleq {\left[\sum _y{P}_{Y|X}{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}.\end{array}$$

(215)

Denoting $J_{\alpha }(X;Y)$ by $J_{\alpha }\left(P_X{P}_{Y|X}\right)$,

$$\begin{array}{c}{J}_{\alpha }\left((\lambda P_X+{\lambda }^{\prime }{P}_X^{\prime })P_{Y|X}\right)\hfill \end{array}$$

$$\begin{array}{c}=\underset{Q_Y}{min}\underset{Q_X}{min}{D}_{\alpha }\left((\lambda P_X+{\lambda }^{\prime }{P}_X^{\prime })P_{Y|X}\parallel Q_X{Q}_Y\right)\hfill \end{array}$$

(216)

$$\begin{array}{c}=\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _x{\left[\sum _y{[\lambda P_X\left(x\right)+{\lambda }^{\prime }{P}_X^{\prime }\left(x\right)]}^{\alpha }{P}_{Y|X}{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\hfill \end{array}$$

(217)

$$\begin{array}{c}\hfill =\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _x[\lambda P_X\left(x\right)+{\lambda }^{\prime }{P}_X^{\prime }\left(x\right)]{\left[\sum _y{P}_{Y|X}{\left(y\right|x)}^{\alpha }{Q}_Y{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\end{array}$$

(218)

$$\begin{array}{c}\hfill =\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log[\lambda \sum _x{P}_X\left(x\right)f(x,Q_Y)+{\lambda }^{\prime }\sum _x{P}_X^{\prime }\left(x\right)f(x,Q_Y)]\end{array}$$

(219)

$$\begin{array}{c}\hfill \ge \underset{Q_Y}{min}\frac{\alpha }{\alpha -1}[\lambda log\sum _x{P}_X\left(x\right)f(x,Q_Y)+{\lambda }^{\prime }log\sum _x{P}_X^{\prime }\left(x\right)f(x,Q_Y)]\end{array}$$

(220)

$$\begin{array}{c}\hfill \ge \lambda \underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _x{P}_X\left(x\right)f(x,Q_Y)+{\lambda }^{\prime }\underset{Q_Y}{min}\frac{\alpha }{\alpha -1}log\sum _x{P}_X^{\prime }\left(x\right)f(x,Q_Y)\end{array}$$

(221)

$$\begin{array}{c}\hfill =\lambda J_{\alpha }\left(P_X{P}_{Y|X}\right)+{\lambda }^{\prime }{J}_{\alpha }\left(P_X^{\prime }{P}_{Y|X}\right),\end{array}$$

(222)

where (217) follows from Proposition 9 with the roles of $Q_X$ and $Q_Y$ swapped; (220) holds because $log(·)$ is concave; (221) holds because optimizing $Q_Y$ separately cannot be worse than optimizing a common $Q_Y$; and (222) can be established using steps similar to (216)–(218). □

Lemma 15.

For every $\alpha \in [\frac{1}{2},\infty ]$, the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is convex, i.e., for all $\lambda ,{\lambda }^{\prime }\in [0,1]$ with $\lambda +{\lambda }^{\prime }=1$, all $Q_X,Q_X^{\prime }\in \mathcal{P}\left(\mathcal{X}\right)$, and all $Q_Y,Q_Y^{\prime }\in \mathcal{P}\left(\mathcal{Y}\right)$,

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{XY}\parallel (\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })(\lambda Q_Y+{\lambda }^{\prime }{Q}_Y^{\prime })\right)\le \lambda D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)+{\lambda }^{\prime }{D}_{\alpha }(P_{XY}\parallel Q_X^{\prime }{Q}_Y^{\prime }).\end{array}$$

(223)

For $\alpha \in [0,\frac{1}{2})$, the mapping need not be convex.

Proof. 

We establish (223) for $\alpha \in [\frac{1}{2},1)$ and for $\alpha \in (1,\infty )$, which also establishes (223) for $\alpha \in \{1,\infty \}$ because the Rényi divergence is continuous in $\alpha$ (Proposition 4). Afterwards, we provide an example where (223) is violated for all $\alpha \in [0,\frac{1}{2})$.

We begin with the case where $\alpha \in [\frac{1}{2},1)$:

$$\begin{array}{c}{2}^{(\alpha -1)\lambda D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)+(\alpha -1){\lambda }^{\prime }{D}_{\alpha }(P_{XY}\parallel Q_X^{\prime }{Q}_Y^{\prime })}\hfill \\ ={\left[\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{1-\alpha }\right]}^{\lambda }·{\left[\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X^{\prime }\left(x\right)Q_Y^{\prime }\left(y\right)\right]}^{1-\alpha }\right]}^{{\lambda }^{\prime }}\hfill \end{array}$$

(224)

$$\begin{array}{c}\hfill \le \lambda \sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{1-\alpha }+{\lambda }^{\prime }\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X^{\prime }\left(x\right)Q_Y^{\prime }\left(y\right)\right]}^{1-\alpha }\end{array}$$

(225)

$$\begin{array}{c}\hfill =\sum _{x,y}P{(x,y)}^{\alpha }[\sqrt{\lambda }{Q}_X{\left(x\right)}^{1-\alpha }\sqrt{\lambda }{Q}_Y{\left(y\right)}^{1-\alpha }+\sqrt{{\lambda }^{\prime }}{Q}_X^{\prime }{\left(x\right)}^{1-\alpha }\sqrt{{\lambda }^{\prime }}{Q}_Y^{\prime }{\left(y\right)}^{1-\alpha }]\end{array}$$

(226)

$$\begin{array}{c}\hfill \le \sum _{x,y}P{(x,y)}^{\alpha }\sqrt{\lambda Q_X{\left(x\right)}^{2(1-\alpha )}+{\lambda }^{\prime }{Q}_X^{\prime }{\left(x\right)}^{2(1-\alpha )}}\sqrt{\lambda Q_Y{\left(y\right)}^{2(1-\alpha )}+{\lambda }^{\prime }{Q}_Y^{\prime }{\left(y\right)}^{2(1-\alpha )}}\end{array}$$

(227)

$$\begin{array}{c}\hfill \le \sum _{x,y}P{(x,y)}^{\alpha }{[\lambda Q_X\left(x\right)+{\lambda }^{\prime }{Q}_X^{\prime }\left(x\right)]}^{1-\alpha }\sqrt{\lambda Q_Y{\left(y\right)}^{2(1-\alpha )}+{\lambda }^{\prime }{Q}_Y^{\prime }{\left(y\right)}^{2(1-\alpha )}}\end{array}$$

(228)

$$\begin{array}{c}\hfill \le \sum _{x,y}P{(x,y)}^{\alpha }{[\lambda Q_X\left(x\right)+{\lambda }^{\prime }{Q}_X^{\prime }\left(x\right)]}^{1-\alpha }{[\lambda Q_Y\left(y\right)+{\lambda }^{\prime }{Q}_Y^{\prime }\left(y\right)]}^{1-\alpha }\end{array}$$

(229)

$$\begin{array}{c}=2^{(\alpha -1)D_{\alpha }(P_{XY}\parallel (\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })(\lambda Q_Y+{\lambda }^{\prime }{Q}_Y^{\prime }))},\hfill \end{array}$$

(230)

where (225) follows from the arithmetic mean-geometric mean inequality; (227) follows from the Cauchy–Schwarz inequality; and (228) and (229) hold because the mapping $z\mapsto z^{2(1-\alpha )}$ is concave on ${\mathbb{R}}_{\ge 0}$ for $\alpha \in [\frac{1}{2},1)$. Taking the logarithm and multiplying by $\frac{1}{\alpha -1}<0$ establishes (223).

Now, consider $\alpha \in (1,\infty )$. Then,

$$\begin{array}{c}{2}^{(\alpha -1)D_{\alpha }(P_{XY}\parallel (\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })(\lambda Q_Y+{\lambda }^{\prime }{Q}_Y^{\prime }))}\hfill \\ =\sum _{x,y}P{(x,y)}^{\alpha }{[\lambda Q_X\left(x\right)+{\lambda }^{\prime }{Q}_X^{\prime }\left(x\right)]}^{1-\alpha }{[\lambda Q_Y\left(y\right)+{\lambda }^{\prime }{Q}_Y^{\prime }\left(y\right)]}^{1-\alpha }\hfill \end{array}$$

(231)

$$\begin{array}{c}\hfill \le \sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X{\left(x\right)}^{\lambda }{Q}_X^{\prime }{\left(x\right)}^{{\lambda }^{\prime }}\right]}^{1-\alpha }{\left[Q_Y{\left(y\right)}^{\lambda }{Q}_Y^{\prime }{\left(y\right)}^{{\lambda }^{\prime }}\right]}^{1-\alpha }\end{array}$$

(232)

$$\begin{array}{c}\hfill =\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{(1-\alpha )\lambda }{\left[Q_X^{\prime }\left(x\right)Q_Y^{\prime }\left(y\right)\right]}^{(1-\alpha ){\lambda }^{\prime }}\end{array}$$

(233)

$$\begin{array}{c}\le {\left[\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{1-\alpha }\right]}^{\lambda }·{\left[\sum _{x,y}P{(x,y)}^{\alpha }{\left[Q_X^{\prime }\left(x\right)Q_Y^{\prime }\left(y\right)\right]}^{1-\alpha }\right]}^{{\lambda }^{\prime }}\hfill \end{array}$$

(234)

$$\begin{array}{c}=2^{(\alpha -1)\lambda D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)+(\alpha -1){\lambda }^{\prime }{D}_{\alpha }(P_{XY}\parallel Q_X^{\prime }{Q}_Y^{\prime })},\hfill \end{array}$$

(235)

where (232) follows from the arithmetic mean-geometric mean inequality and the fact that the mapping $z\mapsto z^{1-\alpha }$ is decreasing on ${\mathbb{R}}_{>0}$ for $\alpha >1$, and (233) follows from Hölder’s inequality. Taking the logarithm and multiplying by $\frac{1}{\alpha -1}>0$ establishes (223).

Finally, we show that the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ does not need to be convex for $\alpha \in [0,\frac{1}{2})$. Let X be uniformly distributed over $\{0,1\}$, and let $Y=X$. Then, for all $\alpha \in [0,\frac{1}{2})$,

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{XY}\parallel (0.5,0.5)(0.5,0.5)\right)>0.5D_{\alpha }\left(P_{XY}\parallel (1,0)(1,0)\right)+0.5D_{\alpha }\left(P_{XY}\parallel (0,1)(0,1)\right),\end{array}$$

(236)

because the LHS of (236) is equal to $log2$, and the RHS of (236) is equal to $\frac{\alpha }{1-\alpha }log2$. □

Lemma 16.

Let $\alpha \in (0,1)\cup (1,\infty )$. If $(Q_X^{*},Q_Y^{*})$ achieves the minimum in the definition of $J_{\alpha }(X;Y)$, then there exist positive normalization constants c and d such that

$$\begin{array}{cc}\hfill Q_X^{*}\left(x\right)& =c{\left[\sum _{y}P{(x,y)}^{\alpha }{Q}_Y^{*}{\left(y\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\forall x\in \mathcal{X},\hfill \end{array}$$

(237)

$$\begin{array}{cc}\hfill Q_Y^{*}\left(y\right)& =d{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X^{*}{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\forall y\in \mathcal{Y},\hfill \end{array}$$

(238)

with the conventions of (44). The case $\alpha =\infty$ is similar: if $(Q_X^{*},Q_Y^{*})$ achieves the minimum in the definition of $J_{\infty }(X;Y)$, then there exist positive normalization constants c and d such that

$$\begin{array}{cc}\hfill Q_X^{*}\left(x\right)& =c\underset{y}{max}\frac{P(x,y)}{Q_Y^{*}\left(y\right)}\forall x\in \mathcal{X},\hfill \end{array}$$

(239)

$$\begin{array}{cc}\hfill Q_Y^{*}\left(y\right)& =d\underset{x}{max}\frac{P(x,y)}{Q_X^{*}\left(x\right)}\forall y\in \mathcal{Y},\hfill \end{array}$$

(240)

with the conventions of (44). (If $\alpha =1$, then $Q_X^{*}=P_X$ and $Q_Y^{*}=P_Y$ by Proposition 8.) Thus, for all $\alpha \in (0,\infty ]$, both inclusions $supp\left(Q_X^{*}\right)\subseteq supp\left(P_X\right)$ and $supp\left(Q_Y^{*}\right)\subseteq supp\left(P_Y\right)$ hold.

Proof. 

If $(Q_X^{*},Q_Y^{*})$ achieves the minimum in the definition of $J_{\alpha }(X;Y)$, then

$$\begin{array}{c}\hfill \underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y)=D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*}).\end{array}$$

(241)

Hence, (238) and (240) follow from (76) and (78) of Proposition 9 because $D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})=J_{\alpha }(X;Y)$ is finite. Swapping the roles of $Q_X$ and $Q_Y$ establishes (237) and (239). For $\alpha \in (0,1)\cup (1,\infty )$ the claimed inclusions follow from (237) and (238); for $\alpha =\infty$ from (239) and (240); and for $\alpha =1$ from Proposition 8. □

Lemma 17.

For all $\alpha \in (0,\infty ]$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\underset{Q_X}{min}{\varphi }_{\alpha }\left(Q_X\right),\end{array}$$

(242)

where ${\varphi }_{\alpha }\left(Q_X\right)$ is defined as

$$\begin{array}{c}\hfill {\varphi }_{\alpha }\left(Q_X\right)\triangleq \underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)\end{array}$$

(243)

and is given explicitly as follows: for $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{c}\hfill {\varphi }_{\alpha }\left(Q_X\right)=\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }},\end{array}$$

(244)

with the conventions of (44); and for $\alpha \in \{1,\infty \}$,

$$\begin{array}{cc}\hfill {\varphi }_1\left(Q_X\right)& =D(P_{XY}\parallel Q_X{P}_Y),\hfill \end{array}$$

(245)

$$\begin{array}{cc}\hfill {\varphi }_{\infty }\left(Q_X\right)& =log\sum _y\underset{x}{max}\frac{P(x,y)}{Q_X\left(x\right)},\hfill \end{array}$$

(246)

with the conventions of (44). For every $\alpha \in [\frac{1}{2},\infty ]$, the mapping $Q_X\mapsto {\varphi }_{\alpha }\left(Q_X\right)$ is convex. For $\alpha \in (0,\frac{1}{2})$, the mapping need not be convex.

Proof. 

We first establish (242) and (244)–(246): (242) follows from the definition of $J_{\alpha }(X;Y)$; (244) and (246) follow from Proposition 9; and (245) holds because

$$\begin{array}{cc}\hfill \underset{Q_Y}{min}D(P_{XY}\parallel Q_X{Q}_Y)& =\underset{Q_Y}{min}[D(P_{XY}\parallel Q_X{P}_Y)+D(P_Y\parallel Q_Y)]\hfill \end{array}$$

(247)

$$\begin{array}{c}=D(P_{XY}\parallel Q_X{P}_Y),\hfill \end{array}$$

(248)

where (247) follows from a simple computation, and (248) holds because $D(P_Y\parallel Q_Y)\ge 0$ with equality if $Q_Y=P_Y$.

We now show that the mapping $Q_X\mapsto {\varphi }_{\alpha }\left(Q_X\right)$ is convex for every $\alpha \in [\frac{1}{2},\infty ]$. To that end, let $\alpha \in [\frac{1}{2},\infty ]$, let $\lambda ,{\lambda }^{\prime }\in [0,1]$ with $\lambda +{\lambda }^{\prime }=1$, and let $Q_X,Q_X^{\prime }\in \mathcal{P}\left(\mathcal{X}\right)$. Let ${\widehat{Q}}_Y$ and ${\widehat{Q}}_Y^{\prime }$ be PMFs that achieve the minimum in the definitions of ${\varphi }_{\alpha }\left(Q_X\right)$ and ${\varphi }_{\alpha }\left(Q_X^{\prime }\right)$, respectively. Then,

$$\begin{array}{cc}\hfill {\varphi }_{\alpha }(\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })& \le D_{\alpha }\left(P_{XY}\parallel (\lambda Q_X+{\lambda }^{\prime }{Q}_X^{\prime })(\lambda {\widehat{Q}}_Y+{\lambda }^{\prime }{\widehat{Q}}_Y^{\prime })\right)\hfill \end{array}$$

(249)

$$\begin{array}{c}\hfill \le \lambda D_{\alpha }(P_{XY}\parallel Q_X{\widehat{Q}}_Y)+{\lambda }^{\prime }{D}_{\alpha }(P_{XY}\parallel Q_X^{\prime }{\widehat{Q}}_Y^{\prime })\end{array}$$

(250)

$$\begin{array}{c}\hfill =\lambda {\varphi }_{\alpha }\left(Q_X\right)+{\lambda }^{\prime }{\varphi }_{\alpha }\left(Q_X^{\prime }\right),\end{array}$$

(251)

where (249) holds by the definition of ${\varphi }_{\alpha }(·)$; (250) holds because $D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is convex in the pair $(Q_X,Q_Y)$ for $\alpha \in [\frac{1}{2},\infty ]$ (Lemma 15); and (251) follows from our choice of ${\widehat{Q}}_Y$ and ${\widehat{Q}}_Y^{\prime }$.

Finally, we show that the mapping $Q_X\mapsto {\varphi }_{\alpha }\left(Q_X\right)$ need not be convex for $\alpha \in (0,\frac{1}{2})$. Let X be uniformly distributed over $\{0,1\}$, and let $Y=X$. Then, for all $\alpha \in (0,\frac{1}{2})$,

$$\begin{array}{c}\hfill {\varphi }_{\alpha }\left((0.5,0.5)\right)>0.5{\varphi }_{\alpha }\left((1,0)\right)+0.5{\varphi }_{\alpha }\left((0,1)\right),\end{array}$$

(252)

because the LHS of (252) is equal to $log2$, and the RHS of (252) is equal to $\frac{\alpha }{1-\alpha }log2$. □

Lemma 18.

For all $\alpha \in (0,1)\cup (1,\infty ]$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\left\{\begin{array}{cc}{\displaystyle \underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{min}{\psi }_{\alpha }\left(R_{XY}\right)}\hfill & if\alpha \in (0,1),\hfill \\ {\displaystyle \underset{R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})}{max}{\psi }_{\alpha }\left(R_{XY}\right)}\hfill & if\alpha \in (1,\infty ],\hfill \end{array}\right.\end{array}$$

(253)

where

$$\begin{array}{c}\hfill {\psi }_{\alpha }\left(R_{XY}\right)\triangleq \left\{\begin{array}{cc}{\displaystyle D(R_{XY}\parallel R_X{R}_Y)+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY})}\hfill & if\alpha \in (0,1)\cup (1,\infty ),\hfill \\ {\displaystyle D(R_{XY}\parallel R_X{R}_Y)-D(R_{XY}\parallel P_{XY})}\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(254)

For every $\alpha \in (1,\infty ]$, the mapping $R_{XY}\mapsto {\psi }_{\alpha }\left(R_{XY}\right)$ is concave. For all $\alpha \in (1,\infty ]$ and all $R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})$, the statement $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$ is equivalent to ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$.

Proof. 

For $\alpha \in (0,1)\cup (1,\infty )$, (253) follows from Lemma 8 by dividing by $1-\alpha$, which is positive or negative depending on whether $\alpha$ is smaller than or greater than one. For $\alpha =\infty$, we establish (253) as follows: By Lemma 10, its LHS is continuous at $\alpha =\infty$. We argue below that its RHS is continuous at $\alpha =\infty$, i.e., that

$$\begin{array}{c}\hfill \underset{\alpha \to \infty }{lim}\underset{R_{XY}}{max}{\psi }_{\alpha }\left(R_{XY}\right)=\underset{R_{XY}}{max}{\psi }_{\infty }\left(R_{XY}\right).\end{array}$$

(255)

Because (253) holds for $\alpha \in (1,\infty )$ and because both its sides are continuous at $\alpha =\infty$, it must also hold for $\alpha =\infty$.

We now establish (255). Let $R_{XY}^{*}$ be a PMF that achieves the maximum on the RHS of (255). Then, for all $\alpha >1$,

$$\begin{array}{cc}\hfill {\psi }_{\infty }\left(R_{XY}^{*}\right)& =\underset{R_{XY}}{max}{\psi }_{\infty }\left(R_{XY}\right)\hfill \end{array}$$

(256)

$$\begin{array}{c}\hfill \ge \underset{R_{XY}}{max}{\psi }_{\alpha }\left(R_{XY}\right)\end{array}$$

(257)

$$\begin{array}{c}\hfill \ge {\psi }_{\alpha }\left(R_{XY}^{*}\right),\end{array}$$

(258)

where (257) holds because, by (254), ${\psi }_{\infty }\left(R_{XY}\right)={\psi }_{\alpha }\left(R_{XY}\right)+\frac{1}{\alpha -1}D(R_{XY}\parallel P_{XY})\ge {\psi }_{\alpha }\left(R_{XY}\right)$ for all $R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})$. By (254), $\alpha \mapsto {\psi }_{\alpha }\left(R_{XY}^{*}\right)$ is continuous at $\alpha =\infty$, so the RHS of (258) approaches ${\psi }_{\infty }\left(R_{XY}^{*}\right)$ as $\alpha$ tends to infinity, and (255) follows from the sandwich theorem.

We now show that $R_{XY}\mapsto {\psi }_{\alpha }\left(R_{XY}\right)$ is concave for $\alpha \in (1,\infty ]$. A simple computation reveals that for all $\alpha \in (1,\infty )$,

$$\begin{array}{c}\hfill {\psi }_{\alpha }\left(R_{XY}\right)=H\left(R_X\right)+H\left(R_Y\right)+\frac{1}{\alpha -1}H\left(R_{XY}\right)+\frac{\alpha }{\alpha -1}\sum _{x,y}{R}_{XY}(x,y)logP(x,y).\end{array}$$

(259)

Because the entropy is a concave function and because a nonnegative weighted sum of concave functions is concave, this implies that ${\psi }_{\alpha }\left(R_{XY}\right)$ is concave in $R_{XY}$ for $\alpha \in (1,\infty )$. By (254), $\alpha \mapsto {\psi }_{\alpha }\left(R_{XY}\right)$ is continuous at $\alpha =\infty$, so ${\psi }_{\alpha }\left(R_{XY}\right)$ is concave in $R_{XY}$ also for $\alpha =\infty$.

We next show that if $\alpha \in (1,\infty ]$ and ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$, then $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$. Let $\alpha \in (1,\infty ]$, and let $R_{XY}$ be a PMF that satisfies ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$. Then,

$$\begin{array}{cc}\hfill {\psi }_{\alpha }\left(R_{XY}\right)& \le J_{\alpha }(X;Y)\hfill \end{array}$$

(260)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel R_X{R}_Y),\end{array}$$

(261)

where (260) follows from (253), and (261) holds by the definition of $J_{\alpha }(X;Y)$. Because ${\psi }_{\alpha }\left(R_{XY}\right)$ is equal to $D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$, both inequalities hold with equality, which implies the claim.

Finally, we show that if $\alpha \in (1,\infty ]$ and $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$, then ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$. We first consider $\alpha \in (1,\infty )$. Let $R_{XY}$ be a PMF that satisfies $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$, and let $Q_X^{*}$ and $Q_Y^{*}$ be PMFs that achieve the minimum in the definition of $J_{\alpha }(X;Y)$. Then,

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& ={\psi }_{\alpha }\left(R_{XY}\right)\hfill \end{array}$$

(262)

$$\begin{array}{c}=D(R_{XY}\parallel R_X{R}_Y)+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY})\hfill \end{array}$$

(263)

$$\begin{array}{c}\hfill \le D(R_{XY}\parallel Q_X^{*}{Q}_Y^{*})+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY})\end{array}$$

(264)

$$\begin{array}{c}\hfill \le D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})\end{array}$$

(265)

$$\begin{array}{c}\hfill =J_{\alpha }(X;Y),\end{array}$$

(266)

where (264) follows from Proposition 8, and (265) follows from [22] (Theorem 30). Thus, all inequalities hold with equality. Because (264) holds with equality, $Q_X^{*}=R_X$ and $Q_Y^{*}=R_Y$ by Proposition 8. Hence, ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$ as desired. We now consider $\alpha =\infty$. Here, (262)–(266) remain valid after replacing $\frac{\alpha }{1-\alpha }$ by $-1$. (Now, (265) follows from a short computation.) Consequently, ${\psi }_{\alpha }\left(R_{XY}\right)=D_{\alpha }(P_{XY}\parallel R_X{R}_Y)$ holds also for $\alpha =\infty$.

Lemma 19.

For all $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{c}\hfill J_{\alpha }(X;Y)=\underset{R_X\ll P_X}{min}\frac{1}{\alpha -1}\left[D_{\frac{\alpha }{\alpha -1}}(P_X\parallel R_X)-\alpha E_0\left(\frac{1-\alpha }{\alpha },R_X\right)\right],\end{array}$$

(267)

where the minimization is over all PMFs $R_X$ satisfying $R_X\ll P_X$ $\left(i.e.,supp\left(R_X\right)\subseteq supp\left(P_X\right)\right)$; $D_{\alpha }(P\parallel Q)$ for negative α is given by (54); and Gallager’s $E_0$ function [29] is defined as

$$\begin{array}{c}\hfill E_0(\rho ,R_X)\triangleq -log\sum _y{\left[\sum _x{R}_X\left(x\right)P{\left(y\right|x)}^{\frac{1}{1+\rho }}\right]}^{1+\rho }.\end{array}$$

(268)

Proof. 

Let $\alpha \in (0,1)\cup (1,\infty )$, and define the set $\mathcal{R}\triangleq \{R_X\in \mathcal{P}\left(\mathcal{X}\right):supp\left(R_X\right)\subseteq supp\left(P_X\right)\}$. We establish (267) by showing that for all $R_X\in \mathcal{R}$,

$$\begin{array}{c}\hfill \frac{1}{\alpha -1}\left[D_{\frac{\alpha }{\alpha -1}}(P_X\parallel R_X)-\alpha E_0\left(\frac{1-\alpha }{\alpha },R_X\right)\right]\ge J_{\alpha }(X;Y),\end{array}$$

(269)

with equality for some $R_X\in \mathcal{R}$.

Fix $R_X\in \mathcal{R}$. If the LHS of (269) is infinite, then (269) holds trivially. Otherwise, define the PMF ${\widehat{Q}}_X$ as

$$\begin{array}{c}\hfill {\widehat{Q}}_X\left(x\right)\triangleq \frac{P_X{\left(x\right)}^{\frac{\alpha }{\alpha -1}}{R}_X{\left(x\right)}^{\frac{1}{1-\alpha }}}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\frac{\alpha }{\alpha -1}}{R}_X{\left(x^{\prime }\right)}^{\frac{1}{1-\alpha }}},\end{array}$$

(270)

where we use the convention that $0^{\frac{\alpha }{\alpha -1}}·0^{\frac{1}{1-\alpha }}=0$. (The RHS of (270) is finite whenever the LHS of (269) is finite.) Then, (269) holds because

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\underset{Q_X}{min}\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{Q}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\hfill \end{array}$$

(271)

$$\begin{array}{c}\hfill \le \frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _{x}P{(x,y)}^{\alpha }{\widehat{Q}}_X{\left(x\right)}^{1-\alpha }\right]}^{\frac{1}{\alpha }}\end{array}$$

(272)

$$\begin{array}{c}\hfill =log\sum _x{P}_X{\left(x\right)}^{\frac{\alpha }{\alpha -1}}{R}_X{\left(x\right)}^{\frac{1}{1-\alpha }}+\frac{\alpha }{\alpha -1}log\sum _y{\left[\sum _x{R}_X\left(x\right)P{\left(y\right|x)}^{\alpha }\right]}^{\frac{1}{\alpha }}\end{array}$$

(273)

$$\begin{array}{c}=\frac{1}{\alpha -1}\left[D_{\frac{\alpha }{\alpha -1}}(P_X\parallel R_X)-\alpha E_0\left(\frac{1-\alpha }{\alpha },R_X\right)\right],\hfill \end{array}$$

(274)

where (271) follows from Lemma 17, and (273) follows from (270) using some algebra. It remains to show that there exists an $R_X\in \mathcal{R}$ for which (272) holds with equality. To that end, let $Q_X^{*}$ be a PMF that achieves the minimum on the RHS of (271), and define the PMF $R_X$ as

$$\begin{array}{c}\hfill R_X\left(x\right)\triangleq \frac{P_X{\left(x\right)}^{\alpha }{Q}_X^{*}{\left(x\right)}^{1-\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\alpha }{Q}_X^{*}{\left(x^{\prime }\right)}^{1-\alpha }},\end{array}$$

(275)

where we use the convention that $0^{\alpha }·0^{1-\alpha }=0$. Because $supp\left(Q_X^{*}\right)\subseteq supp\left(P_X\right)$ (Lemma 16), the definitions (275) and (270) imply that ${\widehat{Q}}_X=Q_X^{*}$. Hence, (272) holds with equality for this $R_X\in \mathcal{R}$.

Lemma 20.

For every $\alpha \in (\frac{1}{2},\infty ]$, the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ has a unique minimizer. This need not be the case when $\alpha \in [0,\frac{1}{2}]$.

Proof. 

First consider $\alpha \in (\frac{1}{2},1)$. Let $(Q_X^{*},Q_Y^{*})$ and $({\widehat{Q}}_X,{\widehat{Q}}_Y)$ be pairs of PMFs that both minimize $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$. We establish uniqueness by arguing that $(Q_X^{*},Q_Y^{*})$ and $({\widehat{Q}}_X,{\widehat{Q}}_Y)$ must be identical. Observe that

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& \le D_{\alpha }\left(P_{XY}\parallel (0.5Q_X^{*}+0.5{\widehat{Q}}_X)(0.5Q_Y^{*}+0.5{\widehat{Q}}_Y)\right)\hfill \end{array}$$

(276)

$$\begin{array}{c}\hfill \le 0.5D_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})+0.5D_{\alpha }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y)\end{array}$$

(277)

$$\begin{array}{c}\hfill =J_{\alpha }(X;Y),\end{array}$$

(278)

where (276) holds by the definition of $J_{\alpha }(X;Y)$, and (277) follows from Lemma 15. Hence, (277) holds with equality, which implies that (228) in the proof of Lemma 15 holds with equality, i.e.,

$$\begin{array}{c}\sum _{x,y}P{(x,y)}^{\alpha }\sqrt{0.5Q_X^{*}{\left(x\right)}^{2(1-\alpha )}+0.5{\widehat{Q}}_X{\left(x\right)}^{2(1-\alpha )}}\sqrt{0.5Q_Y^{*}{\left(y\right)}^{2(1-\alpha )}+0.5{\widehat{Q}}_Y{\left(y\right)}^{2(1-\alpha )}}\hfill \\ =\sum _{x,y}P{(x,y)}^{\alpha }{[0.5Q_X^{*}\left(x\right)+0.5{\widehat{Q}}_X\left(x\right)]}^{1-\alpha }\sqrt{0.5Q_Y^{*}{\left(y\right)}^{2(1-\alpha )}+0.5{\widehat{Q}}_Y{\left(y\right)}^{2(1-\alpha )}}.\hfill \end{array}$$

(279)

We first argue that $Q_X^{*}={\widehat{Q}}_X$. Since $Q_X^{*}$ and ${\widehat{Q}}_X$ are PMFs, it suffices to show that $Q_X^{*}\left(x\right)={\widehat{Q}}_X\left(x\right)$ for every $x\in supp\left({\widehat{Q}}_X\right)$. Let $\widehat{x}\in supp\left({\widehat{Q}}_X\right)$. Because $supp\left({\widehat{Q}}_X\right)\subseteq supp\left(P_X\right)$ (Lemma 16), there exists a $\widehat{y}\in \mathcal{Y}$ such that $P(\widehat{x},\widehat{y})>0$. Again by Lemma 16, this implies that ${\widehat{Q}}_Y\left(\widehat{y}\right)>0$. Because the mapping $z\mapsto z^{2(1-\alpha )}$ is strictly concave on ${\mathbb{R}}_{\ge 0}$ for $\alpha \in (\frac{1}{2},1)$, it follows from (279) that $Q_X^{*}\left(\widehat{x}\right)={\widehat{Q}}_X\left(\widehat{x}\right)$. Swapping the roles of $Q_X$ and $Q_Y$, we obtain that $Q_Y^{*}={\widehat{Q}}_Y$.

For $\alpha =1$, the minimizer is unique by Proposition 8 because $D_1(P_{XY}\parallel Q_X{Q}_Y)=D(P_{XY}\parallel Q_X{Q}_Y)$.

Now consider $\alpha \in (1,\infty ]$. Here, we establish uniqueness via the characterization of $J_{\alpha }(X;Y)$ provided by Lemma 18. Let ${\psi }_{\alpha }\left(R_{XY}\right)$ be defined as in Lemma 18. Let $R_{XY}$ be a PMF that satisfies $J_{\alpha }(X;Y)={\psi }_{\alpha }\left(R_{XY}\right)$, and let $(Q_X^{*},Q_Y^{*})$ be a pair of PMFs that minimizes $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$. If $\alpha \in (1,\infty )$, then (264) in the proof of Lemma 18 holds with equality, i.e.,

$$\begin{array}{c}\hfill D(R_{XY}\parallel R_X{R}_Y)+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY})=D(R_{XY}\parallel Q_X^{*}{Q}_Y^{*})+\frac{\alpha }{1-\alpha }D(R_{XY}\parallel P_{XY}).\end{array}$$

(280)

Because the LHS of (280) is finite, Proposition 8 implies that $Q_X^{*}=R_X$ and $Q_Y^{*}=R_Y$, thus the minimizer is unique. As shown in the proof of Lemma 18, (280) remains valid for $\alpha =\infty$ after replacing $\frac{\alpha }{1-\alpha }$ by $-1$, thus the same argument establishes the uniqueness for $\alpha =\infty$.

Finally, we show that, for $\alpha \in [0,\frac{1}{2}]$, the mapping $(Q_X,Q_Y)\mapsto D_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ can have more than one minimizer. Let X be uniformly distributed over $\{0,1\}$, and let $Y=X$. Then, for all $\alpha \in [0,\frac{1}{2}]$,

$$\begin{array}{cc}\hfill J_{\alpha }(X;Y)& =\frac{\alpha }{1-\alpha }log2\hfill \end{array}$$

(281)

$$\begin{array}{c}=D_{\alpha }\left(P_{XY}\parallel (1,0)(1,0)\right)\hfill \end{array}$$

(282)

$$\begin{array}{c}=D_{\alpha }\left(P_{XY}\parallel (0,1)(0,1)\right),\hfill \end{array}$$

(283)

where (281) follows from Lemma 11. □

Lemma 21.

For every $\alpha \in [0,\infty ]$, the minimum in the definition of $K_{\alpha }(X;Y)$ in (2) exists and is finite.

Proof. 

Let $\alpha \in [0,\infty ]$, and denote by $U_X$ and $U_Y$ the uniform distribution over $\mathcal{X}$ and $\mathcal{Y}$, respectively. Then ${inf}_{Q_X,Q_Y}{\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is finite because ${\Delta }_{\alpha }(P_{XY}\parallel U_X{U}_Y)$ is finite and because the relative $\alpha$-entropy is nonnegative (Proposition 5). For $\alpha \in (0,\infty )$, the minimum exists because the set $\mathcal{P}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{Y}\right)$ is compact and the mapping $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ is continuous. For $\alpha \in \{0,\infty \}$, the minimum exists because $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ takes on only a finite number of values: if $\alpha =0$, then ${\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ depends on $Q_X{Q}_Y$ only via $supp\left(Q_X{Q}_Y\right)\subseteq \mathcal{X}\times \mathcal{Y}$; and if $\alpha =\infty$, then ${\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ depends on $Q_X{Q}_Y$ only via $argmax\left(Q_X{Q}_Y\right)\subseteq \mathcal{X}\times \mathcal{Y}$. □

Lemma 22.

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)\ge 0$. If $\alpha \in (0,\infty )$, then $K_{\alpha }(X;Y)=0$ if and only if X and Y are independent (nonnegativity).

Proof. 

The nonnegativity follows from the definition of $K_{\alpha }(X;Y)$ because the relative $\alpha$-entropy is nonnegative for $\alpha \in [0,\infty ]$ (Proposition 5). If X and Y are independent, then $P_{XY}=P_X{P}_Y$, and the choice $Q_X=P_X$ and $Q_Y=P_Y$ in the definition of $K_{\alpha }(X;Y)$ achieves $K_{\alpha }(X;Y)=0$. Conversely, if $K_{\alpha }(X;Y)=0$, then there exist PMFs $Q_X^{*}$ and $Q_Y^{*}$ satisfying ${\Delta }_{\alpha }(P_{XY}\parallel Q_X^{*}{Q}_Y^{*})=0$. If, in addition, $\alpha \in (0,\infty )$, then $P_{XY}=Q_X^{*}{Q}_Y^{*}$ by Proposition 5, and hence X and Y are independent. □

Lemma 23.

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)=K_{\alpha }(Y;X)$ (symmetry).

Proof. 

The definition of $K_{\alpha }(X;Y)$ is symmetric in X and Y. □

Lemma 24.

For all $\alpha \in (0,\infty )$,

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)=\underset{Q_X,Q_Y}{min}-log{M}_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y),\end{array}$$

(284)

where $M_{\beta }(Q_X,Q_Y)$ is the following weighted power mean [30] (Chapter III): For $\beta \in \mathbb{R}\setminus \left\{0\right\}$,

$$\begin{array}{c}\hfill M_{\beta }(Q_X,Q_Y)\triangleq {\left[\sum _{x,y}P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\beta }\right]}^{\frac{1}{\beta }},\end{array}$$

(285)

where for $\beta <0$, we read $P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\beta }$ as $P(x,y)/{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{-\beta }$ and use the conventions (44); and for $\beta =0$, using the convention $0^0=1$,

$$\begin{array}{c}\hfill M_0(Q_X,Q_Y)\triangleq \prod _{x,y}{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{P(x,y)}.\end{array}$$

(286)

Proof. 

Let $\alpha \in (0,\infty )$, and define the PMF ${\tilde{P}}_{XY}$ as

$$\begin{array}{c}\hfill {\tilde{P}}_{XY}(x,y)\triangleq \frac{P_{XY}{(x,y)}^{\alpha }}{{\sum }_{(x^{\prime },y^{\prime })\in \mathcal{X}\times \mathcal{Y}}{P}_{XY}{(x^{\prime },y^{\prime })}^{\alpha }}.\end{array}$$

(287)

Then,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& =J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y})\hfill \end{array}$$

(288)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}{D}_{\frac{1}{\alpha }}({\tilde{P}}_{XY}\parallel Q_X{Q}_Y),\end{array}$$

(289)

where (288) follows from Proposition 7, and (289) follows from the definition of $J_{1/\alpha }(\tilde{X};\tilde{Y})$. A simple computation reveals that for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{c}\hfill D_{\frac{1}{\alpha }}({\tilde{P}}_{XY}\parallel Q_X{Q}_Y)=-log{M}_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y)-H_{\alpha }(X,Y).\end{array}$$

(290)

Hence, (284) follows from (289) and (290). □

Lemma 25.

For $\alpha =0$,

$$\begin{array}{cc}\hfill K_0(X;Y)& =log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill \end{array}$$

(291)

$$\begin{array}{c}\ge \underset{Q_X,Q_Y}{min}log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\frac{1}{Q_X\left(x\right)Q_Y\left(y\right)}-log|supp\left(P_{XY}\right)|\hfill \end{array}$$

(292)

$$\begin{array}{c}\hfill =\underset{\alpha \downarrow 0}{lim}{K}_{\alpha }(X;Y),\end{array}$$

(293)

where in the RHS of (292), we use the conventions (44). The inequality can be strict, so $\alpha \mapsto K_{\alpha }(X;Y)$ need not be continuous at $\alpha =0$.

Proof. 

We first prove (291). Recall that

$$\begin{array}{c}\hfill {\Delta }_0(P_{XY}\parallel Q_X{Q}_Y)=\left\{\begin{array}{cc}log\frac{|supp\left(Q_X{Q}_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill & \mathrm{if}supp\left(P_{XY}\right)\subseteq supp\left(Q_X{Q}_Y\right),\hfill \\ \infty \hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(294)

Observe that ${\Delta }_0(P_{XY}\parallel Q_X{Q}_Y)$ is finite only if $supp\left(P_X\right)\subseteq supp\left(Q_X\right)$ and $supp\left(P_Y\right)\subseteq supp\left(Q_Y\right)$. For such PMFs $Q_X$ and $Q_Y$, we have $|supp\left(Q_X{Q}_Y\right)|\ge |supp\left(P_X{P}_Y\right)|$. Thus, for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{c}\hfill {\Delta }_0(P_{XY}\parallel Q_X{Q}_Y)\ge log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|}.\end{array}$$

(295)

Choosing $Q_X=P_X$ and $Q_Y=P_Y$ achieves equality in (295), which establishes (291).

We now show (292). Let $Q_X$ and $Q_Y$ be the uniform distributions over $supp\left(P_X\right)$ and $supp\left(P_Y\right)$, respectively. Then,

$$\begin{array}{c}\hfill log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\frac{1}{Q_X\left(x\right)Q_Y\left(y\right)}-log|supp\left(P_{XY}\right)|=log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|},\end{array}$$

(296)

and hence (292) holds.

We next establish (293). To that end, define

$$\begin{array}{c}\hfill \tau \triangleq \underset{(x,y)\in supp\left(P_{XY}\right)}{min}P(x,y).\end{array}$$

(297)

We bound $K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ as follows: For all $\alpha \in (0,1)$,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)& =\underset{Q_X,Q_Y}{min}\frac{\alpha }{1-\alpha }log\sum _{x,y}P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\frac{\alpha -1}{\alpha }}\hfill \end{array}$$

(298)

$$\begin{array}{c}\ge \underset{Q_X,Q_Y}{min}\frac{\alpha }{1-\alpha }log\sum _{(x,y)\in supp\left(P_{XY}\right)}\tau {\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\frac{\alpha -1}{\alpha }}\hfill \end{array}$$

(299)

$$\begin{array}{c}\ge \underset{Q_X,Q_Y}{min}\frac{\alpha }{1-\alpha }log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\tau {\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\frac{\alpha -1}{\alpha }}\hfill \end{array}$$

(300)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\frac{1}{Q_X\left(x\right)Q_Y\left(y\right)}-\frac{\alpha }{1-\alpha }log\frac{1}{\tau },\end{array}$$

(301)

where (298) follows from Lemma 24. Similarly, for all $\alpha \in (0,1)$,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)& =\underset{Q_X,Q_Y}{min}\frac{\alpha }{1-\alpha }log\sum _{x,y}P(x,y){\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\frac{\alpha -1}{\alpha }}\hfill \end{array}$$

(302)

$$\begin{array}{c}\le \underset{Q_X,Q_Y}{min}\frac{\alpha }{1-\alpha }log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}{\left[Q_X\left(x\right)Q_Y\left(y\right)\right]}^{\frac{\alpha -1}{\alpha }}\hfill \end{array}$$

(303)

$$\begin{array}{c}\hfill =\underset{Q_X,Q_Y}{min}log\underset{(x,y)\in supp\left(P_{XY}\right)}{max}\frac{1}{Q_X\left(x\right)Q_Y\left(y\right)},\end{array}$$

(304)

where (302) is the same as (298). Now (293) follows from (301), (304), and the sandwich theorem because ${lim}_{\alpha \downarrow 0}\frac{\alpha }{1-\alpha }log\frac{1}{\tau }=0$ and because ${lim}_{\alpha \downarrow 0}{H}_{\alpha }(X,Y)=log|supp\left(P_{XY}\right)|$ (Proposition 3).

Finally, we provide an example for which (292) holds with strict inequality. Let $\mathcal{X}=\{1,2,3\}$, let $\mathcal{Y}=\{1,2\}$, and let $(X,Y)$ be uniformly distributed over $\left\{\right(1,1),(2,2),(3,1\left)\right\}$. The LHS of (292) then equals $log2$. Using

$$\begin{array}{cc}\hfill Q_X\left(x\right)& \triangleq \left\{\begin{array}{cc}0.28\hfill & \mathrm{if}x\in \{1,3\},\hfill \\ 0.44\hfill & \mathrm{if}x=2,\hfill \end{array}\right.\hfill \end{array}$$

(305)

$$\begin{array}{cc}\hfill Q_Y\left(y\right)& \triangleq \left\{\begin{array}{cc}0.60\hfill & \mathrm{if}y=1,\hfill \\ 0.40\hfill & \mathrm{if}y=2,\hfill \end{array}\right.\hfill \end{array}$$

(306)

we see that the RHS of (292) is upper bounded by $log\frac{5.952\dots }{3}$, which is smaller than $log2$. □

Lemma 26.

$K_1(X;Y)=I(X;Y)$.

Proof. 

The claim follows from Proposition 8 because ${\Delta }_1(P_{XY}\parallel Q_X{Q}_Y)$ in the definition of $K_1(X;Y)$ is equal to $D(P_{XY}\parallel Q_X{Q}_Y)$. □

Lemma 27.

Let $f:\{1,\dots ,|\mathcal{X}\left|\right\}\to \mathcal{X}$ and $g:\{1,\dots ,|\mathcal{Y}\left|\right\}\to \mathcal{Y}$ be bijective functions, and let $\mathsf{B}$ be the $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ matrix whose Row-i Column-j entry ${\mathsf{B}}_{i,j}$ equals $P_{XY}(f\left(i\right),g\left(j\right))$. Then,

$$\begin{array}{c}\hfill K_2(X;Y)=-2log{\sigma }_1\left(\mathsf{B}\right)-H_2(X,Y),\end{array}$$

(307)

where ${\sigma }_1\left(\mathsf{B}\right)$ denotes the largest singular value of $\mathsf{B}$. (Because the singular values of a matrix are invariant under row and column permutations, the result does not depend on f or g.)

Proof. 

Let $(\tilde{X},\tilde{Y})$ be distributed according to the joint PMF

$$\begin{array}{c}\hfill {\tilde{P}}_{XY}(x,y)\triangleq {\left[\beta P_{XY}(x,y)\right]}^2,\end{array}$$

(308)

where

$$\begin{array}{c}\hfill \beta \triangleq {\left[\sum _{x,y}{P}_{XY}{(x,y)}^2\right]}^{-\frac{1}{2}}.\end{array}$$

(309)

Then,

$$\begin{array}{cc}\hfill K_2(X;Y)& =J_{\frac{1}{2}}(\tilde{X};\tilde{Y})\hfill \end{array}$$

(310)

$$\begin{array}{c}\hfill =-2log{\sigma }_1\left(\beta \mathsf{B}\right)\end{array}$$

(311)

$$\begin{array}{c}\hfill =-2log\left[\beta {\sigma }_1\left(\mathsf{B}\right)\right]\end{array}$$

(312)

$$\begin{array}{c}\hfill =-2log{\sigma }_1\left(\mathsf{B}\right)-H_2(X,Y),\end{array}$$

(313)

where (310) follows from Proposition 7; (311) follows from Lemma 6 and (308); (312) holds because $\beta >0$; and (313) follows from the definition of $H_2(X,Y)$. □

Lemma 28.

$K_{\infty }(X;Y)=0$.

Proof. 

Let the pair $(\widehat{x},\widehat{y})$ be such that $P(\widehat{x},\widehat{y})={max}_{x,y}P(x,y)$, and define the PMFs ${\widehat{Q}}_X$ and ${\widehat{Q}}_Y$ as ${\widehat{Q}}_X\left(x\right)=𝟙\{x=\widehat{x}\}$ and ${\widehat{Q}}_Y\left(y\right)=𝟙\{y=\widehat{y}\}$. Then, ${\Delta }_{\infty }(P_{XY}\parallel {\widehat{Q}}_X{\widehat{Q}}_Y)=0$, so $K_{\infty }(X;Y)\le 0$. Because $K_{\infty }(X;Y)\ge 0$ (Lemma 22), this implies $K_{\infty }(X;Y)=0$. □

Lemma 29.

The mapping $\alpha \mapsto K_{\alpha }(X;Y)$ need not be monotonic on $[0,\infty ]$.

Proof. 

Let $P_{XY}$ be such that $supp\left(P_{XY}\right)=\mathcal{X}\times \mathcal{Y}$ and $I(X;Y)>0$. Then,

$$\begin{array}{cc}\hfill K_0(X;Y)& =0,\hfill \end{array}$$

(314)

$$\begin{array}{cc}\hfill K_1(X;Y)& >0,\hfill \end{array}$$

(315)

$$\begin{array}{cc}\hfill K_{\infty }(X;Y)& =0,\hfill \end{array}$$

(316)

which follow from Lemmas 25, 26, and 28, respectively. Thus, $\alpha \mapsto K_{\alpha }(X;Y)$ is not monotonic on $[0,\infty ]$. □

Lemma 30.

The mapping $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is nonincreasing on $[0,\infty ]$.

Proof. 

We first show the monotonicity for $\alpha \in (0,\infty )$. To that end, let $\alpha ,{\alpha }^{\prime }\in (0,\infty )$ with $\alpha \le {\alpha }^{\prime }$, and let $M_{\beta }(Q_X,Q_Y)$ be defined as in (285) and (286). Then, for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{c}\hfill M_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y)\le M_{\frac{{\alpha }^{\prime }-1}{{\alpha }^{\prime }}}(Q_X,Q_Y),\end{array}$$

(317)

which follows from the power mean inequality [30] (III 3.1.1 Theorem 1) because $\frac{\alpha -1}{\alpha }\le \frac{{\alpha }^{\prime }-1}{{\alpha }^{\prime }}$. Hence,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)& =\underset{Q_X,Q_Y}{min}-log{M}_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y)\hfill \end{array}$$

(318)

$$\begin{array}{c}\hfill \ge \underset{Q_X,Q_Y}{min}-log{M}_{\frac{{\alpha }^{\prime }-1}{{\alpha }^{\prime }}}(Q_X,Q_Y)\end{array}$$

(319)

$$\begin{array}{c}\hfill =K_{{\alpha }^{\prime }}(X;Y)+H_{{\alpha }^{\prime }}(X,Y),\end{array}$$

(320)

where (318) and (320) follow from Lemma 24, and (319) follows from (317).

The monotonicity extends to $\alpha =0$ because

$$\begin{array}{cc}\hfill K_0(X;Y)+H_0(X,Y)& \ge \underset{\alpha \downarrow 0}{lim}{K}_{\alpha }(X;Y)+H_0(X,Y)\hfill \end{array}$$

(321)

$$\begin{array}{c}=\underset{\alpha \downarrow 0}{lim}[K_{\alpha }(X;Y)+H_{\alpha }(X,Y)],\hfill \end{array}$$

(322)

where (321) follows from Lemma 25, and (322) holds because $\alpha \mapsto H_{\alpha }(X,Y)$ is continuous at $\alpha =0$ (Proposition 3).

The monotonicity extends to $\alpha =\infty$ because for all $\alpha \in (0,\infty )$,

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)+H_{\alpha }(X,Y)& \ge H_{\alpha }(X,Y)\hfill \end{array}$$

(323)

$$\begin{array}{c}\hfill \ge H_{\infty }(X,Y)\end{array}$$

(324)

$$\begin{array}{c}\hfill =K_{\infty }(X;Y)+H_{\infty }(X,Y),\end{array}$$

(325)

where (323) holds because $K_{\alpha }(X;Y)\ge 0$ (Lemma 22); (324) holds because $H_{\alpha }(X,Y)$ is nonincreasing in $\alpha$ (Proposition 3); and (325) holds because $K_{\infty }(X;Y)=0$ (Lemma 28). □

Lemma 31.

The mapping $\alpha \mapsto K_{\alpha }(X;Y)$ is continuous on $(0,\infty ]$. (See Lemma 25 for the behavior at $\alpha =0$.)

Proof. 

Because $\alpha \mapsto H_{\alpha }(X,Y)$ is continuous on $[0,\infty ]$ (Proposition 3), it suffices to show that the mapping $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is continuous on $(0,\infty ]$. We first show that it is continuous on $(0,1)\cup (1,\infty )$ by showing that $\alpha \mapsto \left(1-\frac{1}{\alpha }\right)\left[K_{\alpha }(X;Y)+H_{\alpha }(X,Y)\right]$ is concave and hence continuous on $(0,\infty )$. For a fixed $\alpha \in (0,\infty )$, let $(\tilde{X},\tilde{Y})$ be distributed according to the joint PMF

$$\begin{array}{c}\hfill {\tilde{P}}_{XY}(x,y)\triangleq \frac{P_{XY}{(x,y)}^{\alpha }}{{\sum }_{(x^{\prime },y^{\prime })\in \mathcal{X}\times \mathcal{Y}}{P}_{XY}{(x^{\prime },y^{\prime })}^{\alpha }}.\end{array}$$

(326)

Then, for all $\alpha \in (0,\infty )$,

$$\begin{array}{c}\left(1-\frac{1}{\alpha }\right)\left[K_{\alpha }(X;Y)+H_{\alpha }(X,Y)\right]\hfill \\ =\left(1-\frac{1}{\alpha }\right)J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y})+\left(1-\frac{1}{\alpha }\right)H_{\alpha }(X,Y)\hfill \end{array}$$

(327)

$$\begin{array}{c}\hfill =\underset{R_{XY}}{min}\left[\left(1-\frac{1}{\alpha }\right)D(R_{XY}\parallel R_X{R}_Y)+\frac{1}{\alpha }D(R_{XY}\parallel {\tilde{P}}_{XY})+\left(1-\frac{1}{\alpha }\right)H_{\alpha }(X,Y)\right]\end{array}$$

(328)

$$\begin{array}{c}\hfill =\underset{R_{XY}}{min}\left[\left(1-\frac{1}{\alpha }\right)D(R_{XY}\parallel R_X{R}_Y)+\left(1-\frac{1}{\alpha }\right)H\left(R_{XY}\right)+D(R_{XY}\parallel P_{XY})\right],\end{array}$$

(329)

where (327) follows from Proposition 7; (328) follows from Lemma 8; and (329) follows from a short computation. For every $R_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})$, the expression in square brackets on the RHS of (329) is concave in $\alpha$ because the mapping $\alpha \mapsto 1-\frac{1}{\alpha }$ is concave on $(0,\infty )$ and because $D(R_{XY}\parallel R_X{R}_Y)$ and $H\left(R_{XY}\right)$ are nonnegative. The pointwise minimum preserves the concavity, thus the LHS of (327) is concave in $\alpha$ and hence continuous in $\alpha \in (0,\infty )$. This implies that $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ and hence $\alpha \mapsto K_{\alpha }(X;Y)$ is continuous on $(0,1)\cup (1,\infty )$.

We now establish continuity at $\alpha =\infty$. Let $(\widehat{x},\widehat{y})$ be such that $P(\widehat{x},\widehat{y})={max}_{x,y}P(x,y)$; define the PMFs ${\widehat{Q}}_X$ and ${\widehat{Q}}_Y$ as ${\widehat{Q}}_X\left(x\right)\triangleq 𝟙\{x=\widehat{x}\}$ and ${\widehat{Q}}_Y\left(y\right)\triangleq 𝟙\{y=\widehat{y}\}$; and let $M_{\beta }(Q_X,Q_Y)$ be defined as in (285). Then, for all $\alpha \in (1,\infty )$,

$$\begin{array}{cc}\hfill K_{\infty }(X;Y)+H_{\infty }(X,Y)& \le K_{\alpha }(X;Y)+H_{\alpha }(X,Y)\hfill \end{array}$$

(330)

$$\begin{array}{c}\hfill \le -log{M}_{\frac{\alpha -1}{\alpha }}({\widehat{Q}}_X,{\widehat{Q}}_Y)\end{array}$$

(331)

$$\begin{array}{c}\hfill =\frac{\alpha }{\alpha -1}{H}_{\infty }(X,Y)\end{array}$$

(332)

$$\begin{array}{c}\hfill =K_{\infty }(X;Y)+\frac{\alpha }{\alpha -1}{H}_{\infty }(X,Y),\end{array}$$

(333)

where (330) holds because $K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is nonincreasing in $\alpha$ (Lemma 30); (331) follows from Lemma 24; (332) follows from the definitions of $M_{\beta }(Q_X,Q_Y)$ in (285) and $H_{\infty }(X,Y)$ in (46); and (333) holds because $K_{\infty }(X;Y)=0$ (Lemma 28). Because ${lim}_{\alpha \to \infty }\frac{\alpha }{\alpha -1}=1$, (330)–(333) and the sandwich theorem imply that $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is continuous at $\alpha =\infty$. This and the continuity of $\alpha \mapsto H_{\alpha }(X,Y)$ at $\alpha =\infty$ (Proposition 3) establish the continuity of $\alpha \mapsto K_{\alpha }(X;Y)$ at $\alpha =\infty$.

It remains to show the continuity at $\alpha =1$. Let $\alpha \in (\frac{4}{5},1)\cup (1,\frac{4}{3})$, and define $\delta \triangleq \frac{\left|\alpha -1\right|}{\alpha }\in (0,\frac{1}{4})$. (These definitions ensure that on the RHS of (340) ahead, $1-4\delta$ will be positive.) Let $M_{\beta }(Q_X,Q_Y)$ be defined as in (285) and (286). Then, for all PMFs $Q_X$ and $Q_Y$,

$$\begin{array}{cc}\hfill M_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y)& \le M_{\delta }(Q_X,Q_Y)\hfill \end{array}$$

(334)

$$\begin{array}{c}={\left[\sum _{x,y}P(x,y){\left[P_X\left(x\right)P_Y\left(y\right)\right]}^{\delta }{\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P_X\left(x\right)P_Y\left(y\right)}\right]}^{\delta }\right]}^{\frac{1}{\delta }}\hfill \end{array}$$

(335)

$$\begin{array}{c}\hfill \le {\left[\sum _{x,y}P(x,y){\left[P_X\left(x\right)P_Y\left(y\right)\right]}^{2\delta }\right]}^{\frac{1}{2\delta }}·{\left[\sum _{x,y}P(x,y){\left[\frac{Q_X\left(x\right)Q_Y\left(y\right)}{P_X\left(x\right)P_Y\left(y\right)}\right]}^{2\delta }\right]}^{\frac{1}{2\delta }}\end{array}$$

(336)

$$\begin{array}{c}\hfill \le {\left[\sum _{x,y}P(x,y){\left[P_X\left(x\right)P_Y\left(y\right)\right]}^{2\delta }\right]}^{\frac{1}{2\delta }}\end{array}$$

(337)

$$\begin{array}{c}\hfill =M_{2\delta }(P_X,P_Y),\end{array}$$

(338)

where (334) follows from the power mean inequality [30] (III 3.1.1 Theorem 1) because $\frac{\alpha -1}{\alpha }\le \delta$; (336) follows from the Cauchy–Schwarz inequality; and (337) holds because

$$\begin{array}{c}{\left[\sum _{x,y}P(x,y){\left[\frac{Q_X\left(x\right)}{P_X\left(x\right)}\right]}^{2\delta }{\left[\frac{Q_Y\left(y\right)}{P_Y\left(y\right)}\right]}^{2\delta }\right]}^{\frac{1}{2\delta }}\hfill \\ \le {\left[\sum _x{P}_X\left(x\right){\left[\frac{Q_X\left(x\right)}{P_X\left(x\right)}\right]}^{4\delta }\right]}^{\frac{1}{4\delta }}·{\left[\sum _y{P}_Y\left(y\right){\left[\frac{Q_Y\left(y\right)}{P_Y\left(y\right)}\right]}^{4\delta }\right]}^{\frac{1}{4\delta }}\hfill \end{array}$$

(339)

$$\begin{array}{c}\hfill =2^{-D_{1-4\delta }(P_X\parallel Q_X)}·2^{-D_{1-4\delta }(P_Y\parallel Q_Y)}\end{array}$$

(340)

$$\begin{array}{c}\hfill \le 1,\end{array}$$

(341)

where (339) follows from the Cauchy–Schwarz inequality, and (341) holds because $1-4\delta >0$ and because the Rényi divergence is nonnegative for positive orders (Proposition 4). Thus, for all $\alpha \in (\frac{4}{5},\frac{4}{3})$,

$$\begin{array}{cc}\hfill -log{M}_{\frac{2\left|\alpha -1\right|}{\alpha }}(P_X,P_Y)& \le \underset{Q_X,Q_Y}{min}-log{M}_{\frac{\alpha -1}{\alpha }}(Q_X,Q_Y)\hfill \end{array}$$

(342)

$$\begin{array}{c}\le -log{M}_{\frac{\alpha -1}{\alpha }}(P_X,P_Y),\hfill \end{array}$$

(343)

where (342) follows from (338) if $\alpha \ne 1$ and from Proposition 8 and a simple computation if $\alpha =1$. By Lemma 24, this implies that for all $\alpha \in (\frac{4}{5},\frac{4}{3})$,

$$\begin{array}{cc}\hfill -log{M}_{\frac{2\left|\alpha -1\right|}{\alpha }}(P_X,P_Y)& \le K_{\alpha }(X;Y)+H_{\alpha }(X,Y)\hfill \end{array}$$

(344)

$$\begin{array}{c}\le -log{M}_{\frac{\alpha -1}{\alpha }}(P_X,P_Y).\hfill \end{array}$$

(345)

Because $\beta \mapsto M_{\beta }(P_X,P_Y)$ is continuous at $\beta =0$ [30] (III 1 Theorem 2(b)), (344)–(345) and the sandwich theorem imply that $\alpha \mapsto K_{\alpha }(X;Y)+H_{\alpha }(X,Y)$ is continuous at $\alpha =1$. This and the continuity of $\alpha \mapsto H_{\alpha }(X,Y)$ at $\alpha =1$ (Proposition 3) establish the continuity of $\alpha \mapsto K_{\alpha }(X;Y)$ at $\alpha =1$. □

Lemma 32.

If $X=Y$ with probability one, then

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)=\left\{\begin{array}{cc}2H_{\frac{\alpha }{2-\alpha }}\left(X\right)-H_{\alpha }\left(X\right)\hfill & if\alpha \in [0,2),\hfill \\ \frac{\alpha }{\alpha -1}{H}_{\infty }\left(X\right)-H_{\alpha }\left(X\right)\hfill & if\alpha \ge 2,\hfill \\ 0\hfill & if\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(346)

Proof. 

We first treat the cases $\alpha =0$, $\alpha =1$, and $\alpha =\infty$. For $\alpha =0$, (346) holds because

$$\begin{array}{cc}\hfill K_0(X;Y)& =log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill \end{array}$$

(347)

$$\begin{array}{c}\hfill =log|supp\left(P_X\right)|\end{array}$$

(348)

$$\begin{array}{c}\hfill =H_0\left(X\right),\end{array}$$

(349)

where (347) follows from Lemma 25, and (348) holds because the hypothesis $Pr[X=Y]=1$ implies that $|supp\left(P_X{P}_Y\right)|={|supp\left(P_X\right)|}^2$ and $|supp\left(P_{XY}\right)|=|supp\left(P_X\right)|$. For $\alpha =1$, (346) holds because $K_1(X;Y)=I(X;Y)$ (Lemma 26) and because $Pr[X=Y]=1$ implies that $I(X;Y)=H\left(X\right)=H_1\left(X\right)$. For $\alpha =\infty$, (346) holds because $K_{\infty }(X;Y)=0$ (Lemma 28).

Now let $\alpha \in (0,1)\cup (1,\infty )$, and let $(\tilde{X},\tilde{Y})$ be distributed according to the joint PMF

$$\begin{array}{cc}\hfill {\tilde{P}}_{XY}(x,y)& \triangleq \frac{P_{XY}{(x,y)}^{\alpha }}{{\sum }_{(x^{\prime },y^{\prime })\in \mathcal{X}\times \mathcal{Y}}{P}_{XY}{(x^{\prime },y^{\prime })}^{\alpha }}\hfill \end{array}$$

(350)

$$\begin{array}{c}=\frac{P_X{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\alpha }}𝟙\{x=y\},\hfill \end{array}$$

(351)

where (351) holds because $P_{XY}(x,y)=P_X\left(x\right)𝟙\{x=y\}$ for all $x\in \mathcal{X}$ and all $y\in \mathcal{Y}$. If $\alpha <2$, then (346) holds because

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& =J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y})\hfill \end{array}$$

(352)

$$\begin{array}{c}\hfill =H_{\frac{1}{2-\alpha }}\left(\tilde{X}\right)\end{array}$$

(353)

$$\begin{array}{c}=\frac{2-\alpha }{1-\alpha }log\sum _x{\left[\frac{P_X{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\alpha }}\right]}^{\frac{1}{2-\alpha }}\hfill \end{array}$$

(354)

$$\begin{array}{c}\hfill =2H_{\frac{\alpha }{2-\alpha }}\left(X\right)-H_{\alpha }\left(X\right),\end{array}$$

(355)

where (352) follows from Proposition 7; (353) follows from Lemma 11 because $Pr[\tilde{X}=\tilde{Y}]=1$ and because $\frac{1}{\alpha }>\frac{1}{2}$; and (355) follows from a simple computation. If $\alpha \ge 2$, then (346) holds because

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& =J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y})\hfill \end{array}$$

(356)

$$\begin{array}{c}\hfill =\frac{1}{\alpha -1}{H}_{\infty }\left(\tilde{X}\right)\end{array}$$

(357)

$$\begin{array}{c}\hfill =\frac{-1}{\alpha -1}log\underset{x}{max}\frac{P_X{\left(x\right)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\alpha }}\end{array}$$

(358)

$$\begin{array}{c}\hfill =\frac{\alpha }{\alpha -1}{H}_{\infty }\left(X\right)-H_{\alpha }\left(X\right),\end{array}$$

(359)

where (356) follows from Proposition 7; (357) follows from Lemma 11 because $Pr[\tilde{X}=\tilde{Y}]=1$ and because $\frac{1}{\alpha }\le \frac{1}{2}$; and (359) follows from a simple computation. □

Lemma 33.

For every $\alpha \in (0,2)$, the mapping $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ in the definition of $K_{\alpha }(X;Y)$ in (2) has a unique minimizer. This need not be the case when $\alpha \in \left\{0\right\}\cup [2,\infty ]$.

Proof. 

Let $\alpha \in (0,2)$. By Proposition 7, $K_{\alpha }(X;Y)=J_{1/\alpha }(\tilde{X};\tilde{Y})$, where the pair $(\tilde{X},\tilde{Y})$ is distributed according to the joint PMF ${\tilde{P}}_{XY}$ defined in Proposition 7. The mapping $(Q_X,Q_Y)\mapsto D_{1/\alpha }({\tilde{P}}_{XY}\parallel Q_X{Q}_Y)$ in the definition of $J_{1/\alpha }(\tilde{X};\tilde{Y})$ has a unique minimizer by Lemma 20 because $\frac{1}{\alpha }>\frac{1}{2}$. By Proposition 6, there is a bijection between the minimizers of $D_{1/\alpha }({\tilde{P}}_{XY}\parallel Q_X{Q}_Y)$ and ${\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$, so the mapping $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ also has a unique minimizer.

We next show that for $\alpha \in \left\{0\right\}\cup [2,\infty ]$, the mapping $(Q_X,Q_Y)\mapsto {\Delta }_{\alpha }(P_{XY}\parallel Q_X{Q}_Y)$ can have more than one minimizer. Let X be uniformly distributed over $\{0,1\}$, and let $Y=X$. Then, by Lemma 32,

$$\begin{array}{c}\hfill K_{\alpha }(X;Y)=\left\{\begin{array}{cc}log2\hfill & \mathrm{if}\alpha =0,\hfill \\ \frac{1}{\alpha -1}log2\hfill & \mathrm{if}\alpha \ge 2,\hfill \\ 0\hfill & \mathrm{if}\alpha =\infty .\hfill \end{array}\right.\end{array}$$

(360)

If $\alpha =0$, then it follows from the definition of ${\Delta }_0(P\parallel Q)$ in (56) that ${\Delta }_0(P_{XY}\parallel Q_X{Q}_Y)=log2$ whenever $supp\left(Q_X\right)=supp\left(Q_Y\right)=\{0,1\}$, so the minimizer is not unique. Otherwise, if $\alpha \in [2,\infty ]$, it can be verified that

$$\begin{array}{cc}\hfill {\Delta }_{\alpha }\left(P_{XY}\parallel (1,0)(1,0)\right)& ={\Delta }_{\alpha }\left(P_{XY}\parallel (0,1)(0,1)\right)\hfill \end{array}$$

(361)

$$\begin{array}{c}=\left\{\begin{array}{cc}\frac{1}{\alpha -1}log2\hfill & \mathrm{if}\alpha \ge 2,\hfill \\ 0\hfill & \mathrm{if}\alpha =\infty ,\hfill \end{array}\right.\hfill \end{array}$$

(362)

so the minimizer is not unique in this case either. □

Lemma 34.

If the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ are independent, then $K_{\alpha }(X_1,X_2;Y_1,Y_2)=K_{\alpha }(X_1;Y_1)+K_{\alpha }(X_2;Y_2)$ for all $\alpha \in [0,\infty ]$ (additivity).

Proof. 

We first treat the cases $\alpha =0$ and $\alpha =\infty$. For $\alpha =0$, the claim is true because

$$\begin{array}{cc}\hfill K_0(X_1,X_2;Y_1,Y_2)& =log\frac{|supp\left(P_{X_1{X}_2}{P}_{Y_1{Y}_2}\right)|}{|supp\left(P_{X_1{X}_2{Y}_1{Y}_2}\right)|}\hfill \end{array}$$

(363)

$$\begin{array}{c}\hfill =log\frac{|supp\left(P_{X_1}{P}_{Y_1}\right)|·|supp\left(P_{X_2}{P}_{Y_2}\right)|}{|supp\left(P_{X_1{Y}_1}\right)|·|supp\left(P_{X_2{Y}_2}\right)|}\end{array}$$

(364)

$$\begin{array}{c}\hfill =K_0(X_1;Y_1)+K_0(X_2;Y_2),\end{array}$$

(365)

where (363) and (365) follow from Lemma 25, and (364) follows from the independence hypothesis $P_{X_1{X}_2{Y}_1{Y}_2}=P_{X_1{Y}_1}{P}_{X_2{Y}_2}$. For $\alpha =\infty$, the claim is true because $K_{\infty }(X;Y)=0$ (Lemma 28).

Now let $\alpha \in (0,\infty )$, and let $({\tilde{X}}_1,{\tilde{X}}_2,{\tilde{Y}}_1,{\tilde{Y}}_2)$ be distributed according to the joint PMF

$$\begin{array}{cc}\hfill {\tilde{P}}_{X_1{X}_2{Y}_1{Y}_2}(x_1,x_2,y_1,y_2)& \triangleq \frac{P_{X_1{X}_2{Y}_1{Y}_2}{(x_1,x_2,y_1,y_2)}^{\alpha }}{{\sum }_{x_1^{\prime },x_2^{\prime },y_1^{\prime },y_2^{\prime }}{P}_{X_1{X}_2{Y}_1{Y}_2}{(x_1^{\prime },x_2^{\prime },y_1^{\prime },y_2^{\prime })}^{\alpha }}\hfill \end{array}$$

(366)

$$\begin{array}{c}=\frac{P_{X_1{Y}_1}{(x_1,y_1)}^{\alpha }}{{\sum }_{x_1^{\prime },y_1^{\prime }}{P}_{X_1{Y}_1}{(x_1^{\prime },y_1^{\prime })}^{\alpha }}·\frac{P_{X_2{Y}_2}{(x_2,y_2)}^{\alpha }}{{\sum }_{x_2^{\prime },y_2^{\prime }}{P}_{X_2{Y}_2}{(x_2^{\prime },y_2^{\prime })}^{\alpha }},\hfill \end{array}$$

(367)

where (366) follows from the independence hypothesis $P_{X_1{X}_2{Y}_1{Y}_2}=P_{X_1{Y}_1}{P}_{X_2{Y}_2}$. Then,

$$\begin{array}{cc}\hfill K_{\alpha }(X_1,X_2;Y_1,Y_2)& =J_{\frac{1}{\alpha }}({\tilde{X}}_1,{\tilde{X}}_2;{\tilde{Y}}_1,{\tilde{Y}}_2)\hfill \end{array}$$

(368)

$$\begin{array}{c}\hfill =J_{\frac{1}{\alpha }}({\tilde{X}}_1;{\tilde{Y}}_1)+J_{\frac{1}{\alpha }}({\tilde{X}}_2;{\tilde{Y}}_2)\end{array}$$

(369)

$$\begin{array}{c}\hfill =K_{\alpha }(X_1;Y_1)+K_{\alpha }(X_2;Y_2),\end{array}$$

(370)

where (368) and (370) follow from Proposition 7, and (369) follows from Lemma 12 because the pairs $({\tilde{X}}_1,{\tilde{Y}}_1)$ and $({\tilde{X}}_2,{\tilde{Y}}_2)$ are independent by (367). □

Lemma 35.

For all $\alpha \in [0,\infty ]$, $K_{\alpha }(X;Y)\le log\left|\mathcal{X}\right|$.

Proof. 

For $\alpha =0$, this is true because

$$\begin{array}{cc}\hfill K_0(X;Y)& =log\frac{|supp\left(P_X{P}_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill \end{array}$$

(371)

$$\begin{array}{c}\le log\frac{\left|\mathcal{X}\right|·|supp\left(P_Y\right)|}{|supp\left(P_{XY}\right)|}\hfill \end{array}$$

(372)

$$\begin{array}{c}\le log\left|\mathcal{X}\right|,\hfill \end{array}$$

(373)

where (371) follows from Lemma 25. For $\alpha \in (0,\infty )$, the claim is true because

$$\begin{array}{cc}\hfill K_{\alpha }(X;Y)& =J_{\frac{1}{\alpha }}(\tilde{X};\tilde{Y})\hfill \end{array}$$

(374)

$$\begin{array}{c}\hfill \le log\left|\mathcal{X}\right|,\end{array}$$

(375)

where (374) follows from Proposition 7, and (375) follows from Lemma 13. For $\alpha =\infty$, the claim is true because $K_{\infty }(X;Y)=0$ (Lemma 28). □

Lemma 36.

There exists a Markov chain $X⊸--Y⊸--Z$ for which $K_2(X;Z)>K_2(X;Y)$.

Proof. 

Let the Markov chain $X⊸--Y⊸--Z$ be given by

$P_{Z|Y}$$P_{XY}(x,y)$	$y=0$	$y=1$
$x=0$	$0.6$	0
$x=1$	0	$0.4$

$P_{Z|Y}$$P_{Z|Y}\left(z\right|y)$	$z=0$	$z=1$
$y=0$	$0.9$	$0.1$
$y=1$	0	1

Using Lemma 27, we see that $K_2(X;Z)\approx 0.605$ bits, which is larger than $K_2(X;Y)\approx 0.531$ bits. □