A Forward-Reverse Brascamp-Lieb Inequality: Entropic Duality and Gaussian Optimality

Abstract

Inspired by the forward and the reverse channels from the image-size characterization problem in network information theory, we introduce a functional inequality that unifies both the Brascamp-Lieb inequality and Barthe’s inequality, which is a reverse form of the Brascamp-Lieb inequality. For Polish spaces, we prove its equivalent entropic formulation using the Legendre-Fenchel duality theory. Capitalizing on the entropic formulation, we elaborate on a “doubling trick” used by Lieb and Geng-Nair to prove the Gaussian optimality in this inequality for the case of Gaussian reference measures.

1. Introduction

The Brascamp-Lieb inequality and its reverse [1] concern the optimality of Gaussian functions in a certain type of integral inequality. (Not to be confused with the “variance Brascamp-Lieb inequality” (cf. [2,3,4]), which generalizes the Poincaré inequality). These inequalities have been generalized in various ways since their discovery, nearly 40 years ago. A modern formulation due to Barthe [5] may be stated as follows:

Brascamp-Lieb Inequality and Its Reverse 

([5] Theorem 1). Let E, $E_1$, …, $E_m$ be Euclidean spaces and ${\mathbf{B}}_i:E\to E_i$ be linear maps. Let ${(c_i)}_{i=1}^m$ and D be positive real numbers. Then, the Brascamp-Lieb inequality:

$$\begin{array}{c}\hfill \int \prod _{i=1}^m{f}_i^{c_i}({\mathbf{B}}_i\mathbf{x})\mathrm{d}\mathbf{x}\le D\prod _{i=1}^m{\left(\int f_i({\mathbf{x}}_i)\mathrm{d}{\mathbf{x}}_i\right)}^{c_i},\end{array}$$

(1)

for all nonnegative measurable functions $f_i$ on $E_i$, $i=1,\dots ,m$, holds if and only if it holds whenever $f_i$, $i=1,\dots ,m$ are centered Gaussian functions (a centered Gaussian function is of the form $\mathbf{x}\mapsto exp(r-{\mathbf{x}}^{\top }\mathbf{A}\mathbf{x})$, where $\mathbf{A}$ is a positive semidefinite matrix and $r\in \mathbb{R}$). Similarly, for F a positive real number, the reverse Brascamp-Lieb inequality, also known as Barthe’s inequality (${\mathbf{B}}_i^{\ast }$ denotes the adjoint of ${\mathbf{B}}_i$),

$$\begin{array}{c}\hfill \int \underset{({\mathbf{y}}_i):{\sum }_{i=1}^m{c}_i{\mathbf{B}}_i^{\ast }{\mathbf{y}}_i=\mathbf{x}}{sup}\prod _{i=1}^m{f}_i^{c_i}({\mathbf{y}}_i)\mathrm{d}\mathbf{x}\ge F\prod _{i=1}^m{\left(\int f_i({\mathbf{y}}_i)\mathrm{d}{\mathbf{y}}_i\right)}^{c_i},\end{array}$$

(2)

for all nonnegative measurable functions $f_i$ on $E_i$, $i=1,\dots ,m$, holds if and only if it holds for all centered Gaussian functions.

For surveys on the history of both the Brascamp-Lieb inequality and Barthe’s inequality and their applications, see, e.g., [6,7]. The Brascamp-Lieb inequality can be seen as a generalization of several other inequalities, including Hölder’s inequality, the sharp Young inequality, the Loomis-Whitney inequality, the entropy power inequality (cf. [6] or the survey paper [8]), hypercontractivity and the logarithmic Sobolev inequality [9]. Furthermore, the Prékopa-Leindler inequality can be seen as a special case of Barthe’s inequality. Due in part to their utility in establishing impossibility bounds, these functional inequalities have attracted much attention in information theory [10,11,12,13,14,15,16,17], theoretical computer science [18,19,20,21,22] and statistics [23,24,25,26,27,28], to name only a small subset of the literature. Over the years, various proofs of these inequalities have been proposed [1,29,30,31,32,33,34]. Among these, Lieb’s elegant proof [29], which is very close to one of the techniques that will be used in this paper, employs a doubling trick that capitalizes on the rotational invariance property of the Gaussian function: if f is a one-dimensional Gaussian function, then:

$$\begin{array}{c}\hfill f(x)f(y)=f\left(\frac{x-y}{\sqrt{2}}\right)f\left(\frac{x+y}{\sqrt{2}}\right).\end{array}$$

(3)

Since (1) and (2) have the same structure modulo the direction of the inequality, a common viewpoint is to consider (1) and (2) as dual inequalities. This viewpoint successfully captures the geometric aspects of (1) and (2). Indeed, it is known that:

$$\begin{array}{c}\hfill D·F=1\end{array}$$

(4)

as long as $D,F<\infty$ [5]. Moreover, both D and F are equal to one under Ball’s geometric condition [35]: $E_1$, …, $E_m$ are dimension one, and:

$$\begin{array}{c}\hfill \sum _{i=1}^m{c}_i{\mathbf{B}}_i{\mathbf{B}}_i^{\ast }=\mathbf{I}\end{array}$$

(5)

is the identity matrix. While fruitful, this “dual” viewpoint does not fully explain the asymmetry between the forward and the reverse inequalities: there is a sup in (2), but not in (1).

This paper explores a different viewpoint. In particular, we propose a single inequality that unifies (1) and (2). Accordingly, we should reverse both sides of (2) to make the inequality sign consistent with (1). To be concrete, let us first observe that (1) and (2) can be respectively restated in the following more symmetrical forms (with changes of certain symbols):

• For all nonnegative functions g and $f_1,\dots ,f_m$ such that:

  $$\begin{array}{c}\hfill g(\mathbf{x})\le \prod _{i=1}^m{f}_j^{c_j}({\mathbf{B}}_j\mathbf{x}),\forall \mathbf{x},\end{array}$$

  (6)

  we have:

  $$\begin{array}{c}\hfill {\int }_{E}g\le D\prod _{j=1}^m{\left({\int }_{E_j}{f}_j\right)}^{c_j}.\end{array}$$

  (7)
• For all nonnegative measurable functions $g_1,\dots g_l$ and f such that:

  $$\begin{array}{c}\hfill \prod _{i=1}^l{g}_i^{b_i}({\mathbf{z}}_i)\le f(\sum _{i=1}^l{b}_i{\mathbf{B}}_i^{\ast }{\mathbf{z}}_i),\forall {\mathbf{z}}_1,\dots ,{\mathbf{z}}_l,\end{array}$$

  (8)

  we have:

  $$\begin{array}{c}\hfill \prod _{i=1}^l{\left({\int }_{E_i}{g}_i\right)}^{b_i}\le D{\int }_{E}f.\end{array}$$

  (9)

Note that in both cases, the optimal choice of one function (f or g) can be explicitly computed from the constraints, hence the conventional formulations in (1) and (2). Generalizing further, we can consider the following problem: Let $\mathcal{X}$, ${\mathcal{Y}}_1,\dots ,{\mathcal{Y}}_m$, ${\mathcal{Z}}_1,\dots ,{\mathcal{Z}}_l$ be measurable spaces. Consider measurable maps ${\varphi }_j:\mathcal{X}\to {\mathcal{Y}}_j$, $j=1,\dots ,m$ and $\psi :\mathcal{X}\to {\mathcal{Z}}_i$, $i=1,\dots ,l$. Let $b_1,\dots ,b_l$ and $c_1,\dots ,c_m$ be nonnegative real numbers. Let ${\nu }_1,\dots ,{\nu }_l$ be measures on ${\mathcal{Z}}_1,\dots ,{\mathcal{Z}}_l$ and ${\mu }_1,\dots ,{\mu }_m$ be measures on ${\mathcal{Y}}_1,\dots ,{\mathcal{Y}}_m$, respectively. What is the smallest $D>0$ such that for all nonnegative $f_1,\dots ,f_m$ on ${\mathcal{Y}}_1,\dots {\mathcal{Y}}_m$ and $g_1,\dots ,g_l$ on ${\mathcal{Z}}_1,\dots ,{\mathcal{Z}}_l$ satisfying:

$$\begin{array}{c}\hfill \prod _{i=1}^l{g}_i^{b_i}({\psi }_i(x))\le \prod _{j=1}^m{f}_j^{c_j}({\varphi }_j(x)),\forall x,\end{array}$$

(10)

we have:

$$\begin{array}{c}\hfill \prod _{i=1}^l{\left(\int g_i\mathrm{d}{\nu }_i\right)}^{b_i}\le D\prod _{j=1}^m{\left(\int f_j\mathrm{d}{\mu }_j\right)}^{c_j}?\end{array}$$

(11)

Except for special case of $l=1$ (resp. $m=1$), it is generally not possible to deduce a simple expression from (10) for the optimal choice of $g_i$ (resp. $f_j$) in terms of the rest of the functions. We will refer to (11) as a forward-reverse Brascamp-Lieb inequality.

One of the motivations for considering multiple functions on both sides of (11) comes from multiuser information theory: independently, but almost simultaneously with the discovery of the Brascamp-Lieb inequality in mathematical physics, in the late 1970s, information theorists including Ahslwede, Gács and Körner [36,37] invented the image-size technique for proving strong converses in source and channel networks. An image-size inequality is a characterization of the tradeoff of the measures of certain sets connected by given random transformations (channels); we refer the interested readers to [37] for expositions on the image-size problem. Although not the way treated in [36,37], an image-size inequality can essentially be obtained from a functional inequality similar to (11) by taking the functions to be (roughly speaking) the indicator functions of sets. In the case of (10), the forward channels ${\varphi }_1,\dots ,{\varphi }_m$ and the reverse channels ${\psi }_1,\dots ,{\psi }_l$ degenerate into deterministic functions. In this paper, motivated by information theoretic applications similar to those of the image-size problems, we will consider further generalizations of (11) to the case of random transformations. Since the functional inequality is not restricted to indicator functions, it is strictly stronger than the corresponding image-size inequality. As a side remark, [38] uses functional inequalities that are variants of (11) together with a reverse hypercontractivity machinery to improve the image-size plus the blowing-up machinery of [39] and shows that the non-indicator function generalization is crucial for achieving the optimal scaling of the second-order rate expansion.

Of course, to justify the proposal of (11), we must also prove that (11) enjoys certain nice mathematical properties; this is the main goal of the present paper. Specifically, we focus on two aspects of (11): equivalent entropic formulation and Gaussian optimality.

In the mathematical literature, e.g., [32,36,40,41,42,43,44,45,46], it is known that certain integral inequalities are equivalent to inequalities involving relative entropies. In particular, Carlen, Loss and Lieb [47] and Carlen and Cordero-Erausquin [32] proved that the Brascamp-Lieb inequality is equivalent to the superadditivity of relative entropy. In this paper, we prove that the forward-reverse Brascamp-Lieb inequality (11) also has an entropic formulation, which turns out to be very close to the rate region of certain multiuser information theory problems (but we will clarify the difference in the text). In fact, Ahlswede, Csiszár and Körner [37,39] essentially derived image-size inequalities from similar entropic inequalities. Because of the reverse part, the proof of the equivalence of (11) and corresponding entropic inequality is more involved than the forward case considered in [32] beyond the case of finite $\mathcal{X}$, ${\mathcal{Y}}_j$, ${\mathcal{Z}}_i$, and certain machinery from min-max theory appears necessary. In particular, the proof involves a novel use of the Legendre-Fenchel duality theory. Next, we give a basic version of our main result on the functional-entropic duality (more general versions will be given later). In order to streamline its presentation, all formal definitions of notation are postponed to Section 2.

Theorem 1 (Dual formulation of the forward-reverse Brascamp-Lieb inequality).

Assume that:

(i) 

m and l are positive integers; $d\in \mathbb{R}$, $\mathcal{X}$ is a compact metric space;

(ii) 

$b_i\in (0,\infty )$, ${\nu }_i$ is a finite Borel measure on a Polish space ${\mathcal{Z}}_i$, and $Q_{Z_i|X}$ is a random transformation from $\mathcal{X}$ to ${\mathcal{Z}}_i$, for each $i=1,\dots ,l$;

(iii) 

$c_j\in (0,\infty )$, ${\mu }_j$ is a finite Borel measure on a Polish space ${\mathcal{Y}}_j$, and $Q_{Y_j|X}$ is a random transformation from $\mathcal{X}$ to ${\mathcal{Y}}_i$, for each $j=1,\dots ,m$;

(iv) 

For any ${(P_{Z_i})}_{i=1}^l$ such that ${\sum }_{i=1}^{l}D(P_{Z_i}\parallel {\nu }_i)<\infty$, there exists $P_X$ such that $P_X\to Q_{Z_i|X}\to P_{Z_i}$, $i=1,\dots ,l$ and ${\sum }_{j=1}^{m}D(P_{Y_j}\parallel {\mu }_j)<\infty$, where $P_X\to Q_{Y_j|X}\to P_{Y_j}$, $j=1,\dots ,m$.

Then, the following two statements are equivalent:

1.

If the nonnegative continuous functions $(g_i)$, $(f_j)$ are bounded away from zero and satisfy:

$$\begin{array}{c}\hfill \sum _{i=1}^l{b}_i{Q}_{Z_i|X}(g_i)\le \sum _{j=1}^m{c}_j{Q}_{Y_j|X}(f_j)\end{array}$$

(12)

then:

$$\begin{array}{c}\hfill \prod _{i=1}^l{\left(\int g_i\mathrm{d}{\nu }_i\right)}^{b_i}\le exp(d)\prod _{j=1}^m{\left(\int f_j\mathrm{d}{\mu }_j\right)}^{c_j}\end{array}$$

(13)

2.

For any $(P_{Z_i})$ such that $D(P_{Z_i}\parallel {\nu }_i)<\infty$ (of course, this assumption is not essential (if we adopt the convention that the infimum in (14) is $+\infty$ when it runs over an empty set)), $i=1,\dots ,l$,

$$\begin{array}{c}\hfill \sum _{i=1}^l{b}_{i}D(P_{Z_i}\parallel {\nu }_i)+d\ge \underset{P_X}{inf}\sum _{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)\end{array}$$

(14)

where $P_X\to Q_{Y_j|X}\to P_{Y_j}$, $j=1,\dots ,m$, and the infimum is over $P_X$ such that $P_X\to Q_{Z_i|X}\to P_{Z_i}$, $i=1,\dots ,l$.

Next, in a similar vein as the proverbial result that “Gaussian functions are optimal” for the forward or the reverse Brascamp-Lieb inequality, we show in this paper that Gaussian functions are also optimal for the forward-reverse Brascamp-Lieb inequality, particularized to the case of Gaussian reference measures and linear maps. The proof scheme is based on rotational invariance (3), which can be traced back in the functional setting to Lieb [29]. More specifically, we use a variant for the entropic setting introduced by Geng and Nair [48], thereby taking advantage of the dual formulation of Theorem 1.

Theorem 2.

Consider $b_1,\dots ,b_l,c_1,\dots ,c_m,D\in (0,\infty )$. Let $E_1,\dots ,E_l,E^1,\dots ,E^m$ be Euclidean spaces, and let ${\mathbf{B}}_{ji}:E_i\to E^j$ be a linear map for each $i\in \{1,\dots ,l\}$ and $j\in \{1,\dots ,m\}$. Then, for all continuous functions $f_j:E^j\to [0,+\infty )$, $g_i:E_i\to [0,\infty )$ satisfying:

$$\begin{array}{c}\hfill \prod _{i=1}^l{g}_i^{b_i}({\mathbf{x}}_i)\le \prod _{j=1}^m{f}_j^{c_j}\left(\sum _{i=1}^l{\mathbf{B}}_{ji}{\mathbf{x}}_i\right),\forall {\mathbf{x}}_1,\dots ,{\mathbf{x}}_l,\end{array}$$

(15)

we have:

$$\begin{array}{c}\hfill \prod _{i=1}^l{\left(\int g_i\right)}^{b_i}\le D\prod _{j=1}^m{\left(\int f_j\right)}^{c_j},\end{array}$$

(16)

if and only if for all centered Gaussian functions $f_1,\dots ,f_m,g_1,\dots ,g_l$ satisfying (15), we have (16).

As mentioned, in the literature on the forward or the reverse Brascamp-Lieb inequalities, it is known that a certain geometric condition (5) ensures that the best constant equals one. Now, for the forward-reverse inequality, there is a simple example where the best constant equals one:

Example 1.

Let l be a positive integer, and let $\mathbf{M}:={(m_{ji})}_{1\le j\le l,1\le i\le l}$ be an orthogonal matrix. For any nonnegative continuous functions ${(f_j)}_{j=1}^l$${(g_i)}_{i=1}^l$ on $\mathbb{R}$ such that:

$$\begin{array}{c}\hfill \prod _{i=1}^l{g}_i(x_i)\le \prod _{j=1}^l{f}_j\left(\sum _{i=1}^l{m}_{ji}{x}_i\right),\forall x^l\in {\mathbb{R}}^l,\end{array}$$

(17)

we have:

$$\begin{array}{c}\hfill \prod _{i=1}^l\int g_i(x)\mathrm{d}x\le \prod _{i=1}^l\int f_j(x)\mathrm{d}x.\end{array}$$

(18)

The rest of the paper is organized as follows: Section 2 defines the notation and reviews some basic theory of convex duality. Section 3 proves Theorem 1 and also presents its extensions to the settings of noncompact spaces or general reverse channels. Section 4 proves the Gaussian optimality in the entropic formulation, with the caveat that a certain “non-degenerate” assumption is imposed to ensure the existence of extremizers. At the end of Section 4, we give a proof sketch of Example 1 and also propose a generalization of the example. To completely prove Theorem 2, in Appendix F, we use a limiting argument to drop the non-degenerate assumption and apply the equivalence between the functional and entropic formulations.

2. Review of the Legendre-Fenchel Duality Theory

Our proof of the equivalence of the functional and the entropic inequalities uses the Legendre-Fenchel duality theory, a topic from convex analysis. Before getting into that, a recap of some basics on the duality of topological vector spaces seems appropriate. Unless otherwise indicated, we assume Polish spaces and Borel measures. Recall that metric space. It enjoys several nice properties that we use heavily in this section, including the Prokhorov theorem and the Riesz-Kakutani theorem. Of course, the Polish space assumption covers the cases of Euclidean and discrete spaces (endowed with the Hamming metric, which induces the discrete topology, making every function on the discrete set continuous), among others. Readers interested in discrete spaces only may refer to the (much simpler) argument in [49] based on the KKT condition.

Notation 1.

Let $\mathcal{X}$ be a topological space.

• $C_c(\mathcal{X})$ denotes the space of continuous functions on $\mathcal{X}$ with a compact support;
• $C_0(\mathcal{X})$ denotes the space of all continuous functions f on $\mathcal{X}$ that vanish at infinity (i.e., for any $ϵ>0$, there exists a compact set $\mathcal{K}\subseteq \mathcal{X}$ such that $|f(x)|<ϵ$ for $x\in \mathcal{X}\backslash \mathcal{K}$);
• $C_b(\mathcal{X})$ denotes the space of bounded continuous functions on $\mathcal{X}$;
• $\mathcal{M}(\mathcal{X})$ denotes the space of finite signed Borel measures on $\mathcal{X}$;
• $\mathcal{P}(\mathcal{X})$ denotes the space of probability measures on $\mathcal{X}$.

We consider $C_c$, $C_0$ and $C_b$ as topological vector spaces, with the topology induced from the sup norm. The following theorem, usually attributed to Riesz, Markov and Kakutani, is well known in functional analysis and can be found in, e.g., [50,51].

Theorem 3 (Riesz-Markov-Kakutani).

If $\mathcal{X}$ is a locally compact, σ-compact Polish space, the dual (the dual of a topological vector space consists of all continuous linear functionals on that space, which is naturally also topological vector space (with the weak${}^{\ast }$ topology)) of both $C_c(\mathcal{X})$ and $C_0(\mathcal{X})$ is $\mathcal{M}(\mathcal{X})$.

Remark 1.

The dual space of $C_b(\mathcal{X})$ can be strictly larger than $\mathcal{M}(\mathcal{X})$, since it also contains those linear functionals that depend on the “limit at infinity” of a function $f\in C_b(\mathcal{X})$ (originally defined for those f that do have a limit at infinity and then extended to the whole $C_b(\mathcal{X})$ by the Hahn-Banach theorem; see, e.g., [50]).

Of course, any $\mu \in \mathcal{M}(\mathcal{X})$ is a continuous linear functional on $C_0(\mathcal{X})$ or $C_c(\mathcal{X})$, given by:

$$\begin{array}{c}\hfill f\mapsto \int f\mathrm{d}\mu \end{array}$$

(19)

where f is a function in $C_0(\mathcal{X})$ or $C_c(\mathcal{X})$. As is well known, Theorem 3 states that the converse is also true under mild regularity assumptions on the space. Thus, we can view measures as continuous linear functionals on a certain function space (in fact, some authors prefer to construct measure theory by defining a measure as a linear functional on a suitable measure space; see Lax [50] or Bourbaki [52]); this justifies the shorthand notation:

$$\begin{array}{c}\hfill \mu (f):=\int f\mathrm{d}\mu \end{array}$$

(20)

which we employ in the rest of the paper. This viewpoint is the most natural for our setting since in the proof of the equivalent formulation of the forward-reverse Brascamp-Lieb inequality, we shall use the Hahn-Banach theorem to show the existence of certain linear functionals.

Definition 1.

Let $\mathsf{\Lambda }:C_b(\mathcal{X})\to (-\infty ,+\infty ]$ be a lower semicontinuous, proper convex function. Its Legendre-Fenchel transform ${\mathsf{\Lambda }}^{\ast }:{\mathcal{C}}_b{(\mathcal{X})}^{\ast }\to (-\infty ,+\infty ]$ is given by:

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}^{\ast }(\ell ):=\underset{u\in C_b(\mathcal{X})}{sup}[\ell (u)-\mathsf{\Lambda }(u)].\end{array}$$

(21)

Let $\nu$ be a nonnegative finite Borel measure on a Polish space $\mathcal{X}$, and define the convex functional on $C_b(\mathcal{X})$:

$$\begin{array}{cc}\hfill \mathsf{\Lambda }(f)& :=log\nu (exp(f))\hfill \end{array}$$

(22)

$$\begin{array}{c}=log\int exp(f)\mathrm{d}\nu .\hfill \end{array}$$

(23)

Then, note that the relative entropy has the following alternative definition: for any $\mu \in \mathcal{M}(\mathcal{X})$,

$$\begin{array}{c}\hfill D(\mu \parallel \nu ):=\underset{f\in C_b(\mathcal{X})}{sup}[\mu (f)-\mathsf{\Lambda }(f)]\end{array}$$

(24)

which agrees with the more familiar definition $D(\mu \parallel \nu ):=\mu (log\frac{\mathrm{d}\mu }{\mathrm{d}\nu })$ when $\nu$ is a probability measure, by the Donsker-Varadhan formula (cf. [53] Lemma 6.2.13). If $\mu$ is not a probability measure, then $D(\mu \parallel \nu )$ as defined in (24) is $+\infty$.

Given a bounded linear operator $T:C_b(\mathcal{Y})\to C_b(\mathcal{X})$, the dual operator $T^{\ast }:C_b{(\mathcal{X})}^{\ast }\to C_b{(\mathcal{Y})}^{\ast }$ is defined in terms of:

$$\begin{array}{cc}\hfill T^{\ast }{\mu }_X& :C_b(\mathcal{Y})\to \mathbb{R};\hfill \\ \hfill f& \mapsto {\mu }_X(Tf),\hfill \end{array}$$

(25)

for any ${\mu }_X\in C_b{(\mathcal{X})}^{\ast }$. Since $\mathcal{P}(\mathcal{X})\subseteq \mathcal{M}(\mathcal{X})\subseteq C_b{(\mathcal{X})}^{\ast }$, T is said to be a conditional expectation operator if $T^{\ast }P\in \mathcal{P}(\mathcal{Y})$ for any $P\in \mathcal{P}(\mathcal{X})$. The operator $T^{\ast }$ is defined as the dual of a conditional expectation operator T and, in a slight abuse of terminology, is said to be a random transformation from $\mathcal{X}$ to $\mathcal{Y}$.

For example, in the notation of Theorem 1, if $g\in C_b(\mathcal{Y})$ and $Q_{Y|X}$ is a random transformation from $\mathcal{X}$ to $\mathcal{Y}$, the quantity $Q_{Y|X}(g)$ is a function on $\mathcal{X}$, defined by taking the conditional expectation. Furthermore, if $P_X\in \mathcal{P}(\mathcal{X})$, we write $P_X\to Q_{Y|X}\to P_Y$ to indicate that $P_Y\in \mathcal{P}(\mathcal{Y})$ is the measure induced on $\mathcal{Y}$ by applying $Q_{Y|X}$ to $P_X$.

Remark 2.

From the viewpoint of category theory (see for example [54,55]), $C_b$ is a functor from the category of topological spaces to the category of topological vector spaces, which is contra-variant because for any continuous, $\varphi :\mathcal{X}\to \mathcal{Y}$ (morphism between topological spaces), we have $C_b(\varphi ):C_b(\mathcal{Y})\to C_b(\mathcal{X})$, $u\mapsto u\circ f$ where $u\circ \varphi$ denotes the composition of two continuous functions, reversing the arrows in the maps (i.e., the morphisms). On the other hand, $\mathcal{M}$ is a covariant functor and $\mathcal{M}(\varphi ):\mathcal{M}(\mathcal{X})\to \mathcal{M}(\mathcal{Y})$, $\mu \mapsto \mu \circ {\varphi }^{-1}$, where $\mu \circ {\varphi }^{-1}(\mathcal{B}):=\mu ({\varphi }^{-1}(\mathcal{B}))$ for any Borel measurable $\mathcal{B}\subseteq \mathcal{Y}$. “Duality” itself is a contra-variant functor between the category of topological spaces (note the reversal of arrows in Figure 1). Moreover, $C_b{(\mathcal{X})}^{\ast }=\mathcal{M}(\mathcal{X})$ and $C_b{(\varphi )}^{\ast }=\mathcal{M}(\varphi )$ if $\mathcal{X}$ and $\mathcal{Y}$ are compact metric spaces and $\varphi :\mathcal{X}\to \mathcal{Y}$ is continuous. Definition 2 can therefore be viewed as the special case where ϕ is the projection map:

Definition 2.

Suppose $\varphi :{\mathcal{Z}}_1\times {\mathcal{Z}}_2\to {\mathcal{Z}}_1,(z_1,z_2)\mapsto z_1$ is the projection to the first coordinate.

• $C_b(\varphi ):C_b({\mathcal{Z}}_1)\to C_b({\mathcal{Z}}_1\times {\mathcal{Z}}_2)$ is called a canonical map, whose action is almost trivial: it sends a function of $z_i$ to itself, but viewed as a function of $(z_1,z_2)$.
• $\mathcal{M}(\varphi ):\mathcal{M}({\mathcal{Z}}_1\times {\mathcal{Z}}_2)\to \mathcal{M}({\mathcal{Z}}_1)$ is called marginalization, which simply takes a joint distribution to a marginal distribution.

The Fenchel-Rockafellar duality (see [40] Theorem 1.9, or [56] in the case of finite dimensional vector spaces) usually refers to the $k=1$ special case of the following result.

Theorem 4.

Assume that A is a topological vector space whose dual is $A^{\ast }$. Let ${\mathsf{\Theta }}_j:A\to \mathbb{R}\cup \{+\infty \}$, $j=0,1,\dots ,k$, for some positive integer k. Suppose there exist some ${(u_j)}_{j=1}^k$ and $u_0:=-(u_1+\dots +u_k)$ such that:

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_j(u_j)<\infty ,j=0,\dots ,k\end{array}$$

(26)

and ${\mathsf{\Theta }}_0$ is upper semicontinuous at $u_0$. Then:

$$\begin{array}{c}\hfill -\underset{\ell \in A^{\ast }}{inf}\left[\sum _{j=0}^k{\mathsf{\Theta }}_j^{\ast }(\ell )\right]=\underset{u_1,\dots ,u_k\in A}{inf}\left[{\mathsf{\Theta }}_0\left(-\sum _{j=1}^k{u}_j\right)+\sum _{j=1}^k{\mathsf{\Theta }}_j(u_j)\right].\end{array}$$

(27)

For completeness, we provide a proof of this result, which is based on the Hahn-Banach theorem (Theorem 5) and is similar to the proof of [40] Theorem 1.9.

Proof. 

Let $m_0$ be the right side of (27). The ≤ part of (27) follows trivially from the (weak) min-max inequality since:

$$\begin{array}{cc}\hfill m_0& =\underset{u_0,\dots ,u_k\in A}{inf}\underset{\ell \in A^{\ast }}{sup}\left\{\sum _{j=0}^k{\mathsf{\Theta }}_j(u_j)-\ell (\sum _{j=0}^k{u}_j)\right\}\hfill \\ \hfill & \ge \underset{\ell \in A^{\ast }}{sup}\underset{u_0,\dots ,u_k\in A}{inf}\left\{\sum _{j=0}^k{\mathsf{\Theta }}_j(u_j)-\ell (\sum _{j=0}^k{u}_j)\right\}\hfill \\ \hfill & =-\underset{\ell \in A^{\ast }}{inf}\left[\sum _{j=0}^k{\mathsf{\Theta }}_j^{\ast }(\ell )\right].\hfill \end{array}$$

(28)

It remains to prove the ≥ part, and it suffices to assume without loss of generality that $m_0>-\infty$. Note that (26) also implies that $m_0<+\infty$. Define convex sets:

$$\begin{array}{cc}\hfill C_j& :=\{(u,r)\in A\times \mathbb{R}:r>{\mathsf{\Theta }}_j(u)\},j=0,\dots ,k;\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill B& :=\{(0,m)\in A\times \mathbb{R}:m\le m_0\}.\hfill \end{array}$$

(30)

Observe that these are nonempty sets because of (26). Furthermore, $C_0$ has a nonempty interior by the assumption that ${\mathsf{\Theta }}_0$ is upper semicontinuous at $u_0$. Thus, the Minkowski sum:

$$\begin{array}{c}\hfill C:=C_0+\dots +C_k\end{array}$$

(31)

is a convex set with a nonempty interior. Moreover, $C\cup B=\mathsf{\varnothing }$. By the Hahn-Banach theorem (Theorem 5), there exists $(\ell ,s)\in A^{\ast }\times \mathbb{R}$ such that:

$$\begin{array}{c}\hfill sm\le \ell \left(\sum _{j=0}^k{u}_j\right)+s\sum _{j=0}^k{r}_j.\end{array}$$

(32)

For any $m\le m_0$ and $(u_j,r_j)\in C_j$, $j=0,\dots ,k$. From (30), we see (32) can only hold when $s\ge 0$. Moreover, from (26) and the upper semicontinuity of ${\mathsf{\Theta }}_0$ at $u_0$, we see that the ${\sum }_{j=0}^k{u}_j$ in (32) can take a value in a neighborhood of $0\in A$; hence, $s\ne 0$. Thus, by dividing s on both sides of (32) and setting $\ell \leftarrow -\ell /s$, we see that:

$$\begin{array}{cc}\hfill m_0& \le \underset{u_0,\dots ,u_k\in A}{inf}\left[-\ell \left(\sum _{j=0}^k{u}_j\right)+\sum _{j=0}^k{\mathsf{\Theta }}_j(u_j)\right]\hfill \\ \hfill & =-\left[\sum _{j=0}^k{\mathsf{\Theta }}_j^{\ast }(\ell )\right]\hfill \end{array}$$

(33)

which establishes ≥ in (27). ☐

Theorem 5 (Hahn-Banach)

Let C and B be convex, nonempty disjoint subsets of a topological vector space A.

1.

If the interior of C is non-empty, then there exists $\ell \in A^{\ast }$, $\ell \ne 0$ such that:

$$\begin{array}{c}\hfill \underset{u\in B}{sup}\ell (u)\le \underset{u\in C}{inf}\ell (u).\end{array}$$

(34)

2.

If A is locally convex, B is compact and C is closed, then there exists $\ell \in A^{\ast }$ such that:

$$\begin{array}{c}\hfill \underset{u\in B}{sup}\ell (u)<\underset{u\in C}{inf}\ell (u).\end{array}$$

(35)

Remark 3.

The assumption in Theorem 5 that C has a nonempty interior is only necessary in the infinite dimensional case. However, even if A in Theorem 4 is finite dimensional, the assumption in Theorem 4 that ${\mathsf{\Theta }}_0$ is upper semicontinuous at $u_0$ is still necessary, because this assumption was not only used in applying Hahn-Banach, but also in concluding that $s\ne 0$ in (32).

3. The Entropic-Functional Duality

In this section, we prove Theorem 1 and some of its generalizations.

3.1. Compact $\mathcal{X}$

We first state a duality theorem for the case of compact spaces to streamline the proof. Later, we show that the argument can be extended to a particular non-compact case (Theorem 1 is not included in the conference paper [49], but was announced in the conference presentation). Our proof based on the Legendre-Fenchel duality (Theorem 4) was inspired by the proof of the Kantorovich duality in the theory of optimal transportation (see [40] Chapter 1, where the idea was credited to Brenier).

Recall from Section 2 that a random transformation (a mapping between probability measures) is formally the dual of a conditional expectation operator. Suppose $P_{Y_j|X}=T_j^{\ast }$, $j=1,\dots ,m$ and $P_{Z_i|X}=S_i^{\ast }$, $i=1,\dots ,l$.

Proof of Theorem 1. 

We can safely assume $d=0$ below without loss of generality (since otherwise, we can always substitute ${\mu }_1\leftarrow exp\left(\frac{d}{c_1}\right){\mu }_1$).

1)⇒2) 

This is the nontrivial direction, which relies on certain (strong) min-max type results. In Theorem 4, put (in (36), $u\le 0$ means that u is pointwise non-positive):

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_0:u\in C_b(\mathcal{X})\mapsto \left\{\begin{array}{cc}0& u\le 0;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\end{array}$$

(36)

Then,

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_0^{\ast }:\pi \in \mathcal{M}(\mathcal{X})\mapsto \left\{\begin{array}{cc}0& \pi \ge 0;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\end{array}$$

(37)

For each $j=1,\dots ,m$, set:

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_j(u):=c_{j}inflog{\mu }_j\left(exp\left(\frac{1}{c_j}v\right)\right)\end{array}$$

(38)

where the infimum is over $v\in C_b(\mathcal{Y})$ such that $u=T_{j}v$; if there is no such v, then ${\mathsf{\Theta }}_j(u):=+\infty$ as a convention. Observe that:

• ${\mathsf{\Theta }}_j$ is convex: indeed, given arbitrary $u^0$ and $u^1$, suppose that $v^0$ and $v^1$ respectively achieve the infimum in (38) for $u^0$ and $u^1$ (if the infimum is not achievable, the argument still goes through by the approximation and limit argument). Then, for any $\alpha \in [0,1]$, $v^{\alpha }:=(1-\alpha )v^0+\alpha v^1$ satisfies $u^{\alpha }=T_j{v}^{\alpha }$ where $u^{\alpha }:=(1-\alpha )u^0+\alpha u^1$. Thus, the convexity of ${\mathsf{\Theta }}_j$ follows from the convexity of the functional in (23);
• ${\mathsf{\Theta }}_j(u)>-\infty$ for any $u\in C_b(\mathcal{X})$. Otherwise, for any $P_X$ and $P_{Y_j}:=T_j^{\ast }{P}_X$, we have:

  $$\begin{array}{cc}\hfill D(P_{Y_j}\parallel {\mu }_j)& =\underset{v}{sup}\{P_{Y_j}(v)-log{\mu }_j(exp(v))\}\hfill \end{array}$$

  (39)

  $$\begin{array}{c}=\underset{v}{sup}\{P_X(T_{j}v)-log{\mu }_j(exp(v))\}\hfill \end{array}$$

  (40)

  $$\begin{array}{c}=\underset{u\in C_b(\mathcal{X})}{sup}\left\{P_X(u)-\frac{1}{c_j}{\mathsf{\Theta }}_j(c_{j}u)\right\}\hfill \end{array}$$

  (41)

  $$\begin{array}{c}=+\infty \hfill \end{array}$$

  (42)

  which contradicts the assumption that ${\sum }_{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)<\infty$ in the theorem;
• From Steps (39)–(41), we see ${\mathsf{\Theta }}_j^{\ast }(\pi )=c_{j}D(T_j^{\ast }\pi \parallel {\mu }_j)$ for any $\pi \in \mathcal{M}(\mathcal{X})$, where the definition of $D(·\parallel {\mu }_j)$ is extended using the Donsker-Varadhan formula (that is, it is infinite when the argument is not a probability measure).

Finally, for the given ${(P_{Z_i})}_{i=1}^l$, choose:

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_{m+1}:u\in C_b(\mathcal{X})\mapsto \left\{\begin{array}{cc}\sum _{i=1}^l{P}_{Z_i}(w_i)& \mathrm{if}u=\sum _{i=1}^l{S}_i{w}_i \mathrm{for} \mathrm{some} w_i\in C_b({\mathcal{Z}}_i);\\ +\infty & \mathrm{otherwise}.\end{array}\right.\end{array}$$

(43)

Notice that:

• ${\mathsf{\Theta }}_{m+1}$ is convex;
• ${\mathsf{\Theta }}_{m+1}$ is well defined (that is, the choice of $(w_i)$ in (43) is inconsequential). Indeed, if ${(w_i)}_{i=1}^l$ is such that ${\sum }_{i=1}^l{S}_i{w}_i=0$, then:

  $$\begin{array}{cc}\hfill \sum _{i=1}^l{P}_{Z_i}(w_i)& =\sum _{i=1}^l{S}_i^{\ast }{P}_X(w_i)\hfill \\ \hfill & =\sum _{i=1}^l{P}_X(S_i{w}_i)\hfill \\ \hfill & =0,\hfill \end{array}$$

  (44)

  where $P_X$ is such that $S_i^{\ast }{P}_X=P_{Z_i}$, $i=1,\dots ,l$, whose existence is guaranteed by the assumption of the theorem. This also shows that ${\mathsf{\Theta }}_{m+1}>-\infty$.
• $$\begin{array}{cc}\hfill {\mathsf{\Theta }}_{m+1}^{\ast }(\pi )& :=\underset{u}{sup}\{\pi (u)-{\mathsf{\Theta }}_{m+1}(u)\}\hfill \\ \hfill & =\underset{w_1,\dots ,w_l}{sup}\left\{\pi \left(\sum _{i=1}^l{S}_i{w}_i\right)-\sum _{i=1}^l{P}_{Z_i}(w_i)\right\}\hfill \\ \hfill & =\underset{w_1,\dots ,w_l}{sup}\left\{\sum _{i=1}^l{S}_i^{\ast }\pi (w_i)-\sum _{i=1}^l{P}_{Z_i}(w_i)\right\}\hfill \\ \hfill & =\left\{\begin{array}{cc}0& \mathrm{if}{S}_i^{\ast }\pi =P_{Z_i},i=1,\dots ,l;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\hfill \end{array}$$

  (45)

Invoking Theorem 4 (where the $u_j$ in Theorem 4 can be chosen as the constant function $u_j\equiv 1$, $j=1,\dots ,m+1$):

$$\begin{array}{c}\underset{\pi :\pi \ge 0,S_i^{\ast }\pi =P_{Z_i}}{inf}\sum _{j=1}^m{c}_{j}D(T_j^{\ast }\pi \parallel {\mu }_j)\hfill \\ =-\underset{v^m,w^l:{\sum }_{j=1}^m{T}_j{v}_j+{\sum }_{i=1}^l{S}_i{w}_i\ge 0}{inf}\left[\sum _{j=1}^m{c}_{j}log{\mu }_j\left(exp\left(\frac{1}{c_j}{v}_j\right)\right)+\sum _{i=1}^l{P}_{Z_i}(w_i)\right]\hfill \end{array}$$

(46)

where $v^m$ denotes the collection of the functions $v_1,\dots ,v_m$, and similarly for $w^l$. Note that the left side of (46) is exactly the right side of (14). For any $ϵ>0$, choose $v_j\in C_b({\mathcal{Y}}_j)$, $j=1,\dots ,m$ and $w_i\in C_b({\mathcal{Z}}_i)$, $i=1,\dots ,l$ such that ${\sum }_{j=1}^m{T}_j{v}_j+{\sum }_{i=1}^l{S}_i{w}_i\ge 0$ and:

$$\begin{array}{c}\hfill ϵ-\sum _{j=1}^m{c}_{j}log{\mu }_j\left(exp\left(\frac{1}{c_j}{v}_j\right)\right)-\sum _{i=1}^l{P}_{Z_i}(w_i)>\underset{\pi :\pi \ge 0,S_i^{\ast }\pi =P_{Z_i}}{inf}\sum _{j=1}^m{c}_{j}D(T_j^{\ast }\pi \parallel {\mu }_j)\end{array}$$

(47)

Now, invoking (13) with $f_j:=exp\left(\frac{1}{c_j}{v}_j\right)$, $j=1,\dots ,m$ and $g_i:=exp\left(-\frac{1}{b_i}{w}_i\right)$, $i=1,\dots ,l$, we upper bound the left side of (47) by:

$$\begin{array}{c}\hfill ϵ-\sum _{i=1}^l{b}_{i}log{\nu }_i(g_i)+\sum _{i=1}^l{b}_i{P}_{Z_i}(log{g}_i)\le ϵ+\sum _{i=1}^l{b}_{i}D(P_{Z_i}\parallel {\nu }_i)\end{array}$$

(48)

where the last step follows by the Donsker-Varadhan formula. Therefore, (14) is established since $ϵ>0$ is arbitrary.

2)⇒1) 

Since ${\nu }_i$ is finite and $g_i$ is bounded by assumption, we have ${\nu }_i(g_i)<\infty$, $i=1,\dots ,l$. Moreover, (13) is trivially true when ${\nu }_i(g_i)=0$ for some i, so we will assume below that ${\nu }_i(g_i)\in (0,\infty )$ for each i. Define $P_{Z_i}$ by:

$$\begin{array}{c}\hfill \frac{\mathrm{d}{P}_{Z_i}}{\mathrm{d}{\nu }_i}=\frac{g_i}{{\nu }_i(g_i)},i=1,\dots ,l.\end{array}$$

(49)

Then, for any $ϵ>0$,

$$\begin{array}{cc}\hfill \sum _{i=1}^l{b}_{i}log{\nu }_i(g_i)& =\sum _{i=1}^l{b}_i[P_{Z_i}(log{g}_i)-D(P_{Z_i}\parallel {\nu }_i)]\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill & <\sum _{j=1}^m{c}_j{P}_{Y_j}(log{f}_j)+ϵ-\sum _{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill & \le ϵ+\sum _{j=1}^m{c}_{j}log{\mu }_j(f_j)\hfill \end{array}$$

(52)

where:

• (51) uses the Donsker-Varadhan formula, and we have chosen $P_X$, $P_{Y_j}:=T_j^{\ast }{P}_X$, $j=1,\dots ,m$ such that:

  $$\begin{array}{c}\hfill \sum _{i=1}^l{b}_{i}D(P_{Z_i}\parallel {\nu }_i)>\sum _{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)-ϵ\end{array}$$

  (53)
• (52) also follows from the Donsker-Varadhan formula.

The result follows since $ϵ>0$ can be arbitrary.

☐

Remark 4.

Condition (iv) in the theorem imposes a rather strong assumption on $(S_i)$: for simplicity, consider the case where $\left|\mathcal{X}\right|,|Z_i|<\infty$. Then, Condition (iv) assumes that for any $(P_{Z_i})$, there exists $P_X$ such that $P_{Z_i}=S_i^{\ast }{P}_X$. This assumption is certainly satisfied when $(S_i)$ are induced by coordinate projections; the case of $l=1$ and $P_{Z|X}$ being a reverse erasure channel gives a simple example where $P_{Z|X}$ is not a deterministic map.

Next, we give a generalization of Theorem 1, which alleviates the restriction on $(S_i)$:

Theorem 6.

Theorem 1 continues to hold if Condition (iv) therein is weakened to the following:

• For any $P_X$ such that $D(S_i^{\ast }{P}_X\parallel {\nu }_i)<\infty$, $i=1,\dots ,l$, there exists ${\tilde{P}}_X$ such that $S_i^{\ast }{\tilde{P}}_X=S_i^{\ast }{P}_X$ for each i and ${\sum }_{j=1}^m{c}_{j}D(T_j^{\ast }{\tilde{P}}_X\parallel {\mu }_j)<\infty$ for each j.

and the conclusion of the theorem will be replaced by the equivalence of the following two statements:

1.

For any nonnegative continuous functions $(g_i)$, $(f_j)$ bounded away from zero and such that:

$$\begin{array}{c}\hfill \sum _{i=1}^l{b}_i{S}_{i}log{g}_i\le \sum _{j=1}^m{c}_j{T}_{j}log{f}_j\end{array}$$

(54)

we have:

$$\begin{array}{c}\hfill \underset{({\tilde{g}}_i):{\sum }_{i=1}^l{b}_i{S}_{i}log{\tilde{g}}_i\ge {\sum }_{i=1}^l{b}_i{S}_{i}log{g}_i}{inf}\prod _{i=1}^l{\nu }_i^{b_i}({\tilde{g}}_i)\le exp(d)\prod _{j=1}^m{\mu }_j^{c_j}(f_j).\end{array}$$

(55)

2.

For any $(P_X)$ such that $D(S_i^{\ast }{P}_X\parallel {\nu }_i)<\infty$, $i=1,\dots ,l$,

$$\begin{array}{c}\hfill \sum _{i=1}^l{b}_{i}D(S_i^{\ast }{P}_X\parallel {\nu }_i)+d\ge \underset{{\tilde{P}}_X:S_i^{\ast }{\tilde{P}}_X=S_i^{\ast }{P}_X}{inf}\sum _{j=1}^m{c}_{j}D(T_j^{\ast }{\tilde{P}}_X\parallel {\mu }_j).\end{array}$$

(56)

In Appendix A, we show that Theorem 6 indeed recovers Theorem 1 for the more restricted class of random transformations.

Proof. 

Here, we mention the parts of the proof that need to be changed: upon specifying $(f_j)$ and $(g_i)$ right after (47), we select $({\tilde{g}}_i)$ such that:

$$\begin{array}{cc}\hfill \sum _{i=1}^l{b}_i{S}_{i}log{\tilde{g}}_i& \ge \sum _{i=1}^l{b}_i{S}_{i}log{g}_i\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill \sum _{i=1}^l{b}_{i}log{\nu }_i({\tilde{g}}_i)& \le \sum _{j=1}^m{c}_{j}log{\mu }_j(f_j)+ϵ.\hfill \end{array}$$

(58)

Then, in lieu of (59), we upper-bound the left side of (47) by:

$$\begin{array}{c}\hfill 2ϵ-\sum _{i=1}^l{b}_{i}log{\nu }_i({\tilde{g}}_i)+\sum _{i=1}^l{b}_i{P}_{Z_i}(log{\tilde{g}}_i)\le 2ϵ+\sum _{i=1}^l{b}_{i}D(P_{Z_i}\parallel {\nu }_i)\end{array}$$

(59)

which establishes the 1)⇒2) part. For the other direction, for each $i\in \{1,2,\dots ,l\}$, define:

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_i(u):=\underset{{\tilde{g}}_i>0:b_i{S}_{i}log{\tilde{g}}_i=u}{inf}{b}_{i}log{\nu }_i({\tilde{g}}_i).\end{array}$$

(60)

Then, following essentially the same proof as that of ${\mathsf{\Theta }}_j$ in (38), we see that ${\mathsf{\Lambda }}_i$ is proper convex and:

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_i^{\ast }(\pi )=b_{i}D(S_i^{\ast }\pi \parallel {\mu }_j).\end{array}$$

(61)

Moreover, let:

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{l+1}(u):=\left\{\begin{array}{cc}0& \mathrm{if}u=-\sum b_i{S}_{i}log{g}_i;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\end{array}$$

(62)

Then, ${\mathsf{\Lambda }}_{l+1}^{\ast }(\pi )=-\sum b_i{S}_i^{\ast }\pi (log{g}_i)$. Using the Legendre-Fenchel duality, we see that for any $ϵ>0$,

$$\begin{array}{c}\underset{({\tilde{g}}_i):{\sum }_{i=1}^l{b}_i{S}_{i}log{\tilde{g}}_i\ge {\sum }_{i=1}^l{b}_i{S}_{i}log{g}_i}{inf}\sum _{i=1}^l{b}_{i}log{\nu }_i({\tilde{g}}_i)\hfill \end{array}$$

$$\begin{array}{c}=\underset{u_1,\dots ,u_{l+1}}{inf}\left\{{\mathsf{\Theta }}_0\left(-\sum _{i=1}^{l+1}{u}_i\right)+\sum _{i=1}^{l+1}{\mathsf{\Lambda }}_i(u_i)\right\}\hfill \end{array}$$

(63)

$$\begin{array}{c}=\underset{\pi }{sup}\left\{-\sum _{i=0}^{l+1}{\mathsf{\Theta }}_i^{\ast }(\pi )\right\}\hfill \end{array}$$

(64)

$$\begin{array}{c}=\underset{\pi \ge 0}{sup}\left\{-\sum _{i=1}^{l+1}{\mathsf{\Theta }}_i^{\ast }(\pi )\right\}\hfill \end{array}$$

(65)

$$\begin{array}{c}=\underset{\pi \ge 0}{sup}\left\{\sum _{i=1}^l{b}_i{S}_i^{\ast }\pi (log{g}_i)-\sum _{i=1}^l{b}_{i}D(S_i^{\ast }\pi \parallel {\nu }_i)\right\}\hfill \end{array}$$

(66)

$$\begin{array}{c}\le \sum _{i=1}^l{b}_i{S}_i^{\ast }{P}_X(log{g}_i)-\sum _{i=1}^l{b}_{i}D(S_i^{\ast }{P}_X\parallel {\nu }_i)+ϵ\hfill \end{array}$$

(67)

$$\begin{array}{c}\le \sum _{j=1}^m{c}_j{T}_j^{\ast }{\tilde{P}}_X(log{f}_j)-\sum _{j=1}^m{c}_{j}D(T_j^{\ast }{\tilde{P}}_X\parallel {\mu }_j)+2ϵ\hfill \end{array}$$

(68)

$$\begin{array}{c}\le 2ϵ+\sum _{j=1}^m{c}_{j}log{\mu }_j(f_j)\hfill \end{array}$$

(69)

where:

• To see (67), we note that the sup in (66) can be restricted to $\pi$, which is a probability measure, since otherwise, the relative entropy terms in (66) are $+\infty$ by its definition via the Donsker-Varadhan formula. Then, we select $P_X$ such that (67) holds.
• In (68), we have chosen ${\tilde{P}}_X$ such that:

  $$\begin{array}{cc}\hfill S_i^{\ast }{\tilde{P}}_X& =S_i^{\ast }{P}_X,1\le i\le l;\hfill \end{array}$$

  (70)

  $$\begin{array}{cc}\hfill \sum _{i=1}^l{b}_{i}D(S_i^{\ast }{P}_X)& >\sum _{j=1}^m{c}_{j}D(T_j^{\ast }{\tilde{P}}_X\parallel {\mu }_j)-ϵ,\hfill \end{array}$$

  (71)

  and then applied the assumption (54). The result follows since $ϵ>0$ can be arbitrary.
  ☐

Remark 5.

The infimum in (14) is in fact achievable: for any $(P_{Z_i})$, there exists a $P_X$ that minimizes ${\sum }_{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)$ subject to the constraints $S_i^{\ast }{P}_X=P_{Z_i}$, $i=1,\dots m$, where $P_{Y_j}:=T_j^{\ast }{P}_X$, $j=1,\dots ,m$. Indeed, since the singleton $\left\{P_{Z_i}\right\}$ is weak${}^{\ast }$-closed and $S_i^{\ast }$ is weak${}^{\ast }$-continuous (Generally, if $T:A\to B$ is a continuous map between two topologically vector spaces, then $T^{\ast }:B^{\ast }\to A^{\ast }$ is a weak${}^{\ast }$ continuous map between the dual spaces. Indeed, if $y_n\to y$ is a weak${}^{\ast }$-convergent subsequence in $B^{\ast }$, meaning $y_n(b)\to y(b)$ for any $b\in B$, then, we must have $T^{\ast }{y}_n(a)=y_n(Ta)\to y(Ta)=T^{\ast }y(a)$ for any $a\in A$, meaning that $T^{\ast }{y}_n$ converges to $T^{\ast }y$ in the weak${}^{\ast }$ topology.), the set ${\bigcap }_{i=1}^l{(S_i^{\ast })}^{-1}{P}_{Z_i}$ is weak${}^{\ast }$-closed in $\mathcal{M}(X)$; hence, its intersection with $\mathcal{P}(\mathcal{X})$ is weak${}^{\ast }$-compact in $\mathcal{P}(\mathcal{X})$, because $\mathcal{P}(\mathcal{X})$ is weak${}^{\ast }$-compact by (a simple version for the setting of a compact underlying space $\mathcal{X}$ of) the Prokhorov theorem [57]. Moreover, by the weak${}^{\ast }$-lower semicontinuity of $D(·\parallel {\mu }_j)$ (easily seen from the variational formula/Donsker-Varadhan formula of the relative entropy, cf. [58]) and the weak${}^{\ast }$-continuity of $T_j^{\ast }$, $j=1,\dots ,m$, we see that ${\sum }_{j=1}^m{c}_{j}D(T_j^{\ast }{P}_X\parallel {\mu }_j)$ is weak${}^{\ast }$-lower semicontinuous in $P_X$, and hence, the existence of a minimizing $P_X$ is established.

Remark 6.

Abusing the terminology from min-max theory, Theorem 1 may be interpreted as a “strong duality” result, which establishes the equivalence of two optimization problems. The 1)⇒2) part is the non-trivial direction, which requires regularity on the spaces. In contrast, the 2)⇒1) direction can be thought of as a “weak duality”, which establishes only a partial relation, but holds for more general spaces.

3.2. Noncompact $\mathcal{X}$

Our proof of 1)⇒2) in Theorem 1 makes use of the Hahn-Banach theorem and hence relies crucially on the fact that the measure space is the dual of the function space. Naively, one might want to extend the the proof to the case of locally compact $\mathcal{X}$ by considering $C_0(\mathcal{X})$ instead of $C_b(\mathcal{X})$, so that the dual space is still $\mathcal{M}(\mathcal{X})$. However, this would not work: consider the case when $\mathcal{X}={\mathcal{Z}}_1\times ,\dots ,\times {\mathcal{Z}}_l$ and each $S_i$ is the canonical map. Then, ${\mathsf{\Theta }}_{m+1}(u)$ as defined in (43) is $+\infty$ unless $u\equiv 0$ (because $u\in C_0(\mathcal{X})$ requires that u vanishes at infinity); thus, ${\mathsf{\Theta }}_{m+1}^{\ast }\equiv 0$. Luckily, we can still work with $C_b(\mathcal{X})$; in this case, $\ell \in C_b{(\mathcal{X})}^{\ast }$ may not be a measure, but we can decompose it into $\ell =\pi +R$ where $\pi \in \mathcal{M}(\mathcal{X})$ and R is a linear functional “supported at infinity”. Below, we use the techniques in [40] (Chapter 1.3) to prove a particular extension of Theorem 1 to a non-compact case.

Theorem 7.

Theorem 1 still holds if

• The assumption that $\mathcal{X}$ is a compact metric space is relaxed to the assumption that it is a locally compact and σ-compact Polish space;
• $\mathcal{X}={\prod }_{i=1}^l{\mathcal{Z}}_i$ and $S_i:C_b({\mathcal{Z}}_i)\to C_b(\mathcal{X})$, $i=1,\dots ,l$ are canonical maps (see Definition 2).

Proof. 

The proof of the “weak duality” part 2)⇒1) still works in the noncompact case, so we only need to explain what changes need to be made in the proof of the 1)⇒2) part. Let ${\mathsf{\Theta }}_0$ be defined as before, in (36). Then, for any $\ell \in C_b{(\mathcal{X})}^{\ast }$,

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_0^{\ast }(\ell )=\underset{u\le 0}{sup}\ell (u)\end{array}$$

(72)

which is zero if ℓ is nonnegative (in the sense that $\ell (u)\ge 0$ for every $u\ge 0$), and $+\infty$ otherwise. This means that when computing the infimum on the left side of (27), we only need to take into account those nonnegative ℓ.

Next, let ${\mathsf{\Theta }}_{m+1}$ be also defined as before. Then, directly from the definition, we have:

$$\begin{array}{cc}\hfill {\mathsf{\Theta }}_{m+1}^{\ast }(\ell )& =\left\{\begin{array}{cc}0& \mathrm{if}\ell (\sum _i{S}_i{w}_i)=\sum _i{P}_{Z_i}(w_i),\forall w_i\in C_b({\mathcal{Z}}_i),i=1,\dots l;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\hfill \end{array}$$

(73)

For any $\ell \in C_b^{\ast }(\mathcal{X})$. Generally, the condition in the first line of (73) does not imply that ℓ is a measure. However, if ℓ is also nonnegative, then using a technical result in [40] Lemma 1.25, we can further simplify:

$$\begin{array}{cc}\hfill {\mathsf{\Theta }}_{m+1}^{\ast }(\ell )& =\left\{\begin{array}{cc}0& \mathrm{if}\ell \in \mathcal{M}(\mathcal{X})\mathrm{and}{S}_i^{\ast }\ell =P_{Z_i},i=1,\dots ,l;\\ +\infty & \mathrm{otherwise}.\end{array}\right.\hfill \end{array}$$

(74)

This further shows that when we compute the left side of (27), the infimum can be taken over ℓ, which is a coupling of $(P_{Z_i})$. In particular, if ℓ is a probability measure, then ${\mathsf{\Theta }}_j^{\ast }(\ell )=c_{j}D(T_j^{\ast }\ell \parallel {\mu }_j)$ still holds with the ${\mathsf{\Theta }}_j$ defined in (38), $j=1,\dots ,m$. Thus, the rest of the proof can proceed as before. ☐

Remark 7.

The second assumption is made in order to achieve (74) in the proof.

4. Gaussian Optimality

Recall that the conventional Brascamp-Lieb inequality and its reverse ((1) and (2)) state that centered Gaussian functions exhaust such inequalities, and in particular, verifying those inequalities is reduced to a finite dimensional optimization problem (only the covariance matrices in these Gaussian functions are to be optimized). In this section, we show that similar results hold for the forward-reverse Brascamp-Lieb inequality, as well. Our proof uses the rotational invariance argument mentioned in Section 1. Since the forward-reverse Brascamp-Lieb inequality has dual representations (Theorem 7), in principle, the rotational invariance argument can be applied either to the functional representation (as in Lieb’s paper [29]) or the entropic representation (as in Geng-Nair [48]). Here, we adopt the latter approach. We first consider a certain “non-degenerate” case where the existence of an extremizer is guaranteed. Then, Gaussian optimality in the general case follows by a limiting argument (Appendix F), establishing Theorem 2.

4.1. Non-Degenerate Forward Channels

This subsection focuses on the following case:

Assumption 1.

• Fix Lebesgue measures ${({\mu }_j)}_{j=1}^m$ and Gaussian measures ${({\nu }_i)}_{i=1}^l$ on $\mathbb{R}$;
• non-degenerate (Definition 3 below) linear Gaussian random transformation ${(P_{Y_j|\mathbf{X}})}_{j=1}^m$ (where $\mathbf{X}:=(X_1,\dots ,X_l)$) associated with conditional expectation operators ${(T_j)}_{j=1}^m$;
• ${(S_i)}_{i=1}^l$ are induced by coordinate projections;
• positive $(c_j)$ and $(b_i)$.

Definition 3.

We say $(Q_{{\mathbf{Y}}_1|\mathbf{X}},\dots ,Q_{{\mathbf{Y}}_m|\mathbf{X}})$ is non-degenerate if each $Q_{{\mathbf{Y}}_j|\mathbf{X}=\mathbf{0}}$ is an $n_j$-dimensional Gaussian distribution with an invertible covariance matrix.

Given Borel measures $P_{X_i}$ on $\mathbb{R}$, $i=1,\dots ,l$, define:

$$\begin{array}{c}\hfill F_0((P_{X_i})):=\underset{P_{\mathbf{X}}}{inf}\sum _{j=1}^m{c}_{j}D(P_{Y_j}\parallel {\mu }_j)-\sum _{i=1}^l{b}_{i}D(P_{X_i}\parallel {\nu }_i)\end{array}$$

(75)

where the infimum is over Borel measures $P_{\mathbf{X}}$ that have $(P_{X_i})$ as marginals. Note that (75) is well defined since the first term cannot be $+\infty$ under the non-degenerate assumption, and the second term cannot be $-\infty$. The aim of this subsection is to prove the following:

Theorem 8.

${sup}_{(P_{X_i})}{F}_0((P_{X_i}))$, where the supremum is over Borel measures $P_{X_i}$ on $\mathbb{R}$, and $i=1,\dots ,l$, is achieved by some Gaussian ${(P_{X_i})}_{i=1}^l$, in which case the infimum in (75) is achieved by some Gaussian $P_{\mathbf{X}}$.

Naturally, one would expect that Gaussian optimality can be established when ${({\mu }_j)}_{j=1}^m$ and ${({\nu }_i)}_{i=1}^l$ are either Gaussian or Lebesgue. We made the assumption that the former is Lebesgue and the latter is Gaussian so that certain technical conditions can be justified more easily. More precisely, the following observation shows that we can regularize the distributions by a second moment constraint for free:

Proposition 1.

${sup}_{(P_{X_i})}{F}_0((P_{X_i}))$ is finite and there exist ${\sigma }_i^2\in (0,\infty )$, $i=1,\dots ,l$ such that it equals:

$$\begin{array}{c}\hfill \underset{(P_{X_i}):\mathbb{E}\left[X_i^2\right]\le {\sigma }_i^2}{sup}{F}_0((P_{X_i})).\end{array}$$

(76)

Proof. 

When ${\mu }_j$ is Lebesgue and $P_{Y_j|\mathbf{X}}$ is non-degenerate, $D(P_{Y_j}\parallel {\mu }_j)=-h(P_{Y_j})\le -h(P_{Y_j}|\mathbf{X})$ is bounded above (in terms of the variance of the additive noise of $P_{Y_j|\mathbf{X}}$). Moreover, $D(P_{X_i}\parallel {\nu }_i)\ge 0$ when ${\nu }_i$ is Gaussian, so ${sup}_{(P_{X_i})}{F}_0((P_{X_i}))<\infty$. Further, choosing $(P_{X_i})=({\nu }_i)$ and using the covariance matrix to lower bound the first term in (75) show that ${sup}_{(P_{X_i})}{F}_0((P_{X_i}))>-\infty$.

To see (76), notice that:

$$\begin{array}{cc}\hfill D(P_{X_i}\parallel {\nu }_i)& =D(P_{X_i}\parallel {\nu }_i^{\prime })+\mathbb{E}\left[{ı}_{{\nu }_i^{\prime }\parallel {\nu }_i}(X)\right]\hfill \\ \hfill & =D(P_{X_i}\parallel {\nu }_i^{\prime })+D({\nu }_i^{\prime }\parallel {\nu }_i)\hfill \\ \hfill & \ge D({\nu }_i^{\prime }\parallel {\nu }_i)\hfill \end{array}$$

(77)

where ${\nu }_i^{\prime }$ is a Gaussian distribution with the same first and second moments as $X_i\sim P_{X_i}$. Thus, $D(P_{X_i}\parallel {\nu }_i)$ is bounded below by some function of the second moment of $X_i$, which tends to ∞ as the second moment of $X_i$ tends to ∞. Moreover, as argued in the preceding paragraph, the first term in (75) is bounded above by some constant depending only on $(P_{Y_j|\mathbf{X}})$. Thus, we can choose ${\sigma }_i^2>0$, $i=1,\dots ,l$ large enough such that if $\mathbb{E}\left[X_i^2\right]>{\sigma }_i^2$ for some of i, then $F_0((P_{X_i}))<{sup}_{(P_{X_i})}{F}_0((P_{X_i}))$, irrespective of the choices of $P_{X_1},\dots ,P_{X_{i-1}},P_{X_{i+1}},\dots ,P_{X_l}$. Then, these ${\sigma }_1,\dots ,{\sigma }_l$ are as desired in the proposition. ☐

The non-degenerate assumption ensures that the supremum is achieved:

Proposition 2.

Under Assumption 1,

1.

For any ${(P_{X_i})}_{i=1}^l$, the infimum in (75) is attained by some Borel $P_{\mathbf{X}}$.

2.

If ${(P_{Y_j|X^l})}_{j=1}^m$ are non-degenerate (Definition 3), then the supremum in (76) is achieved by some Borel ${(P_{X_i})}_{i=1}^l$.

The proof of Proposition 2 is given in Appendix E. After taking care of the existence of the extremizers, we get into the tensorization properties, which are the crux of the proof:

Lemma 1.

Fix $(P_{X_i^{(1)}})$, $(P_{X_i^{(2)}})$, $({\mu }_j)$, $(T_j)$, $(c_j)\in {[0,\infty )}^m$, and let $S_j$ be induced by coordinate projections. Then:

$$\begin{array}{c}\hfill \underset{P_{{\mathbf{X}}^{(1,2)}}:S_i^{\ast \otimes 2}{P}_{{\mathbf{X}}^{(1,2)}}=P_{X_i^{(1)}}\times P_{X_i^{(2)}}}{inf}\sum _{j=1}^m{c}_{j}D(P_{Y_j^{(1,2)}}\parallel {\mu }_j^{\otimes 2})=\sum _{t=1,2}\sum _{j=1}^m{c}_j\underset{P_{{\mathbf{X}}^{(t)}}:S_i^{\ast }{P}_{{\mathbf{X}}^{(t)}}=P_{X_i^{(t)}}}{inf}D(P_{Y_j^{(t)}}\parallel {\mu }_j)\end{array}$$

(78)

where for each j,

$$\begin{array}{c}\hfill P_{Y_j^{(1,2)}}:=T_j^{\ast \otimes 2}{P}_{{\mathbf{X}}^{(1,2)}}\end{array}$$

(79)

on the left side and:

$$\begin{array}{c}\hfill P_{Y_j^{(t)}}:=T_j^{\ast \otimes 2}{P}_{{\mathbf{X}}^{(t)}}\end{array}$$

(80)

on the right side, $t=1,2$.

Proof. 

We only need to prove the nontrivial ≥ part. For any $P_{{\mathbf{X}}^{(1,2)}}$ on the left side, choose $P_{{\mathbf{X}}^{(t)}}$ on the right side by marginalization. Then:

$$\begin{array}{cc}\hfill D(P_{Y_j^{(1,2)}}\parallel {\mu }_j^{\otimes 2})-\sum _{t}D(P_{Y_j^{(t)}}\parallel {\mu }_j)& =I(Y_j^{(1)};Y_j^{(2)})\hfill \\ \hfill & \ge 0\hfill \end{array}$$

(81)

for each j. ☐

We are now ready to show the main result of this section.

Proof of Theorem 8. 

• Assume that $(P_{X_i^{(1)}})$ and $(P_{X_i^{(2)}})$ are maximizers of $F_0$ (possibly equal). Let $P_{X_i^{1,2}}:=P_{X_i^{(1)}}\times P_{X_i^{(2)}}$. Define:

  $$\begin{array}{cc}\hfill {\mathbf{X}}^{+}& :=\frac{1}{\sqrt{2}}\left({\mathbf{X}}^{(1)}+{\mathbf{X}}^{(2)}\right);\hfill \end{array}$$

  (82)

  $$\begin{array}{cc}\hfill {\mathbf{X}}^{-}& :=\frac{1}{\sqrt{2}}\left({\mathbf{X}}^{(1)}-{\mathbf{X}}^{(2)}\right).\hfill \end{array}$$

  (83)

  Define $(Y_j^{+})$ and $(Y_j^{-})$ analogously. Then, $Y_j^{+}|\{{\mathbf{X}}^{+}={\mathbf{x}}^{+},{\mathbf{X}}^{-}={\mathbf{x}}^{-}\}\sim Q_{Y_j|\mathbf{X}={\mathbf{x}}^{+}}$ is independent of ${\mathbf{x}}^{-}$, and $Y_j^{-}|\{{\mathbf{X}}^{+}={\mathbf{x}}^{+},{\mathbf{X}}^{-}={\mathbf{x}}^{-}\}\sim Q_{Y_j|\mathbf{X}={\mathbf{x}}^{-}}$ is independent of ${\mathbf{x}}^{+}$.
• Next, we perform the same algebraic expansion as in the proof of tensorization:

  $$\begin{array}{cc}\hfill \sum _{t=1}^2{F}_0\left({\left(P_{X_i^{(t)}}\right)}_{i=1}^l\right)& =\underset{P_{{\mathbf{X}}^{(1,2)}}:S_j^{\ast \otimes 2}{P}_{{\mathbf{X}}^{(1,2)}}=P_{X_j^{(1,2)}}}{inf}\sum _j{c}_{j}D(P_{Y_j^{(1,2)}}\parallel {\mu }_j^{\otimes 2})-\sum _i{b}_{i}D(P_{X_i^{(1,2)}}\parallel {\nu }_i^{\otimes 2})\hfill \end{array}$$

  (84)

  $$\begin{array}{c}=\underset{P_{{\mathbf{X}}^{+}{\mathbf{X}}^{-}}:S_j^{\ast \otimes 2}{P}_{{\mathbf{X}}^{+}{\mathbf{X}}^{-}}=P_{X_j^{+}{X}_j^{-}}}{inf}\sum _j{c}_{j}D(P_{Y_j^{+}{Y}_j^{-}}\parallel {\mu }_j^{\otimes 2})-\sum _i{b}_{i}D(P_{X_i^{+}{X}_i^{-}}\parallel {\nu }_i^{\otimes 2})\hfill \\ \le \underset{P_{{\mathbf{X}}^{+}{\mathbf{X}}^{-}}:S_j^{\ast \otimes 2}{P}_{{\mathbf{X}}^{+}{\mathbf{X}}^{-}}=P_{X_j^{+}{X}_j^{-}}}{inf}\sum _j{c}_j\left[D(P_{Y_j^{+}}\parallel {\mu }_j)+D(P_{Y_j^{-}|{\mathbf{X}}^{+}}\parallel {\mu }_j|P_{{\mathbf{X}}^{+}})\right]\hfill \end{array}$$

  (85)

  $$\begin{array}{c}-\sum _i{b}_i\left[D(P_{X_i^{+}}\parallel {\nu }_i)+D(P_{X_i^{-}|X_i^{+}}\parallel {\nu }_i|P_{X_i^{+}})\right]\hfill \end{array}$$

  (86)

  $$\begin{array}{c}\le \sum _j{c}_j\left[D(P_{Y_j^{+}}^{\star }\parallel {\mu }_j)+D(P_{Y_j^{-}|{\mathbf{X}}^{+}}^{\star }\parallel {\mu }_j|P_{{\mathbf{X}}^{+}}^{\star })\right]\hfill \\ -\sum _i{b}_i\left[D(P_{X_i^{+}}^{\star }\parallel {\nu }_i)+D(P_{X_i^{-}|{\mathbf{X}}^{+}}^{\star }\parallel {\nu }_i|P_{{\mathbf{X}}^{+}}^{\star })\right]\hfill \end{array}$$

  (87)

  $$\begin{array}{c}=F_0\left({\left(P_{X_i^{+}}^{\star }\right)}_{i=1}^l\right)+\int F_0\left({\left(P_{X_i^{-}|{\mathbf{X}}^{+}}^{\star }\right)}_{i=1}^l\right)\mathrm{d}{P}_{{\mathbf{X}}^{+}}^{\star }\hfill \end{array}$$

  (88)

  $$\begin{array}{c}\le \sum _{t=1}^2{F}_0\left({\left(P_{X_i^{(t)}}\right)}_{i=1}^l\right)\hfill \end{array}$$

  (89)

  where:
  • (84) uses Lemma 1.
  • (86) is because of the Markov chain $Y_j^{+}-{\mathbf{X}}^{+}-Y_j^{-}$ (for any coupling).
  • In (87), we selected a particular instance of coupling $P_{{\mathbf{X}}^{+}{\mathbf{X}}^{-}}$, constructed as follows: first, we select an optimal coupling $P_{{\mathbf{X}}^{+}}$ for given marginals $(P_{X_i^{+}})$. Then, for any ${\mathbf{x}}^{+}={(x_i^{+})}_{i=1}^l$, let $P_{{\mathbf{X}}^{-}|{\mathbf{X}}^{+}=x^{+}}$ be an optimal coupling of $(P_{X_i^{-}|X_i^{+}=x_i^{+}})$ (for a justification that we can select optimal coupling $P_{{\mathbf{X}}^{-}|{\mathbf{X}}^{+}={\mathbf{x}}^{+}}$ in a way that $P_{{\mathbf{X}}^{-}|{\mathbf{X}}^{+}}$ is indeed a regular conditional probability distribution, see [7]). With this construction, it is apparent that $X_i^{+}-{\mathbf{X}}^{+}-X_i^{-}$, and hence:

    $$\begin{array}{c}\hfill D(P_{X_i^{-}|X_i^{+}}\parallel {\nu }_i|P_{X_i^{+}})=D(P_{X_i^{-}|{\mathbf{X}}^{+}}\parallel {\nu }_i|P_{{\mathbf{X}}^{+}}).\end{array}$$

    (90)
  • (88) is because in the above, we have constructed the coupling optimally.
  • (89) is because $(P_{X_i}^{(t)})$ maximizes $F_0$, $t=1,2$.
• Thus, in the expansions above, equalities are attained throughout. Using the differentiation technique as in the case of forward inequality, for almost all $(b_i)$, $(c_j)$, we have:

  $$\begin{array}{cc}\hfill D(P_{X_i^{-}|X_i^{+}}\parallel {\nu }_i|P_{X_i^{+}})& =D(P_{X_i^{+}}\parallel {\nu }_i)\hfill \end{array}$$

  (91)

  $$\begin{array}{c}=D(P_{X_i^{-}}\parallel {\nu }_i),\forall i\hfill \end{array}$$

  (92)

  where (92) is because by symmetry, we can perform the algebraic expansions in a different way to show that $(P_{X_i^{-}})$ is also a maximizer of $F_0$. Then, $I(X_i^{+};X_i^{-})=D(P_{X_i^{-}|X_i^{+}}\parallel {\nu }_i|P_{X_i^{+}})-D(P_{X_i^{-}}\parallel {\nu }_i)=0$, which, combined with $I(X_i^{(1)};X_i^{(2)})$, shows that $X_i^{(1)}$ and $X_i^{(2)}$ are Gaussian with the same covariance. Lastly, using Lemma 1 and the doubling trick, one can show that the optimal coupling is also Gaussian.
  ☐

4.2. Analysis of Example 1 Using Gaussian Optimality

We note that Example 1 is a rather simple setting, where (17) can be proven by integrating the two sides of (18) and applying the change of variables, noting that the absolute value of the Jacobian equals one. Nevertheless, it is illuminating to give an alternative proof using the Gaussian optimality result, as a proof of concept. In this section, we only give a proof sketch where certain “technicalities” are not justified. Details of the justifications are deferred to Appendix F.

Proof sketch for the claim in Example 1. 

By duality (Theorem 7), it suffices to prove the corresponding entropic inequality. The Gaussian optimality result in Theorem 8 assumed Gaussian reference measures on the output and non-degenerate forward channels in order to simplify the proof of the existence of minimizers; however, supposing that Gaussian optimality extends beyond those technical conditions, we see that it suffices to prove that for any centered Gaussian $(P_{X_i})$,

$$\begin{array}{c}\hfill \sum _{i=1}^{l}h(P_{X_i})\le \underset{P_{X^l}}{sup}\sum _{j=1}^{l}h(P_{Y_j})\end{array}$$

(93)

where the supremum is over Gaussian $P_{X^l}$ with the marginals $P_{X_1},\dots ,P_{X_l}$ and $Y_j:={\sum }_{i=1}^l{m}_{ji}{X}_i$. Let $a_i:=\mathbb{E}\left[X_i^2\right]$, and choose $P_{X^l}={\prod }_{i=1}^l{P}_{X_i}$; we see that (93) holds if:

$$\begin{array}{c}\hfill \sum _{i=1}^{l}log{a}_i\le \sum _{j=1}^{l}log\left(\sum _{i=1}^l{m}_{ji}^2{a}_i\right),\forall a_i>0,i=1,\dots ,l,\end{array}$$

(94)

where $(a_i)$ are the eigenvalues and ${\left({\sum }_{i=1}^l{m}_{ji}{a}_i\right)}_{i=1}^l$ are the diagonal entries of the matrix:

$$\begin{array}{c}\hfill \mathbf{M}\mathrm{diag}{(a_i)}_{1\le i\le l}{\mathbf{M}}^{\top }.\end{array}$$

(95)

Therefore, (94) holds. ☐

A generalization of Example 1 is as follows.

Proposition 3.

For any orthogonal matrix $\mathbf{M}:={(m_{ji})}_{1\le j\le l,1\le i\le l}$ with nonzero entries, we claim that there exists a neighborhood $\mathcal{U}$ of the uniform probability vector $(\frac{1}{l},\dots ,\frac{1}{l})$, such that for any $(b_1,\dots ,b_l)$ and $(c_1,\dots ,c_l)$ in $\mathcal{U}$, the best constant D in the FR-BLinequality (16) equals $exp(H(c^l)-H(b^l))$ where $H(·)$ is the entropy functional.

The proposition generalizes the claim in Example 1. Indeed, observe that there is no loss of generality in assuming that $(b_1,\dots ,b_l)$ and $(c_1,\dots ,c_l)$ are probability vectors, since by dimensional analysis, we see that the best constant is infinite unless ${\sum }_{i=1}^l{b}_i={\sum }_{j=1}^l{c}_j$; and it is also clear that the best constant is invariant when each $b_i$ and $c_j$ is multiplied by the same positive number. Moreover, any orthogonal matrix can be approximated by a sequence of orthogonal $\mathbf{M}$ with nonzero entries, for which the neighborhood $\mathcal{U}$ shrinks, but always contains the uniform probability vector $(\frac{1}{l},\dots ,\frac{1}{l})$.

Proof sketch for Proposition 3. 

Note that along the same lines as (94), the best constant in the FR-BL inequality equals:

$$\begin{array}{c}\hfill D=\underset{a^l\in \Delta }{sup}\frac{{\prod }_{i=1}^l{a}_i^{b_i}}{{sup}_{\mathbf{A}⪰\mathbf{0}:{\mathbf{A}}_{ii}=a_i}{\prod }_{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}^{c_j}}\end{array}$$

(96)

where without loss of generality, we assumed $a^l\in \Delta$ is in the probability simplex. We first observe that if the positive semidefinite constraint $\mathbf{A}⪰\mathbf{0}$ in (96) were nonexistent, then the sup in the denominator in (96) would equal ${\prod }_{j=1}^l{c}_j^{c_j}$, and consequently, (96) would equal $exp(H(c^l)-H(b^l))$, for any $b^l,c^l\in \Delta$ not necessarily close to the uniform probability vector. Indeed, fixing ${\mathbf{A}}_{ii}=a_i,i=1,\dots ,l$, the linear map from the off-diagonal entries to the diagonal entries of ${\mathbf{MAM}}^{\top }$ is onto the space of l-vectors whose entries sum to one; proof of the surjectivity can be reduced to checking the fact that the only diagonal matrix that commutes with $\mathbf{M}$ is a multiple of the identity matrix. Then, the sup in the denominator is achieved when ${\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}=c_j,j=1\dots l$, which is independent of $a^l$.

Next, we argue that the constraint $\mathbf{A}⪰\mathbf{0}$ in (96) is not active when $b^l$ and $c^l$ are close to the uniform vector. Denote by $\mathcal{U}(t)$ the set of l-vectors whose distance (say in total variation) to the uniform vector $(\frac{1}{l},\dots ,\frac{1}{l})$ is at most t. Observe that:

• There exists $t>0$ such that for every $a^l\in \mathcal{U}(t)$,

  $$\begin{array}{c}\hfill \underset{\mathbf{A}⪰\mathbf{0}:{\mathbf{A}}_{ii}=a_i}{sup}\prod _{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}=1/l^l\end{array}$$

  (97)

  which follows by continuity and the fact that when $a^l$ is uniform, the sup (97) is achieved at the strictly positive definite $\mathbf{A}=l^{-1}\mathbf{I}$.
• When $b^l=c^l=(\frac{1}{l},\dots ,\frac{1}{l})$ is the uniform probability vector, (96) equals one, which is uniquely achieved by $a^l=(\frac{1}{l},\dots ,\frac{1}{l})$. To see the uniqueness, take $\mathbf{A}$ to be diagonal in the denominator and observe that the denominator is strictly bigger than the numerator when the diagonals of $\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }$ are not a permutation of $a^l$. Then, since the extreme value of a continuous functions is achieved on a compact set, we can find $ϵ>0$ such that:

  $$\begin{array}{c}\hfill \frac{{\prod }_{i=1}^l{a}_i^{1/l}}{{sup}_{\mathbf{A}⪰\mathbf{0}:{\mathbf{A}}_{ii}=a_i}{\prod }_{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}^{1/l}}<1-ϵ\end{array}$$

  (98)

  for any $a^l\notin \mathcal{U}(t/2)$.
• Finally, by continuity, we can choose $s\in (0,t/2)$ small enough such that for any $b^l,c^l\in \mathcal{U}(s)$,

  $$\begin{array}{cc}\hfill \frac{{\prod }_{i=1}^l{a}_i^{b_i}}{{sup}_{\mathbf{A}⪰\mathbf{0}:{\mathbf{A}}_{ii}=a_i}{\prod }_{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}^{c_j}}& <1-ϵ/2,\forall a^l\notin \mathcal{U}(t/2);\hfill \end{array}$$

  (99)

  $$\begin{array}{cc}\hfill \underset{\mathbf{A}⪰\mathbf{0}:{\mathbf{A}}_{ii}=a_i}{sup}\prod _{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}^{c_j}& =\underset{\mathbf{A}:{\mathbf{A}}_{ii}=a_i}{sup}\prod _{j=1}^l{\left[\mathbf{M}\mathbf{A}{\mathbf{M}}^{\top }\right]}_{jj}^{c_j},\forall a^l\in \mathcal{U}(t/2);\hfill \end{array}$$

  (100)

  $$\begin{array}{cc}\hfill exp(H(c^l)-H(b^l))& >1-ϵ/2.\hfill \end{array}$$

  (101)

Taking the neighborhood $\mathcal{U}(s)$ proves the claim. ☐

5. Relation to Hypercontractivity and Its Reverses

As alluded to before and illustrated by Figure 2, the forward-reverse Brascamp-Lieb inequality generalizes several other inequalities from functional analysis and information theory; a more complete discussion on these relationships can be found in [7]. In this section, we focus on hypercontractivity and show how its three cases all follow from Theorem 1. Among these, the case in Section 5.3 can be regarded as an instance of the forward-reverse inequality that cannot be reduced to either the forward or the reverse inequality alone. It is also interesting to note that, from the viewpoint of the forward-reverse Brascamp-Lieb inequality, in each of the three special cases, there ought to be three functions involved in the functional formulation; however, the optimal choice of one function can be computed from the other two. Therefore, the conventional functional formulations of the three cases of hypercontractivity involve only two functions, making it non-obvious to find a unifying inequality.

5.1. Hypercontractivity

Fix a joint probability distribution $Q_{Y_1{Y}_2}$ and nonnegative continuous functions $F_1$ and $F_2$ on ${\mathcal{Y}}_1$ and ${\mathcal{Y}}_2$, respectively, both bounded away from zero. In Theorem 1, take $l\leftarrow 1$, $m\leftarrow 2$, $b_1\leftarrow 1$, $d\leftarrow 0$, $f_1\leftarrow F_1^{\frac{1}{c_1}}$, $f_2\leftarrow F_2^{\frac{1}{c_2}}$, ${\nu }_1\leftarrow Q_{Y_1{Y}_2}$, ${\mu }_1\leftarrow Q_{Y_1}$, ${\mu }_2\leftarrow Q_{Y_2}$. Furthermore, put $Z_1=X=(Y_1,Y_2)$, and let $T_1$ and $T_2$ be the canonical maps (Definition 2). The measure spaces and the random transformations are as shown in Figure 3.

The constraint (12) translates to:

$$\begin{array}{c}\hfill g_1(y_1,y_2)\le F_1(y_1)F_2(y_2),\forall y_1,y_2\end{array}$$

(102)

and the optimal choice of $g_1$ is when the equality is achieved. We thus obtain the equivalence between:

$$\begin{array}{c}\hfill \parallel F_1{\parallel }_{\frac{1}{c_1}}{\parallel F_2\parallel }_{\frac{1}{c_2}}\ge \mathbb{E}\left[F_1(Y_1)F_2(Y_2)\right],\forall F_1\in L^{\frac{1}{c_1}}(Q_{Y_1}),F_2\in L^{\frac{1}{c_2}}(Q_{Y_2})\end{array}$$

(103)

and:

$$\begin{array}{c}\hfill \forall P_{Y_1{Y}_2},D(P_{Y_1{Y}_2}\parallel Q_{Y_1{Y}_2})\ge c_{1}D(P_{Y_1}\parallel Q_{Y_1})+c_{2}D(P_{Y_2}\parallel Q_{Y_2}).\end{array}$$

(104)

By a standard dense-subspace argument, we see that it is inconsequential that $F_1$ and $F_2$ in (103) are not assumed to be continuous, nor bounded away from zero. It is also easy to see that the nonnegativity of $F_1$ and $F_2$ is inconsequential for (103).

This equivalence can also be obtained from Theorem 1. By Hölder’s inequality, (103) is equivalent to saying that the norm of the linear operator sending $F_1\in L^{\frac{1}{c_1}}(Q_{Y_1})$ to $\mathbb{E}\left[F_1(Y_1)\right|Y_2=·]\in L^{\frac{1}{1-c_2}}(Q_{Y_2})$ does not exceed one. The interesting case is $\frac{1}{1-c_2}>\frac{1}{c_1}$, hence the name hypercontractivity. The equivalent formulation of hypercontractivity was shown in [44] using a different proof via the method of types/typicality, which requires that $|{\mathcal{Y}}_1|,|{\mathcal{Y}}_2|<\infty$. In contrast, the proof based on the nonnegativity of relative entropy removes this constraint, allowing one to prove Nelson’s Gaussian hypercontractivity from the information-theoretic formulation (see [7]).

5.2. Reverse Hypercontractivity (Positive Parameters)

By “positive parameters” we mean the $b_1$ and $b_2$ in (107) are positive.

Let $Q_{Z_1{Z}_2}$ be a given joint probability distribution, and let $G_1$ and $G_2$ be nonnegative functions on ${\mathcal{Z}}_1$ and ${\mathcal{Z}}_2$, respectively, both bounded away from zero. In Theorem 1, take $l\leftarrow 2$, $m\leftarrow 1$, $c_1\leftarrow 1$, $d\leftarrow 0$, $g_1\leftarrow G_1^{\frac{1}{b_1}}$, $g_2\leftarrow G_2^{\frac{1}{b_2}}$, ${\mu }_1\leftarrow Q_{Z_1{Z}_2}$, ${\nu }_1\leftarrow Q_{Z_1}$, ${\nu }_2\leftarrow Q_{Z_2}$. Furthermore, put $Y_1=X=(Z_1,Z_2)$, and let $S_1$ and $S_2$ be the canonical maps (Definition 2). The measure spaces and the random transformations are as shown in Figure 4.

Note that the constraint (12) translates to:

$$\begin{array}{c}\hfill f_1(z_1,z_2)\ge G_1(z_1)G_2(z_2),\forall z_1,z_2.\end{array}$$

(105)

and the equality case yields the optimal choice of $f_1$ for (13). By Theorem 1, we thus obtain the equivalence between:

$$\begin{array}{c}\hfill \parallel G_1{\parallel }_{\frac{1}{b_1}}{\parallel G_2\parallel }_{\frac{1}{b_2}}\le \mathbb{E}\left[G_1(Z_1)G_2(Z_2)\right],\forall G_1,G_2\end{array}$$

(106)

and:

$$\begin{array}{c}\hfill \forall P_{Z_1},P_{Z_2},\exists P_{Z_1{Z}_2},D(P_{Z_1{Z}_2}\parallel Q_{Z_1{Z}_2})\le b_{1}D(P_{Z_1}\parallel Q_{Z_1})+b_{2}D(P_{Z_2}\parallel Q_{Z_2}).\end{array}$$

(107)

Note that in this setup, if ${\mathcal{Z}}_1$ and ${\mathcal{Z}}_2$ are finite, then Condition (iv) in Theorem 1 is equivalent to $Q_{Z_1{Z}_2}\ll Q_{Z_1}\times Q_{Z_2}$. The equivalent formulations of reverse hypercontractivity were observed in [59], where the proof is based on the method of types.

5.3. Reverse Hypercontractivity (One Negative Parameter)

By “one negative parameter” we mean the $b_1$ is positive and $-c_2$ is negative in (111).

In Theorem 1, take $l\leftarrow 1$, $m\leftarrow 2$, $c_1\leftarrow 1$, $d\leftarrow 0$. Let $Y_1=X=(Z_1,Y_2)$, and let $S_1$ and $T_2$ be the canonical maps (Definition 2). Suppose that $Q_{Z_1{Y}_2}$ is a given joint probability distribution, and set ${\mu }_1\leftarrow Q_{Z_1{Y}_2}$, ${\nu }_1\leftarrow Q_{Z_1}$, ${\mu }_2\leftarrow Q_{Y_2}$ in Theorem 1. Suppose that F and G are arbitrary nonnegative continuous functions on ${\mathcal{Y}}_2$ and ${\mathcal{Z}}_1$, respectively, which are bounded away from zero. Take $g_1\leftarrow G^{\frac{1}{b_1}}$, $f_2\leftarrow F^{-\frac{1}{c_2}}$. in Theorem 1. The measure spaces and the random transformations are as shown in Figure 5.

The constraint (12) translates to:

$$\begin{array}{c}\hfill f_1(z_1,y_2)\ge G(z_1)F(y_2),\forall z_1,y_2.\end{array}$$

(108)

Note that (13) translates to:

$$\begin{array}{c}\hfill {\parallel G\parallel }_{\frac{1}{b_1}}\le Q_{Y_2{Z}_1}(f_1)Q_{Y_2}^{c_2}(F^{-\frac{1}{c_2}})\end{array}$$

(109)

for all F, G and $f_1$ satisfying (108). It suffices to verify (109) for the optimal choice $f_1=GF$, so (109) is reduced to:

$$\begin{array}{c}\hfill {\parallel F\parallel }_{\frac{1}{-c_2}}{\parallel G\parallel }_{\frac{1}{b_1}}\le \mathbb{E}\left[F(Y_2)G(Z_1)\right],\forall F,G.\end{array}$$

(110)

By Theorem 1, (110) is equivalent to:

$$\begin{array}{c}\hfill \forall P_{Z_1},\exists P_{Z_1{Y}_2},D(P_{Z_1{Y}_2}\parallel Q_{Z_1{Y}_2})\le b_{1}D(P_{Z_1}\parallel Q_{Z_1})+(-c_2)D(P_{Y_2}\parallel Q_{Y_2}).\end{array}$$

(111)

Inequality (110) is called reverse hypercontractivity with a negative parameter in [45], where the entropic version (111) is established for $|{\mathcal{Z}}_1|,|{\mathcal{Y}}_2|<\infty$ using the method of types. Multiterminal extensions of (110) and (111) (called the reverse Brascamp-Lieb type inequality with negative parameters in [45]) can also be recovered from Theorem 1 in the same fashion, i.e., we move all negative parameters to the other side of the inequality so that all parameters become positive.

In summary, from the viewpoint of Theorem 1, the results in Section 5.1, Section 5.2 and Section 5.3 are degenerate special cases, in the sense that in any of the three cases, the optimal choice of one of the functions in (13) can be explicitly expressed in terms of the other functions; hence, this “hidden function” disappears in (103), (106) or (110).