Elastic Information Bottleneck

Abstract

Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two different methods have been proposed, i.e., information bottleneck (IB) and deterministic information bottleneck (DIB), and have gained significant progress in explaining the representation mechanisms of deep learning algorithms. However, these theoretical and empirical successes are only valid with the assumption that training and test data are drawn from the same distribution, which is clearly not satisfied in many real-world applications. In this paper, we study their generalization abilities within a transfer learning scenario, where the target error could be decomposed into three components, i.e., source empirical error, source generalization gap (SG), and representation discrepancy (RD). Comparing IB and DIB on these terms, we prove that DIB’s SG bound is tighter than IB’s while DIB’s RD is larger than IB’s. Therefore, it is difficult to tell which one is better. To balance the trade-off between SG and the RD, we propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers, which guarantees a Pareto frontier within the IB framework. Additionally, simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB, which validates the correctness of our theories.

1. Introduction

Representation learning has recently become a core problem in machine learning, especially with the development of deep learning methods. Different from other statistical representation learning approaches, the information bottleneck principle formalizes the extraction of relevant features about Y from X as an information-theoretic optimization problem: ${min}_{p\left(t\right|x)}L={min}_{p\left(t\right|x)}f(X;T)-\beta g(Y;T)$, where $p\left(t\right|x)$ amounts to the encoder of the input signal X, f stands for the compression of representation T with respect to input X, g stands for the preserved information of T with respect to output Y, $Y\leftrightarrow X\leftrightarrow T$ forms a Markov chain, and $\beta$ is the tradeoff parameter. The basic idea of the information bottleneck principle is to obtain the information that X provides about Y through a ‘bottleneck’ representation T. The Markov constraint requires that T is a (possibly stochastic) function of X and can only obtain information about Y through X.

Under this principle, various methods have been proposed, such as information bottleneck (IB) [1], conditional entropy bottleneck (CEB) [2], Gaussian IB [3], multivariate IB [4], distributed IB [5], squared IB [6], deterministic information bottleneck (DIB) [7], etc. Almost all previous methods use the mutual information $I(Y;T)$ as the information preserving function g. As for the compression function f, there are two typical functions, which categorize these methods into two groups. The first group uses the mutual information $I(X;T)$, a common measure of representation cost in channel coding, as the compression function. Typical examples include IB, CEB, Gaussian IB, multivariate IB, squared IB, etc. Please note that CEB uses the conditional mutual information $I(X;T|Y)$ as the compression function, however, it has been proven to be equivalent to mutual information. Similarly, squared IB uses the square of the mutual information $I^2(X;T)$ as the compression function; still, we put it into the same category. Instead of the mutual information, the second group, including DIB, uses entropy $H\left(T\right)$ as the compression function, which is another common measure to quantify the representation cost as in source coding. The reason for this is that entropy is directly related to the quantity to be constrained such as the number of clusters, which is expected to achieve better compression results.

IB has been extensively studied over recent years in theory and application. In theory, IB has been proven to be able to enhance generalization [8] and adversarial robustness [9] by providing a generalization bound and an adversarial robustness bound. In application, IB has been successfully applied in evaluating the representations learned by deep neural networks (DNN) [10,11,12] and completing various tasks such as geometric clustering by iteratively solving a set of self-consistent equations to obtain the optimal solution of IB optimization problem [13,14], classification [2,15], and generation [16] by serving as the loss function of DNN via variational methods. Recent research also shows that if we merge the loss function of IB with other models, such as BERT and Graph Neural Network, better generalization and adversarial robustness results will be obtained [9,17]. Moreover, IB inspires new methods that improve generalization [18,19,20]. Likewise, DIB has been applied in geometric clustering [21] and has the potential to be used in similar applications to IB. More wide-ranging work on IB in deep learning and communication is comprehensively summarized in the surveys found in [22,23,24].

Still, the theoretical results are only valid when the training and test data are drawn from the same distribution, but this is rarely the case in practice. Therefore, it is unclear, especially theoretically, whether IB and DIB are able to learn a representation from a source domain that performs well on the target domain. Moreover, it is worth studying which objective function is better. This is exactly the motivation of this paper. To this end, we formulate the problem as transfer learning, and the target domain test error could be decomposed into three parts: the source domain training error, the source domain generalization gap (SG), and the representation discrepancy (RD), based on the transfer learning theory [25]. Without loss of generality, we assume that both IB and DIB have the ability to achieve small source domain training errors, so our goals become calculating SG and RD and comparing the two methods on these terms.

For SG, Shamir et al. [8] have provided an upper bound related to $I(X;T)$, indicating that minimizing $I(X;T)$ leads to better generalization. However, this theory is not applicable for comparing IB and DIB because DIB minimizes $H\left(T\right)=I(X;T)+H\left(T\right|X)$, and it is unclear whether further minimizing $H\left(T\right|X)$ may bring some advantages. Therefore, we need to derive a new bound. The difficulty lies in that the new bound needs to not only include both $I(X,T)$ and $H\left(T\right|X)$ for convenient comparison but also be tighter than the previous one. Since the previous bound in Shamir et al. [8] was represented as a $\varphi$ function of the variance, we tackle this problem by introducing a different analysis of these two factors. Specifically for the variance, different from relating the variance to the $L_1$ distance; then, to KL-divergence; and lastly, to mutual information in the previous proof, we bound the variances by functions of expectations and successfully relate them to entropy $H\left(T\right)$. Furthermore, we prove a tighter bound for the $\varphi$ function. Consequently, we prove a tighter generalization bound, suggesting that minimizing $H\left(T\right)$ is better than $I(X;T)$. Therefore, our results indicate that DIB may generalize better than IB in the source domain.

As for RD, it is measured by the $\mathcal{H}\Delta \mathcal{H}$-distance, as in Ben-David et al. [26]. However, this term is difficult to compute because $\mathcal{H}$ is the hypothesis space of classifiers and is diverse for different models. Inspired by the fact that IB and DIB solutions are mainly different with the variance in the representations, we assume that data are generated with a Gaussian distribution. Therefore, we define a pair-wise $L_1$ distance to bound the $\mathcal{H}\Delta \mathcal{H}$-distance and relate RD with the variance in representations. Specifically, IB’s representations have larger randomness and thus have smaller RDs. Moreover, the closer the two domains are, the greater the difference between IB and DIB on RD is.

From the above theoretical findings, we conclude that there exists a better objective function under the IB principle. However, how to obtain the optimal objective function remains a challenging problem. Inspired by the trade-off between SG and RD, we propose an elastic information bottleneck (EIB) to interpolate between IB and DIB, i.e., $min{L}_{EIB}=(1-\alpha )H\left(T\right)+\alpha I(X;T)-\beta I(Y;T)$. We can see that EIB includes IB and DIB as special cases. In addition, we provide a variational method to optimize EIB by DNN. We conduct both simulations and real data experiments in this paper. Our results show that EIB is more flexible to different kinds of data and achieves better accuracy than IB and DIB on classification tasks. We also provide an example of combining EIB with a previous domain adaptation algorithm by substituting the cross entropy loss with the EIB objective function, which suggests a promising application of EIB. Our contributions are summarized as follows:

• We derive a transfer learning theory for IB frameworks and find a trade-off between SG and RD. Consequently, We propose a novel representation learning method EIB for better transfer learning results under the IB principle, which is flexible to suit different kinds of data and can be merged into domain adaptation algorithms as a regularizer.
• In the study of SG, we provide a tighter SG upper bound, which serves as a theoretical guarantee for DIB and further develop the generalization theory in the IB framework.
• Comprehensive simulations and experiments validate our theoretic results and demonstrate that EIB outperforms IB and DIB in many cases.

2. Problem Formulation

In domain adaptation, a scenario of transductive transfer learning, we have labeled training data from a source domain, and we wish to learn an algorithm on the training data that performs well on a target domain [27]. The learning task is the same on the two domains, but the population distribution of the source domain and the target domain are different. Specifically, the source and the target instances come from the same instance space $\mathcal{X}$, and the same instance corresponds to the same label, even if they lie in different domains. However, the population instance distribution on the source and the target domain are different, denoted as $p\left(X\right)$ and $q\left(X\right)$. (We use “instance” (X) to distinguish from “label” (Y) and “example” (X,Y) and use “population distribution” (p or q) to distinguish from “empirical distribution” ($\widehat{p}$ or $\widehat{q}$)). Assume that there exists a generic feature space so it can be utilized to transfer knowledge between different domains. That is to say, both the IB and DIB methods involve an encoder with a transition probability $p\left(t\right|x)$ to convert instance $\mathcal{X}$ to an intermediate feature $T\in \mathcal{T}$. The induced marginal distribution is then denoted as $p\left(t\right)$ and $q\left(t\right)$ for the source and target domains, respectively. The IB and DIB methods also have a decoder or classifier $h\in \mathcal{H}$ to map from the feature space $\mathcal{T}$ to the label space $\mathcal{Y}$. $p\left(y\right|x)$ denotes the ground-truth labeling function and $p\left(y\right|t)={\sum }_{x}p\left(y\right|x)\frac{p\left(t\right|x\left)p\right(x)}{{\sum }_{x^{\prime }}p\left(t\right|x^{\prime })p\left(x^{\prime }\right)}$ is a labeling function induced by $p\left(y\right|x)$ and $p\left(t\right|x)$. In classification scenarios, deterministic labels are commonly used. Therefore, we define the deterministic ground-truth-induced labeling function as $f:\mathcal{T}\to \mathcal{Y},f\left(t\right)=argma{x}_{y}p\left(y\right|t)$. If the maximum probability is not unique, i.e., $\{y_1,\dots ,y_n\}=argma{x}_{y}p\left(y\right|t)$, randomly choose a $y_{i,i\in \{1\dots n\}}$ as the output. With the above definitions, the expected error on the source domain could be written as ${ϵ}_S\left(h\right)=E_{t\sim p\left(t\right)}{I}_{\left\{f\right(t)\ne h(t\left)\right\}}$, where $I_{\{·\}}$ is the indicator function. Similarly, the expected error on target domain is ${ϵ}_T\left(h\right)=E_{t\sim q\left(t\right)}{I}_{\left\{f\right(t)\ne h(t\left)\right\}}$. Then, our problem could be formulated as follows: When the IB and DIB methods are trained on the source domain, which method achieves a lower target domain expected error ${ϵ}_T\left(h\right)$?

According to previous work [26], the target domain error could be decomposed to three parts, as shown in the following theorem. A detailed proof is provided in Appendix A.1.

Theorem 1

(Target Error Decomposition). Suppose that $h^{*}$ is the classifier in $\mathcal{H}$, which minimizes the sum of the expected error in two domains, i.e., $h^{*}=argmi{n}_{h\in \mathcal{H}}{ϵ}_S\left(h\right)+{ϵ}_T\left(h\right)$. Let us denote ${ϵ}_S\left(h^{*}\right)+{ϵ}_T\left(h^{*}\right)\triangleq {\lambda }_S+{\lambda }_T=\lambda$. Then, for a classifier h, we have

$${ϵ}_T\left(h\right)\le {\widehat{ϵ}}_S\left(h\right)+{\delta }_S\left(h\right)+d_{\mathcal{H}\Delta \mathcal{H}}(p\left(t\right),q\left(t\right))+\lambda ,$$

(1)

where $\widehat{ϵ}$ is the empirical error; ${\delta }_S\left(h\right)$ is the source generalization error gap, i.e., ${\delta }_S\left(h\right)=|{\widehat{ϵ}}_S\left(h\right)-{ϵ}_S\left(h\right)|$; and $d_{\mathcal{H}\Delta \mathcal{H}}(p\left(t\right),q\left(t\right))$ is the RD defined by the $\mathcal{A}$-distance, i.e., $d_{\mathcal{H}\Delta \mathcal{H}}(p\left(t\right),q\left(t\right))={sup}_{h\in \mathcal{H}}|E_{t\sim p\left(t\right)}{I}_{\{h^{*}\left(t\right)\ne h\left(t\right)\}}-E_{t\sim q\left(t\right)}{I}_{\{h^{*}\left(t\right)\ne h\left(t\right)\}}|$.

Please note that we assume that the ground-truth labeling function on the two domains remains the same, which is common, as used in many previous studies. If there is a conditional shift, i.e., a distance between $f_S$ and $f_T$, where $f_S$ and $f_T$ are the deterministic ground-truth-induced labeling function on the source and target domains, respectively, as defined in Zhao et al. [28], the above decomposition is not valid any more. The assumption is reasonable since the conditional shift phenomenon is not observed in our experiments.

According to the above theorem, we need to minimize the following three terms to achieve a low expected error, i.e., the training error on source domain, and SG and RD between the marginal distributions on the two domains (RD). Assume that both the IB and DIB methods have the ability to achieve comparable small source training errors, so we focus on SG and RD to compare the two methods.

3. Main Results

3.1. Source Generalization Study of IB and DIB

In this subsection, we study the generalization of IB and DIB on the source domain. The generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit the training data. The generalization error gap in statistical learning theory is defined as the expected difference between the population risk and the empirical risk. Russo and Zou [29], Xu and Raginsky [30] provided a generalization upper bound with mutual information of the input and output. Sefidgaran et al. [31], Sefidgaran et al. [32] further related the mutual information with two other studies of generalization error, i.e., compressibility and fractal dimensions, providing a unifying framework for the three directions of studies.

In the theoretic study of generalization and adversarial robustness of IB, however, the measure is not exactly the same for the convenience of illustrating the characteristic of IB [8,9]. Specifically, in Section 4.1 in Shamir et al. [8], the classification error is proven to be exponentially upper bounded by $-I(Y;T)$, indicating that $I(Y;T)$ is a measure of performance. Therefore, the difference between the population performance $I(Y;T)$ and empirical performance $\widehat{I}(Y;T)$ is used to measure the ability of generalization. Shamir et al. [8] provided the following generalization upper bound, which is relevant to the mutual information $I(X;T)$.

Theorem 2

(The previous generalization bound). Denote the empirical estimates by $\widehat{·}$. For any probability distribution $p(x,y)$, with a probability of at least $1-\delta$ over the draw of the sample of size m from p(x,y), we have that for all T,

$$\begin{array}{cc}& |I(Y;T)-\widehat{I}(Y;T)|\le \hfill \\ & \sqrt{\frac{B_{0}log\left(\right|\mathcal{Y}|/\delta )}{m}}\left(B_{1}log\left(m\right)\sqrt{\left|\mathcal{T}\right|I(X;T)}+B_2{\left|\mathcal{T}\right|}^{3/4}{\left(I(X;T)\right)}^{1/4}+B_3\widehat{I}(X;T)\right)\hfill \end{array}$$

(2)

where $B_0$ is a constant and $B_1,B_2,B_3$ only depend on m, δ, $\left|\mathcal{Y}\right|$, ${min}_{x}p\left(x\right)$, and ${min}_{y}p\left(y\right)$, where min refers to the minimum value other than 0.

However, this upper bound cannot be directly used to compare IB and DIB because both IB and DIB minimizes $I(X;T)$. Moreover, the regularizer of DIB: $H\left(T\right)=I(X;T)+H\left(T\right|X)$ further minimizes the term $H\left(T\right|X)$. However, it is not clear whether it may bring any advantage from the previous theoretical result. To tackle this problem, we prove a new generalization bound in this paper, as shown in the following theorem.

Lemma 1.

If $X\in [0,1]$, $EX=\mu$, then $Var\left(X\right)\le (1-\mu )\mu$.

Lemma 2.

If $\sum x=1$, $\forall 0\le x\le 1$, then ${\sum }_x$$-\sqrt{x(1-x)}$$ln\left(\sqrt{x(1-x)}\right)\le {\sum }_x-\sqrt{x}lnx.$

Theorem 3

(Our generalization bound). For any probability distribution $p(x,y)$, with a probability of at least $1-\delta$ over the draw of the sample of size m from p(x, y), we have that for all T,

$$\begin{array}{cc}& |I(Y;T)-\widehat{I}(Y;T)|\le \hfill \\ & \frac{1}{\sqrt{m}}\left((C_1+C_3)\sqrt{\left|\mathcal{T}\right|-1}+C_{2}H\left(T\right)+C_{4}H\left(T\right|Y)+C_5\sqrt{(log|\mathcal{T}|-\widehat{H}\left(T\right|Y))\widehat{H}\left(T\right|Y)}\right)\hfill \end{array}$$

(3)

where $C_1,C_2,C_3,C_4,C_5$ only depend on δ, $\left|\mathcal{Y}\right|$, ${min}_{x,y}p\left(x\right|y)$, ${min}_{t,y}p\left(t\right|y)$, ${min}_{x}p\left(x\right)$, and ${min}_{t}p\left(t\right)$.

Proof. 

Here, we show a proof sketch to help us understand how our bound is different from the previous one. The complete proof is in the appendix. Similarly to the previous proof, SG is firstly divided into three parts.

$$\begin{array}{cc}& |I(Y;T)-\widehat{I}(Y;T)|\le \hfill \\ & |H\left(T\right)-\widehat{H}\left(T\right)|+|\sum _{y}p\left(y\right)\left(H\left(T\right|y)-\widehat{H}\left(T\right|y)\right)|+|\sum _y(p\left(y\right)-\widehat{p}\left(y\right))\widehat{H}\left(T\right|y)|\hfill \end{array}$$

(4)

Denote ${\Delta }_1=|H\left(T\right)-\widehat{H}\left(T\right)|$, ${\Delta }_2=\left|{\sum }_{y}p\left(y\right)\left(H\left(T\right|y)-\widehat{H}\left(T\right|y)\right)\right|$, ${\Delta }_3=\left|{\sum }_y(p\left(y\right)-\widehat{p}\left(y\right))\widehat{H}\left(T\right|y)\right|$. Then, we can summarize the previous proof and our proof in Figure 1.

The main idea of the previous proof is to bound these parts by $\varphi$ functions of sample variances in (5) and (9), where $V_x\left(p\left(t\right|x)\right)\triangleq \frac{1}{\left|\mathcal{X}\right|}{\sum }_x{(p\left(t\right|x)-\frac{1}{\left|\mathcal{X}\right|}{\sum }_{x}p\left(t\right|x))}^2$, and $\varphi \left(x\right)$ is defined by (A1). The variances are then related to the $L_1$ distances in (6) and (10); KL-divergence in (7); and lastly, mutual information in (8) and (11).

We also use the idea of bounding by $\varphi$ functions of variances, as shown in (12), (15), and (18). However, the variance is an exact one, instead of the previous estimation on the samples, i.e., $Va{r}_X\left(p\left(t\right|x)\right)\triangleq E_{p\left(x\right)}{[p\left(t\right|x)-E_{p\left(x\right)}\left[p\left(t\right|x)\right]]}^2$. Consequently, we can obtain a tighter bound. Specifically, we first use Lemma 1 to convert the variance to the expectation in (13), (16), and (19). Then, with the help of Lemma 2, we turn the $\varphi$ functions into entropy, like from (13) to (14). With (14), (17), and (20), we finish the proof of Theorem 3.    □

We can see that this bound is similar to the previous one, but the new bound is related to the regularization term $H\left(T\right)$, $H\left(T\right|Y)$ and $\widehat{H}\left(T\right|Y)$, instead of the mutual information $I(X;T)$. Note that $H\left(T\right|Y)$ and $\widehat{H}\left(T\right|Y)$ are closely related by $|H\left(T\right|Y)-\widehat{H}\left(T\right|Y)|\le \left(17\right)+\left(20\right)$. Additionally, $H\left(T\right)$ is equivalent to $H\left(T\right|Y)$ as a regularizer because $L_{DIB}=H\left(T\right)-\beta I(Y;T)=H\left(T\right|Y)-(\beta -1)I(Y;T)$. With these two observations above, this bound is directly dependent on the entropy $H\left(T\right)$, i.e., the regularizer of DIB. That is to say, compressing the entropy of the representation is beneficial to reducing SG.

Now we will illustrate why DIB generalizes better. Intuitively, since DIB directly minimizes $H\left(T\right)$ while IB only minimizes a part of them, i.e., $I(X;T)$, DIB may have a better generalization than IB. Owing to the lack of close form solutions, we cannot explicitly see the difference between IB and DIB with respect to the concerning information quantities. However, we can solve the IB and DIB problems through a self-consistent iterative algorithm, with $\alpha =0$ or 1 in Appendix B.1. The IB and DIB solutions are shown on the information plain in Appendix C with $\alpha =0$ or 1, showing that the IB solutions have larger $H\left(T\right)$ than DIB solutions, and thus, DIB generalizes better in the sense of Theorem 3 in our paper.

Furthermore, our bound is tighter than the previous one. We provide a comparison of the order in Appendix A.2, which shows that the IB bound is $\sqrt{\left|X\right|}$ times larger than the DIB bound for the first two terms and $\sqrt{\left|Y\right|}$ times larger for the third term. There is also an empirical comparison in Section 4.1.2. Moreover, Theorems 2 and 3 require some constraints for sample size. They are summarized in the Appendix A.2 too. The empirical results show that our bound requires a smaller sample size.

To sum up, we provide a tighter generalization bound in the IB framework, which serves as a theoretical support for DIB’s generalization performance. Experiments on MNIST will validate this theoretical result in Section 4.2.1.

3.2. Representation Discrepancy of IB and DIB

In this section, we compare the RD of IB and DIB. According to the target error decomposition theorem in Section 1, RD is measured by $\mathcal{H}\Delta \mathcal{H}$-distance, which is however difficult for direct computation because of the complex hypothesis space $\mathcal{H}$. To remove the dependence of $\mathcal{H}$, we propose to bound RD by a pair-wise $L_1$ distance.

To start with the simplest case, assume that the sample sizes on the source and target domains are both m, and the samples on two domains have a one-to-one correspondence (i.e., semantically close to each other). Then, RD can be bounded by the following pair-wise $L_1$ distance.

Proposition 1.

The distance of overall representations on the source and target domains is bounded by the distance of individual instance representations.

$$\begin{array}{cc}\hfill d_{\mathcal{H}\Delta \mathcal{H}}(p\left(t\right),q\left(t\right))\le & \frac{1}{m}\sum _{(x_S,x_T)\in \mathcal{CP}}{\parallel p\left(t\right|x_S)-p\left(t\right|x_T)\parallel }_1+ϵ,\hfill \end{array}$$

(21)

where $\mathcal{CP}$ stands for the set of all correspondence pairs and $ϵ=2\sqrt{\left|\mathcal{T}\right|}\frac{2+\sqrt{2log(1/\delta )}}{\sqrt{m}}$, which is small when the sample size is large.

In fact, Proposition 1 is valid for any $(x_S,x_T)$ pair, so there can be different upper bounds. To obtain the lowest upper bound, we define correspondent pairs such that ${\sum }_{(x_S,x_T)\in \mathcal{CP}}{\parallel p\left(t\right|x_S)-p\left(t\right|x_T)\parallel }_1$ is the lowest, i.e., $\mathcal{CP}=\{{(x_{S,i},x_{T,i})}_{i=1\dots m}=argmin{\sum }_{i=1}^m\parallel p\left(t\right|x_{S,i})-p\left(t\right|x_{T,i}){\parallel }_1|{\left\{x_{S,i}\right\}}_{i=1\dots m}={\mathcal{X}}_S,{\left\{x_{T,i}\right\}}_{i=1\dots m}={\mathcal{X}}_T\}$.

We parameterize $p\left(t\right|x)$ to be a d-dimensional Gaussian with a diagonal covariance matrix, which is usually assumed in some variational methods such as Alemi et al. [15]. Compared with IB, DIB additionally reduces $H\left(T\right|X)$, so its $p\left(t\right|x)$ are almost deterministic. Therefore, the variance in $p_{DIB}\left(t\right|x)$ in each dimension are significantly smaller than that of $p_{IB}\left(t\right|x)$. On the other hand, since the only difference between IB and DIB is altering $H\left(T\right|X)$ and the entropy of Gaussian random variable is only dependent on its variance, the expectations of $p_{IB}\left(t\right|x)$ and $p_{DIB}\left(t\right|x)$ are comparable. Therefore, we need to find how the variances affect RD.

Consider the representations of the instances from the source and target domains in the same model. $\forall (x_S,x_T)\in \mathcal{CP}$, since $x_S$ and $x_T$ are semantically close, the expectations of their representations are close. Additionally, the discrepancy between their variances are small compared to the discrepancy in the representation variances between IB and DIB, so we can neglect them. Therefore, denote that $p\left(t\right|x_S)\sim N({\mu }_1,\Sigma )$, $p\left(t\right|x_T)\sim N({\mu }_2,\Sigma )$, $\Sigma =diag({\sigma }_1^2,\dots ,{\sigma }_d^2)$; then,

$$\parallel p\left(t\right|x_S)-p\left(t\right|x_T){\parallel }_1={\Pi }_{i=1}^d(4\Phi \left(\frac{|{\mu }_{1i}-{\mu }_{2i}|}{2{\sigma }_i}\right)-2)$$

(22)

where $\Phi$ is the cumulative distribution function of a standard Gaussian distribution.

As discussed above, compared with IB, DIB has significantly smaller variances in the representations and ${\mu }_1-{\mu }_2$ are comparable for IB and DIB. Therefore, the term $\frac{|{\mu }_{1i}-{\mu }_{2i}|}{2{\sigma }_i}$ for IB is remarkably smaller than DIB. With $\Phi$ monotonically increasing, $\parallel p\left(t\right|x_S)-p\left(t\right|x_T)\parallel$ for IB is smaller than that for DIB. Figure 2 provides an intuitive understanding about how randomness helps reduce RD. Suppose that the blue and red lines are $p\left(t\right|x_S)$ and $p\left(t\right|x_T)$, where $(x_S,x_T)\in \mathcal{CP}$. We can see clearly that their $L_1$ distance drops with the growth in variances. Moreover, because the derivative of $\varphi$ monotonically decreases on $[0,+\infty ]$, when ${\mu }_2-{\mu }_1$ is smaller, the difference between IB and DIB will be larger. This phenomenon is also found in simulations; see Section 4.1.1 and Section 4.1.3.

When the sample size on the two domains are different and the “pair-wise” correspondence does not hold, we can take the correlated instances in the two domains as a general form of correspondent pairs; then, the above comparison result is also valid. Details are found in Appendix A.3.

Please note that the assumption of correspondent pairs is reasonable. In transfer learning, it is widely assumed that there exists shared (high level) features and common distributions of representation on the two domains. The correspondent pairs can be viewed as the instance pairs with the closest distributions of representations from the two domains.When the distributions are distinct, feature alignment is usually implemented in practice.

According to the above theoretical results, we can obtain the following comparisons between IB and DIB, with respect to the three terms for the target error decomposition, as illustrated in

Table 1. A comparison of IB and DIB.

Decomposition Terms	IB vs. DIB
Source Training Error	IB ≈ DIB
Source Generalization Gap (SG)	IB > DIB
Representation Discrepancy (RD)	IB < DIB

. Clearly, there is a trade-off between IB and DIB, in terms of SG and RD.

3.3. Elastic Information Bottleneck Method

From the previous results, IB and DIB still have room for improvement, respectively, on SG and RD. To obtain a Pareto optimal solution under the IB principle, we propose a new bottleneck method as follows, namely EIB. Please note that the generalized IB objective function in  [7] is the same as the objective function of elastic IB, but they are derived from different perspectives. The generalized IB is constructed for the convenience of solving the DIB problem, while EIB is proposed to balance SG and RD.

Definition 1

(Elastic Information bottleneck). The objective function of the elastic information bottleneck method is as follows:

$$\begin{array}{cc}\hfill min{L}_{EIB}=(1-\alpha )H\left(T\right)& +\alpha I(X;T)-\beta I(Y;T),0\le \alpha \le 1,\beta >1\hfill \end{array}$$

(23)

We can see that EIB is a linear combination of IB and DIB, which covers IB and DIB as special cases. Specifically, EIB reduces to IB when $\alpha =1$ and DIB when $\alpha =0$. Since IB and DIB perform better than the other one on SG or RD, respectively, a linear combination may lead to better target performance. In fact, the global optimal solution of the linear combination of the objectives is in the Pareto optimal solution set. Therefore, by adjusting $\alpha$ in $[0,1]$, we can obtain a balance between SG and RD and achieve a Pareto frontier within the IB framework.

As a bottleneck method similar to IB, its optimal solution can be calculated by iterative algorithms when the joint distribution of instances and labels are known. The algorithm and EIB’s optimal solutions on information planes are provided in Appendix B. However, the iterative methods are intractable when the data are numerous and high-dimensional. Therefore, we use the variational approaches similar to the method in Fischer [2] to optimize IB, DIB, and EIB objective functions by neural networks. Assume the representation is a k-dimension Gaussian distribution and all dimensions are independent. The network contains an encoder $p\left(t\right|x)$, a decoder $q\left(y\right|t)$, and a backward encoder $b\left(t\right|y)$. To be specific, the encoder is an MLP which outputs the K-dimensional expectations ${\mu }_{\mathbf{1}}$ and K-dimensional diagonal elements of covariance matrix ${\sigma }_{\mathbf{1}}$ and then yield the Gaussian representation with the reparameterization trick [15]. The decoder is a simple logistic regression model. The backward encoder is an MLP that uses the onehot encoding of the classify outcome as an input and the K-dimensional expectations ${\mu }_{\mathbf{2}}$ and K-dimensional diagonal elements of covariance matrix ${\sigma }_{\mathbf{2}}$ as an output. The loss function of variational EIB is as follows:

$$\begin{array}{cc}& Los{s}_{EIB}=\int p\left(t\right|x)p(x,y)\left(log\frac{p^{\alpha }\left(t\right|x)}{b\left(t\right|y)}-\tilde{\beta }logq\left(y\right|t)\right)\mathrm{d}x\mathrm{d}t\mathrm{d}y,0\le \alpha \le 1,\beta >0.\hfill \\ & Los{s}_{EIB}\approx \frac{1}{m}\sum _{n=1}^m\left[\sum _{j=1}^k\right[\frac{(1-\alpha )}{2}ln\left(2\pi \right)-\alpha ln{\sigma }_{1,j,n}+ln{\sigma }_{2,j,n}-\frac{\alpha }{2}\hfill \\ & +\frac{{\mu }_{1,j,n}^2+{\mu }_{2,j,n}^2-2{\mu }_{1,j,n}{\mu }_{2,j,n}+{\sigma }_{1,j,n}^2}{2{\sigma }_{2,j,n}^2}]+\tilde{\beta }CE[softmax\left(y_n\right)\left|\right|q\left(y_n\right|t_n)\left]\right],\hfill \end{array}$$

(24)

where $CE$ is the cross entropy loss.

Our simulations and real data experiments show that EIB outperforms IB and DIB. It is also worth noticing that EIB can be plugged into previous transfer learning algorithms to expand its application. An example is given in the next section.

4. Experiments

In this section, we evaluate IB, DIB, and EIB by both toy data simulations and real data experiments.

4.1. Toy Data Simulations

We conduct simulations to study the performance of EIB in the transfer learning scenario, a comparison of RD between IB and DIB, and the comparison between our generalization bound and the previous one.

4.1.1. Performance of EIB in Transfer Learning

We design a toy binary classification problem, where instance $X\in {\{0,1\}}^{10}$ and label $Y\in \{0,1\}$. Define $X_0$ = [1,1,1,1,1,0,0,0,0,0], $X_1$ = [0,0,0,0,0,1,1,1,1,1], $Y_0=0$, and $Y_1=1$. First, in the source and target datasets, half of the examples is $(X_0,Y_0)$ and the other half is $(X_1,Y_1)$. Then we introduce random noise into the instances in the following way. We define the reverse digit operation as first choosing one digit in the instance and then reversing the digit (changing from 0 to 1 or the opposite direction). For each instance, we perform the reverse digit operation $\lfloor N\rfloor$ times (round down function), where N is a real-valued uniform random variable $N\sim U[0,R]$. R is named as the noise level because as R increases, more instances are very different from $X_0$ and $X_1$ in the dataset. As a result, we can adjust the similarity between two domains by modifying the parameter R. Lastly, we set the $R\in (1,3)$ for the source domain and $R=3$ for the target domain so that we obtain the toy data in a transfer learning scenario.

We test our EIB model on the toy data. The parameter $\alpha$ ranges in [0,1], indicating different EIB models. The other parameter $\beta$ is chosen to be ${10}^4$ for $R=2,1.5,1.433$ and $\beta =5\times {10}^3$ for $R=1.375,1.25$ in order to discriminate the model performance and to obtain high accuracy. The results are shown in

Table 2. Accuracy of elastic information bottleneck (EIB) on simulated data with different noise levels R. The highest accuracy under each noise levels are marked in bold.

$\mathit{\alpha }$	R = 2	R = 1.5	R = 1.433	R = 1.375	R = 1.25
0	98.85	97.20	98.55	97.65	98.30
0.1	98.30	97.40	98.70	97.30	98.35
0.2	98.45	97.40	98.75	98.15	98.05
0.3	98.45	97.70	98.65	97.85	98.20
0.4	98.50	97.75	98.60	97.45	98.20
0.5	99.10	97.40	98.65	97.55	98.25
0.6	99.15	97.45	98.55	97.40	98.25
0.7	98.60	97.65	98.60	97.35	98.25
0.8	98.65	97.55	98.60	97.25	98.00
0.9	98.60	98.53	98.65	97.20	98.10
1.0	99.25	97.75	98.65	97.50	97.90

. From the results, we can see that EIB outperforms IB and DIB when $R=1.5,1.433,1.375,1.25$. To be more specific, when the two domains become more similar, i.e., the noise level R of the source domain becomes closer to $R=3$ of the target domain, $\alpha$ of the best model changes from 0 to 1, i.e., IB gradually becomes more advantageous. This is in accordance with our theory that when the two domains are similar, the effect of RD is apparent and IB has more advantages in transfer learning. This result also provides an empirical rule to tune the parameter $\alpha$.

4.1.2. Our Generalization Bound vs. the Previous One

In this simulation, we compare the DIB generalization bound (ours) and the IB generalization bound [8] in terms of number size and required sample size.

Assume that the data, representations, and labels are discrete, i.e., $\left|\mathcal{X}\right|$ = 3,$\left|\mathcal{T}\right|$ = 2, $\left|\mathcal{Y}\right|$ = 2, $\delta =0.1$. The sample size increases exponentially from ${10}^1$ to ${10}^6$. The bound and error rate are computed as follows. First, we generate distributions $p(x,y)$, $p\left(t\right|x)$ and sample an empirical distribution $\widehat{p}(x,y)$. To compare the bounds in a general case, $p(x,y)$ is randomly valued by numbers generated from a uniform or a normal distribution and the value of $p\left(t\right|x)$ is also randomly generated from a uniform distribution or a normal distribution. $\widehat{p}(x,y)$ is the empirical probability of $p(x,y)$. Second, we determine whether the constraints for the bound in Appendix A.3 are met. If so, we calculate the value of the bound. If not, we record the number of times the constraints are not satisfied. Third, we repeat this process for 100 times to obtain the rate of constraint violations (error rate) and two average bound values. Finally, we repeat the previous three steps under different sample sizes.

Figure 3 (left) shows that our bound consistently needs less samples than the previous one to satisfy the constraints. Moreover, the more uniform the distribution is, the less samples are needed to satisfy the constraints, which is obvious from the formula of constraints. Figure 3 (right) shows that the generalization bound decreases as the sample size increases, and the DIB generalization bound is always smaller than the IB generalization bound.

4.1.3. Comparison of RD between IB and DIB

To support the claims about RD, we approximate RD by using a classifier to predict which domain the representation sample t comes from, which is proven to be an adequate approximation of RD in [26].

The data generation, model, and parameters (R,$\beta$) are consistent with the simulation in Section 4.1.1. The data are generated two times under different noise levels R and are trained by EIB models with $\alpha$ = 0 or 1. Then, five samples are drawn from the Gaussian representations $p\left(t\right|x)$ for each instance x. If x comes from the source domain, we label the samples as positive; otherwise, we label it as negative. Then, we train a linear classifier to classify the samples. Each classifier is trained 20 times. The average error rate is shown in

Table 3. The average error rate of classification between representation samples on two domains with difference noise levels. A larger error rate indicates smaller RD. The highest error rate under each noise levels are marked in bold.

R	2	1.5	1.433	1.375	1.25
$\beta$	${10}^4$	${10}^4$	${10}^4$	$5\times {10}^3$	$5\times {10}^3$
DIB ($\alpha$ = 0)	43.65% ± 0.20%	28.05% ± 0.15%	28.14% ± 0.35%	28.99% ± 0.71%	27.72% ± 0.50%
IB ($\alpha$ = 1)	45.46% ± 0.26%	28.58% ± 0.18%	28.54% ± 0.35%	32.84% ± 1.25%	27.46% ± 0.35%

. The standard error of the mean (SEM) is in parentheses.

The results show that IB has a larger error rate than DIB in most cases, indicating that IB has a smaller RD, which is consistent with the main result in Section 3.2. Furthermore, the gap between IB’s and DIB’s error rates becomes larger with the growth of R, which validates that when two domains become more similar, IB’s advantage over RD becomes more significant, as is claimed in Section 3.2 and Section 4.1.1. This trend does not seem to be consistent between $R=1.433$ and $R=1.375$ because $\beta$, which is chosen to minimizes the error rate, is different.

4.2. Real Data Experiments

Similar to IB, EIB can also be utilized in many representation learning algorithms as a regularizer. As an example, we combine EIB with DFA-MCD in Wang et al. [33], which is an adversarial domain adaptation algorithm with feature alignment. We replace the features in DFA-MCD with Gaussian representations and add a backward encoder after the source classifier as in variational EIB. Since the feaures are already Gaussian, we remove the KLD regularizer in DFA-MCD, which is the KL-divergence penalty between source domain feature and a Gaussian prior. Then, we substitute cross entropy loss with EIB loss. The adversary training and feature alignment designs are retained.

We test the model on a common transfer learning task, where the source dataset is MNIST [34] and the target dataset is USPS [35]. We only use 2/55 of the training data, and $\lambda$ = 15, $\beta ={10}^9$, other parameters remain the same as in Wang et al. [33]. The experiment is repeated with six random seeds, and the results are shown in Figure 4 (left). For each box, the central mark indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted individually using the ‘+’ symbol. It is automatically generated by the boxplot function in matlab. We can see that the model with EIB performs better than IB and DIB in most cases, e.g., $\alpha =0.2,0.3,0.4,0.6,0.7,0.8$, and some of them beats the DFA-MCD baseline model, which suggests that EIB works as an effective regularizer for transfer learning. Please note that this example is utilized to demonsrate EIB can be combined with domain adaptation algorithms and perform better than IB and DIB, so we simply inherit the structure and hyper-parameters of the original networks in Wang et al. [33]. Further parameter tuning can be conducted to achieve better experimental results.

4.2.1. Source Generalization Analysis

We use variational EIB on MNIST to compare SG of IB and DIB. The networks were trained for 100 epochs and converge at about 60 epochs. The average generalization gap is calculated by the mean discrepancy of the training error and the testing error of 25 epochs with the least testing error. We randomly initialize the network 14 times and utilize EIB models with $\alpha =0$ (DIB), $\alpha =0.5$, $\alpha =1$ (IB) under the same initialization. The results are shown in Figure 4 (right) and

Table 4. T-test on “DIB’s SG < IB’s SG”. Statistically significant results are marked in bold.

$\mathit{\beta }$	${10}^4$	$5\times {10}^4$	${10}^5$	${10}^6$	${10}^7$	${10}^8$
pvalue	0.021	0.020	0.042	0.66	0.085	0.39

. Baseline is a model without a regularizer i.e., $\beta =+\infty$ in EIB. First, let us analyze the results in terms of $\beta$. When $\beta$ is small, models over-compress the representations so that the error rate is large. When $\beta$ is large, the weight of regularization term in objective function is small so the three models’ performance become close. When $\beta ={10}^5$, the three models have the best accuracy. Then, in terms of $\alpha$, with the decrease in $\alpha$, the generalization gap becomes smaller. When $\beta ={10}^4,5\times {10}^4,{10}^5$, the p value of the t-test on “DIB’s SG < IB’s SG” is less than 0.05, indicating that the source generalization error of DIB is statistically significantly smaller than that of IB. (The p value is a term in statistical test that is defined as the probability of obtaining a test statistic as extreme or more extreme than the test statistic of actual observations if the null hypothesis is true. Usually, if p value < 0.05, the null hypothesis should be rejected and the result is considered statistically significant) This validates that DIB generalizes better than IB.

5. Conclusions and Future Work

This work studies the two objective functions of the information bottleneck principle. The motivation comes from our theoretical analysis that neither IB nor DIB is the optimal solution in terms of the generalization ability under the transfer learning scenario. Specifically, we theoretically analyze SG and RD of IB and DIB, and find that there is a trade-off between them. To tackle this problem, we propose a new method, EIB, to interpolate IB and DIB. Consequently, EIB can not only achieve better transfer learning performances but also be plugged into existing domain adaptation methods as a regularizer to suit different kinds of data, which have been shown by our simulations and real data experiments.

We believe that our results take an important step towards understanding different information bottleneck methods and provide some insights into the design of stronger deep domain adaptation algorithms. We qualitatively suggest choosing $\alpha$, but in practice, when the distance of the two domain is fixed, the optimum still needs careful tuning. Therefore, how to choose the best parameters $\alpha$ and $\beta$ efficiently in EIB remain questions that we plan to study in the future.