MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

Abstract

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV–MR). MV–MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV–MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV–MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets.

1. Introduction

Self-supervised learning (SSL) methods are alternatives to supervised ones. In recent years, the gap between SSL and supervised methods has decreased in performing downstream tasks, including image classification [1], object detection [2], and semantic image segmentation [3,4]. A general idea behind the SSL models for image classification is to train an embedding network, often called an encoder, on an unlabeled dataset and then to use this pretrained encoder for the downstream tasks.

The general goal is to ensure the invariance of embeddings to different inputs known as augmentations or views. However, this approach might lead to trivial solutions when two branches of encoders produce the same output. As a result, one observes an effect known as a collapse in training when no meaningful representation can be learned for different inputs. Therefore, there have been a lot of recent works tackling this issue by regularizing the networks to avoid such a collapse. Several key approaches have been developed to mitigate these negative impacts, using different tactics. The first group of methods aims to directly maximize the mutual information between the input image $\mathbf{x}$ and its positive pairs created on the basis of augmentations. In this view, an exponential prior on the conditional distribution in the representation space and an associated contrastive loss with positive–negative pairs as in InfoNCE [5] is assumed. Unfortunately, such an approach is quite computationally expensive in practice, due to the need for a large batch size to incorporate the large number of negative pairs. The second group of methods aims to avoid collapse by introducing different asymmetries in two branches at the training stage. Examples of this approach are training one network with gradient descent and updating the other with an exponential moving average of the weights of the first network [6] or introducing regularizers on the learned representations such as regularization by decorrelation on the dimensions of the embeddings [7], etc. The third group of methods is Masked Image Modeling (MIM). These methods primarily focus on avoiding collapse and learning rich image representations by predicting the missing parts in masked inputs. This methodology relies on masking a portion of the input image and training the model to predict these masked parts, thereby learning contextual and semantic information. A notable method in this domain is the BEiT [8], which introduces a transformer-based model that learns to predict the masked visual tokens, analogous to the masked language modeling in NLP. Another significant approach is the MAE (Masked Autoencoder) [9], which uses an asymmetric encoder–decoder structure, where the encoder processes only visible patches and the decoder reconstructs the masked patches. While MIM effectively learns representations, it is a transformer-specific approach and not transferable to other architectures.

The proposed approach avoids the embeddings’ collapse by introducing the dependence maximization between trainable embeddings and hand-crafted features using distance correlation [10]. Distance correlation, unlike other losses in latent space, allows computing dependencies between feature vectors of different shapes. We maximize the dependence between different embeddings while preserving the variance in them. We show that variance preservation maximizes the entropy of embeddings, which makes them unique and distinguishable. Our approach is different from InfoNCE [5], which advocates a contrastive loss that maximizes the mutual information (MI) between input image $\mathbf{x}$ and its positive pairs. In contrast to the InfoNCE, our approach is not contrastive, does not require large batch sizes, and allows computing the distance between embeddings and features of any shape. It is also different from methods such as Barlow Twins [7] and VICReg [11] since we do not explicitly minimize the dependencies between the components within the embedding.

We also show that the proposed approach can be used for efficient representation learning and latent space-agnostic knowledge distillation. The approach is based on the dependence maximization between the embeddings of the target trainable encoder, represented by the ResNet50 [12], and the embeddings of the pretrained encoder, represented by the CLIP [13] (based on ViT-B-16 [14]). Since the distance correlation is agnostic to the latent space shape, any pretrained encoder with any latent space can be used for knowledge distillation. To our best knowledge, we are the first to propose a model-distillation method that is agnostic to the latent space shape.

The main goal behind MV–MR is twofold: (i) maximizing the invariance of embeddings, i.e., maximizing the proximity of embeddings for the same image observed under different views, and (ii) maximizing the amount of information in each embedding, i.e., maximizing the variability of the embedding. Furthermore, to avoid the collapse during training, we regularize the branch with the augmentations by imposing the dependence constraints on a set of representations extracted from various encodings.

The proposed approach introduces several unique features: (i) we introduce a novel SSL approach that avoids collapse thanks to an additional regularization term that maximizes the dependence between trainable embeddings and various feature vectors using distance correlation; (ii) up to our best knowledge, the proposed method is among the first that uses the dependence maximization of the latent space based on distance correlation for SSL; (iii) the proposed method is agnostic to the latent space shape and, thus, can be used with any types of features; (iv) we introduce a novel knowledge distillation technique that is agnostic to model and shape of latent space; (v) we demonstrate the state-of-the-art classification results on the STL10 [15] (89.71%) and CIFAR20 [16] (73.2%) datasets using a linear evaluation protocol for non-contrastive SSL methods; (vi) we provide the information-theoretic explanation of the proposed method that contributes to the explainable ML; (vii) we demonstrate how the complex CLIP model with 86.2 M parameters trained on 400 M text–image pairs can be distilled into a ResNet50 model with just 23.5 M parameters trained on the STL10, CIFAR100 [17], and ImageNet-1k [18] datasets; (viii) we achieve state-of-the-art performance in knowledge distillation in the image-classification task using the ResNet50 model as student and CLIP ViT-B-16 as teacher on CIFAR100, with 78.6% accuracy.

We have three loss terms in our objective function: (a) the first term ${\mathcal{L}}_1$ consists of the mean square error (MSE) loss between the embeddings from the non-augmented view and augmented views of the same image; it is used for the invariance of embeddings, and we introduce additional variation terms that are used for maximization of the variability of the embeddings (we demonstrate that this term originates from an upper bound on mutual information between these embeddings under corresponding assumptions); (b) the second term ${\mathcal{L}}_2$ stands for the distance correlation between the embeddings from the augmented and non-augmented views that complements the first term to capture non-linear relations between the embeddings; and (c) the third term ${\mathcal{L}}_3$ corresponds to the distance correlation between the embeddings from the augmented view and multiple image representations. For the non-learnable or hand-crafted representations, we have studied various techniques of invariant data representation that are well-known in computer vision and image-processing applications. The studied hand-crafted features include, but are not limited tp, ScatNet [19] features, local standard deviation (LSD)-based [20] filters, and histograms of oriented gradients (HOG) [21]. Additionally, to demonstrate the flexibility of the proposed method, we have also considered random augmentations of the original images flattened into feature vectors as instances of hand-crafted features. Since distance correlation is shape-agnostic for the features, we are able to combine features of different shapes in the loss functions. Also, replacing hand-crafted features with embeddings from pretrained networks is used for model distillation, without the need to change losses, architecture, or feature dimensionality.

2. MV–MR: Motivation and Intuition

MV–MR pretraining and distillation schemes are schematically shown in Figure 1 and Figure 2, respectively. The dimensions of embeddings with and without augmentations are the same, i.e., $\tilde{\mathbf{z}}\in {\mathbb{R}}^D$ and $\mathbf{z}\in {\mathbb{R}}^D$, respectively. These embeddings are extracted from the augmented $\tilde{\mathbf{x}}$ and non-augmented $\mathbf{x}$ via a generalized parametrized embedder $q_{{\varphi }_z}(·|·)$ that can be deterministic or stochastic with parameters ${\varphi }_z$. The encoder can be a parametrized neural network of any architecture. A $k^{th}$ hand-crafted descriptor ${\mathbf{z}}_k^{\ast }$, where $k\in \{1,2,\cdots ,K\}$ and K stands for the total number of hand-crafted descriptors, is generally a tensor of dimensions $H_k\times W_k\times C_k$ and is flattened to $D_k=H_k{W}_k{C}_k$. This descriptor is generally obtained via deterministic assignment ${\mathbf{z}}_k^{\ast }=f_{{\varphi }_{z_k^{\ast }}}\left(\mathbf{x}\right)$ or sometimes via stochastic mapping ${\mathbf{Z}}_k^{\ast }\sim q_{{\varphi }_{z_k^{\ast }}}\left({\mathbf{z}}_k^{\ast }|\mathbf{x}\right)$, where ${\varphi }_{{{\mathbf{z}}^{\ast }}_k}$ denotes the parameters of the $kth$ feature extractor.

2.1. Motivation: Regularization in Self-Supervised Representation Learning

The learned representation should contain the informative representation of data with lower dimensionality and should be invariant under some transformations, i.e., to ensure the same latent representation for the data from the same sample passed through certain transformations. The satisfaction of these conflicting requirements in practice is not a trivial task. Many state-of-the-art SSL techniques try to find a reasonable compromise between these requirements and practical feasibility solely in the scope of machine learning formulation by imposing certain constraints on the properties of the learned representation via the optimization of encoder parameters under augmentations.

At the same time, there exists a rich body of achievements in the computer vision community in the domain of the hand-crafted design of robust, invariant, yet discriminating data representations [21,22,23,24]. Generally, the computer vision descriptors are very rich in terms of targeted invariant features and quite efficient in terms of computation. However, to our best knowledge, such descriptors are not yet fully integrated into the development of SSL techniques. Therefore, one of the objectives of this paper is to propose a framework where the SSL representation learning might be coupled with the constraints on the embedding space offered by the invariant computer vision representations. Our objective is not to consider a case-by-case approach on how to couple SSL with a particular computer vision representation but instead to propose a generic approach where any form of desirable computer vision representation can be integrated into the SSL optimization problem in an easy and tractable way. This ensures that the learned representation possesses the targeted properties inherited from the used computer vision descriptors. Furthermore, features extracted by such descriptors might be considered as a form of invariant data representation, which is one of the desired properties of trained encoders. Thus, maximizing the dependence between the trainable embedding and such representation might be a novel form of regularization, leading to an increased-invariance yet collapse-avoiding technique. Since a single computer vision descriptor might not capture all desirable properties and have different representation formats, the targeted framework should be flexible enough to deal uniformly with all these descriptors within a simple optimization problem. Distance correlation is very useful for this kind of representation learning, since it allows one to incorporate features of any shapes, without the need to match the shape of learnable embeddings and hand-crafted target embeddings.

In summary, our motivation is to include regularization constraints on the solution by borrowing some inherent computer vision feature invariance to certain transformations. In this way, we target learning the low-dimensional embedding, which contains only essential information about the data that might be of interest for the targeted downstream task and where all information about the augmentations is excluded.

2.2. Intuition

The basic idea behind MV–MR is to introduce constraints on the invariance of embedding via a new loss function. Our overall objective is to maximize the mutual information $I(\tilde{\mathbf{Z}};\mathbf{Z})$ between the augmented embedding $\tilde{\mathbf{Z}}$ and the embedding without the augmentation $\mathbf{Z}$ and to maximize the mutual information $I(\tilde{\mathbf{Z}};{\mathbf{Z}}_k^{\ast })$ between $\tilde{\mathbf{Z}}$ and some invariant feature ${\mathbf{Z}}_k^{\ast }$ extracted from $\mathbf{X}$ using a mapper that ensures a known invariance to the desired transformation.

2.2.1. Measuring Dependencies between Embeddings of Non-Augmented and Augmented Data

Upper bound on mutual information: In the first case, one can decompose the mutual information as

$$\begin{array}{ccc}\hfill I(\tilde{\mathbf{Z}};\mathbf{Z})& ={\mathbb{E}}_{p(\tilde{\mathbf{z}},\mathbf{z})}\left[log\frac{p(\tilde{\mathbf{z}},\mathbf{z})}{p\left(\tilde{\mathbf{z}}\right)p\left(\mathbf{z}\right)}\right]\hfill & \hfill ={\mathbb{E}}_{p(\tilde{\mathbf{z}},\mathbf{z})}\left[log\frac{p\left(\tilde{\mathbf{z}}\right|\mathbf{z})}{p\left(\tilde{\mathbf{z}}\right)}\right]\\ \hfill & =h\left(\tilde{\mathbf{Z}}\right)-h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z}),\hfill \end{array}$$

(1)

where $h\left(\tilde{\mathbf{Z}}\right)=-{\mathbb{E}}_{p\left(\tilde{\mathbf{z}}\right)}\left[logp\left(\tilde{\mathbf{z}}\right)\right]$ denotes the differential entropy and $h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z})=-{\mathbb{E}}_{p(\tilde{\mathbf{z}},\mathbf{z})}$$\left[logp\left(\tilde{\mathbf{z}}\right|\mathbf{z})\right]$ denotes conditional differential entropy (we assume that the differential entropy is non-negative under the considered settings). Since the computation of the marginal distribution $p\left(\tilde{\mathbf{z}}\right)$ and conditional distribution $p\left(\tilde{\mathbf{z}}\right|\mathbf{z})$ is difficult in practice, we proceed by bounding these terms. We assume that the desired embeddings need to be bounded by some variance ${\sigma }_z^2$ to avoid a training collapse when the encoders produce constant and non-informative vectors so that the entropy-maximizing distribution for the first entropy term is the Gaussian one, i.e., $p\left(\tilde{\mathbf{z}}\right)\propto \frac{exp\left(-\frac{1}{2}{\tilde{\mathbf{z}}}^{\mathrm{T}}{\Sigma }_{\mathbf{z}}^{-1}\tilde{\mathbf{z}}\right)}{\sqrt{{\left(2\pi \right)}^D\left|{\Sigma }_{\mathbf{z}}\right|}}$, where ${\Sigma }_{\mathbf{z}}$ represents the covariance matrix.

The conditional entropy $h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z})$ is minimized when the embedding $\tilde{\mathbf{Z}}$ contains as much information as possible about $\mathbf{Z}$, i.e., when two vectors are dependent. Assuming that $p\left(\tilde{\mathbf{z}}\right|\mathbf{z})\propto \frac{1}{C_z}\exp (-{\beta }_{z}d(\tilde{\mathbf{z}},\mathbf{z}))$, where $d(\tilde{\mathbf{z}},\mathbf{z})$ denotes some distance between two vectors such as the ${\ell }_2$-norm for the Gaussian distribution or ${\ell }_1$-norm for the Laplacian one, where $C_z$ stands for the normalization constant and ${\beta }_z$ denotes a scaling parameter. Thus, the minimization of the conditional entropy $h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z})$ reduces to the minimization of the distance $d(\tilde{\mathbf{z}},\mathbf{z})$.

Distance covariance: Another way to measure the dependency between the data is based on distance covariance, as proposed by [10]. In the general case of dependence between the data, the distance covariance is non-invariant to strictly monotonic transformations, unlike mutual information. Nevertheless, the distance covariance has several attractive properties: (i) it can be efficiently computed for two vectors that have generally different dimensions $\tilde{\mathbf{z}}\in {\mathbb{R}}^D$ and $\mathbf{z}\in {\mathbb{R}}^{D^{\prime }}$, such that $D\ne D^{\prime }$, and (ii) it is easier to compute in practice in contrast to the mutual information. Additionally, the distance covariance captures higher-order dependencies between the data, in contrast to the Pearson correlation. The distance covariance ${dCov}^2(\tilde{\mathbf{Z}},\mathbf{Z})$, proposed by [10], is defined as

$${dCov}^2(\mathbf{Z},\tilde{\mathbf{Z}})=\frac{1}{c_D{c}_{D^{\prime }}}{\int }_{{\mathbb{R}}^{D+D^{\prime }}}\frac{{\left|{\phi }_{\mathbf{Z},\tilde{\mathbf{Z}}}(\mathbf{t},\mathbf{u})-{\phi }_{\mathbf{Z}}\left(\mathbf{t}\right){\phi }_{\tilde{\mathbf{Z}}}\left(\mathbf{u}\right)\right|}^2}{{\left|\mathbf{t}\right|}_D^{1+D}{\left|\mathbf{u}\right|}_{D^{\prime }}^{1+D^{\prime }}}d\mathbf{t}d\mathbf{u},$$

(2)

which measures the distance between the joint characteristic function ${\phi }_{\mathbf{Z},\tilde{\mathbf{Z}}}(\mathbf{t},\mathbf{u})$ and the product of the marginal characteristic functions ${\phi }_{\mathbf{Z}}\left(\mathbf{t}\right){\phi }_{\tilde{\mathbf{Z}}}\left(\mathbf{u}\right)$ [10]. This definition has a lot of similarities to the mutual information in (1), which measures the ratio between the joint distribution $p(\tilde{\mathbf{z}},\mathbf{z})$ and the product of marginals $p\left(\tilde{\mathbf{z}}\right)p\left(\mathbf{z}\right)$. Since ${\phi }_{\mathbf{Z},\tilde{\mathbf{Z}}}(\mathbf{t},\mathbf{u})={\phi }_{\mathbf{Z}}\left(\mathbf{t}\right){\phi }_{\tilde{\mathbf{Z}}}\left(\mathbf{u}\right)$ when $\tilde{\mathbf{Z}}$ and $\mathbf{Z}$ are independent random vectors, the distance covariance is equal to zero.

In the following, we proceed with the normalized version of distance covariance, known as distance correlation, defined as

$$dCor(\tilde{\mathbf{Z}},\mathbf{Z})=\frac{{\mathrm{dCov}}^2(\tilde{\mathbf{Z}},\mathbf{Z})}{\sqrt{\mathrm{dVar}\left(\tilde{\mathbf{Z}}\right)\mathrm{dVar}\left(\mathbf{Z}\right)}},$$

(3)

where $0\le dCor(\tilde{\mathbf{Z}},\mathbf{Z})\le 1$ and $dVar\left(\mathbf{Z}\right)={dCov}^2(\mathbf{Z},\mathbf{Z})$.

Sample distance covariance, for a given ${\mathbf{Z}}_B=[{\mathbf{z}}_1,\dots ,{\mathbf{z}}_B]$, denoting a batch of size B of embeddings from original views, and ${\tilde{\mathbf{Z}}}_B=[{\tilde{\mathbf{z}}}_1,\dots ,{\tilde{\mathbf{z}}}_B]$, referring to a batch of embeddings from augmented views, is defined as

$${\mathrm{dCov}}_B^2({\tilde{\mathbf{Z}}}_B,{\mathbf{Z}}_B):=\frac{1}{B^2}\sum _{j=1}^B\sum _{i=1}^B{A}_{j,i}{C}_{j,i}.$$

(4)

In Equation (4), we use the notations $A_{j,i}:=a_{j,i}-{\overline{a}}_{j·}-{\overline{a}}_{·i}+{\overline{a}}_{··},$$C_{j,i}:=c_{j,i}-{\overline{c}}_{j·}-{\overline{c}}_{·i}+{\overline{c}}_{··}$, where $a_{j,i}=∥{\tilde{\mathbf{Z}}}_{B_j}-{\tilde{\mathbf{Z}}}_{B_i}∥,$$c_{j,i}=∥{\mathbf{Z}}_{B_j}-{\mathbf{Z}}_{B_i}∥$, where $j,i=1,2,\dots ,B$. Finally, sample distance correlation is defined as:

$${\mathrm{dCov}}_B({\tilde{\mathbf{Z}}}_B,{\mathbf{Z}}_B)=\frac{{\mathrm{dCov}}_{\mathrm{B}}^2({\tilde{\mathbf{Z}}}_B,{\mathbf{Z}}_B)}{\sqrt{{\mathrm{dVar}}_B\left({\tilde{\mathbf{Z}}}_B\right){\mathrm{dVar}}_B\left({\mathbf{Z}}_B\right)}},$$

(5)

with ${\mathrm{dVar}}_B\left({\mathbf{Z}}_B\right)={\mathrm{dCov}}_B^2({\mathbf{Z}}_B,{\mathbf{Z}}_B)$.

2.2.2. Dependence between Embeddings of Augmented Data and Multiple Hand-Crafted Representations

The second mutual information $I(\tilde{\mathbf{Z}};{\mathbf{Z}}_k^{\ast })$ between $\tilde{\mathbf{Z}}$ and some invariant feature ${\mathbf{Z}}_k^{\ast }$ deals with vectors of different dimensions. Thus, one can either map these vectors to the same dimension and apply the above arguments, use the Hilbert–Schmidt proxy [25], or proceed with the distance correlation dependence measure for the uniformity of consideration. We focus on the distance correlation case due to its property of handling vectors of different dimensions and its ability to capture higher-order data statistics.

3. Related Work

Pretext task methods. The main idea behind these methods is to design a specific task, a.k.a. pretext task, for the dataset that contains some “labels” of the pretext task without having any access to the labels of the target task. Such pretext tasks include, but are not limited to, applying and predicting parameters of the geometric transformations [26], jigsaw puzzle solving [27], inpainting [28] and colorization [29] of the images, and reversing augmentations. Typically, the pretext task methods have been coupled with other SSL techniques in recent years [30,31,32].

Contrastive methods. Most of the contrastive SSL methods are based on different extensions of the InfoNCE [5] formulation. The InfoNCE method is based on the direct maximization of the mutual information between the input image and its positive pairs via minimization of the contrastive loss. Examples of contrastive methods are SimCLR [33], SwAV [34], and DINO [35].

Clustering methods. Clustering-based SSL methods are based on the idea of assigning cluster labels to the learned representations in an unsupervised manner with some regularization, such as maintaining uniformity of these cluster labels. The DeepCluster [36] method iteratively groups the features from the encoder using the standard k-means clustering and then uses them as an assignment for the supervision to update the weights of the encoder at the next iterations. SwAV [34] and DINO [35] are other notable clustering-based SSL methods that combine contrastive learning and clustering by clustering the data while requiring the same cluster assignment for different views of the same image.

Distillation methods. Distillation-based SSL methods like BYOL [37], SimSiam [6], and others use the teacher–student type of training, where the student network is trained with the gradient descent, while the teacher network is not updated with gradient descent, but rather with an exponential moving-average update or other method. Such a design is used to avoid collapse.

Collapse- preventing methods. Similar to distillation, collapse-preventing methods try to prevent the collapse by the usage special regularization of embeddings. The Barlow Twins [7] method aims to make the covariance matrix of the embeddings to be an identity matrix. This means that each dimension of the embeddings should be decorrelated with all other dimensions. Additionally, the minimum variance of embedding per each dimension in the batch is constrained. The VICReg [11] method extends the Barlow Twins [7] approach by imposing an additional constraint on the distance between the embeddings with and without augmentations.

Masked Image Modeling. Masked Image Modeling (MIM) for self-supervised learning has emerged as a compelling approach, diverging from traditional methods like pretext task, contrastive, or clustering methods. Central to MIM is the principle of intentionally masking portions of an input image and training a model to predict these occluded parts. This process enables the model to learn valuable representations of the data without relying on explicit labels. Unlike contrastive learning methods like SimCLR or SwAV that require negative samples, MIM directly utilizes the spatial coherence of images to enhance the model’s ability to recognize and predict the structure within masked areas. Pioneering examples include the BEiT [8] algorithm, which employs a transformer architecture to predict the masked visual tokens, drawing inspiration from masked language modeling. Another notable implementation is the MAE (Masked Autoencoder) [9], which uses an asymmetric encoder–decoder structure to efficiently reconstruct masked patches. These approaches contrast with distillation methods like BYOL, where a teacher–student model is used, and clustering methods like DeepCluster that focus on feature clustering. MIM’s uniqueness lies in its direct engagement with the raw image data, offering a pathway to learn intricate image features in a self-supervised manner without the need for complex negative sample handling or clustering mechanisms.

Knowledge distillation. Knowledge distillation [38] is a type of model optimization, where a simple small model (student model) is trained to match the bigger complex model (teacher model). There are multiple types of knowledge-distillation schemes: offline distillation [39], online distillation [40], and self-distillation [41]. There are multiple types of knowledge types that are used for distillation: response-based knowledge [39,42], feature-based knowledge [43], and others. We show how our method can be used for offline feature-based knowledge distillation.

4. MV–MR: Detailed Description

4.1. Method

The training objective consists of two parts: (a) ensuring the invariance of the learned representation under the applied augmentations and, simultaneously, (b) imposing constraints on the learned representation. The final loss integrates the loss based on the upper bound of the mutual information and distance correlation.

4.1.1. Training Objectives for the Representation Learning Based on the Mutual Information

We follow the overall idea of ensuring the invariance of learned representation under the family of applied augmentations and we proceed along the line discussed in the previous section. Since both branches have the same dimension D, we proceed with the maximization of the upper bound on the mutual information between these dimensions, as considered in Section 2.2.1.

To train the deterministic encoder $q_{{\varphi }_z}\left(\mathbf{z}\right|\mathbf{x})=\delta (\mathbf{z}-f_{\varphi }\left(\mathbf{x}\right))$, we penalize the embeddings ${\tilde{\mathbf{z}}}_i$ of augmented view ${\tilde{\mathbf{x}}}_i$ to be as close as possible to the embeddings ${\mathbf{z}}_i$ of non-augmented view ${\mathbf{x}}_i$ using the MSE loss between them:

$$d\left({\mathbf{Z}}_B,{\tilde{\mathbf{Z}}}_B\right)=\frac{1}{B}\sum _{i=1}^B{∥{\mathbf{z}}_i-{\tilde{\mathbf{z}}}_i∥}_2^2.$$

(6)

The MSE is frequently used to ensure the similarity between embeddings. It can be demonstrated that this term equates to the conditional entropy term in the mutual information, as specified in Equation (1), assuming Gaussian conditional distribution. At the same time, it can be proven that InfoNCE aims to minimize the negative cross-entropy $h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z})$ while maximizing the entropy $h\left(\mathbf{Z}\right)$ for the entropy-based model parametrization of $p\left(\tilde{\mathbf{z}}\right|\mathbf{z})$, with $p\left(\tilde{\mathbf{z}}\right)={\mathbb{E}}_{p\left(\mathbf{z}\right)}\left[p\left(\tilde{\mathbf{z}}\right|\mathbf{z})\right]$. Hence, the MSE loss is a non-contrastive loss based on $h\left(\tilde{\mathbf{Z}}\right|\mathbf{Z})$, while InfoNCE operates as its contrastive counterpart.

The variance-regularization term corresponding to the entropy term in the mutual information in (1) is used to control the variance of the embeddings. We use a hinge function of the standard deviation of the embeddings along the batch dimension:

$$v\left({\mathbf{Z}}_B\right)=\frac{1}{D}\sum _{d=1}^{D}max\left(0,\gamma -S\left(\mathbf{z}\left[d\right],ϵ\right)\right),$$

(7)

where $\mathbf{z}\left[d\right]$ denotes the $dth$ dimension of $\mathbf{z}$, $\gamma$ is the margin parameter for the standard deviation, $ϵ$ is a small scalar for numerical stability, and S is the standard deviation, defined as

$$S(a,ϵ)=\sqrt{Var\left(a\right)+ϵ}.$$

(8)

We define the loss that ensures the correspondence between the embeddings from the augmented and non-augmented views, i.e., positive pairs. Simultaneously, we bound the variance of both embeddings as follows:

$${\mathcal{L}}_1\left({\varphi }_z\right)=\lambda d\left({\mathbf{Z}}_B,{\tilde{\mathbf{Z}}}_B\right)+\mu \left[v\left({\mathbf{Z}}_B\right)+v\left({\tilde{\mathbf{Z}}}_B\right)\right],$$

(9)

where $\lambda$ and $\mu$ are hyper-parameters controlling the importance of each term in the loss, $\mathbf{z}=f_{{\varphi }_{\mathbf{z}}}\left(\mathbf{x}\right)$ and $\tilde{\mathbf{z}}=f_{{\varphi }_{\mathbf{z}}}\left(\tilde{\mathbf{x}}\right)$. We set $\lambda$ and $\mu$ to 1 in our experiments. This loss is parametrized by the parameters ${\varphi }_z$ of the encoder and projector. It should be pointed out that, for the symmetry, we impose the constraint on the variance for both augmented and non-augmented embeddings.

It is interesting to point out that the obtained result coincides with the loss used in VICReg [11], where its origin was not considered, from the information-theoretic point of view, as the maximization of mutual information between the embeddings. At the same time, it should be noted that there is no constraint on the covariance matrix of embeddings as in the VICReg [11] and Barlow Twins [7] methods.

4.1.2. Training Objectives for Representation Learning Based on Distance Covariance

The distance correlation is used for the dependence maximization between the embedding from the augmented view with the non-augmented one and the set of representations from the hand-crafted functions.

Accordingly, the loss ${\mathcal{L}}_2\left({\varphi }_z\right)$ denotes the minimization formulation of the distance correlation maximization problem between embeddings from augmented and non-augmented views:

$${\mathcal{L}}_2\left({\varphi }_z\right)=\alpha \left[1-{\mathrm{dCor}}_{\mathrm{B}}({\tilde{\mathbf{Z}}}_B,{\mathbf{Z}}_B)\right],$$

(10)

and the loss ${\mathcal{L}}_3\left({\varphi }_z\right)$ denotes the same for the embedding from the augmented view and $kth$ hand-crafted representation (in the case of self-supervised pretraining) or embeddings from pretrained models (in the case of knowledge distillation):

$${\mathcal{L}}_3\left({\varphi }_z\right)=\sum _{k=1}^K{\beta }_k\left[1-{\mathrm{dCor}}_{\mathrm{B}}({\tilde{\mathbf{Z}}}_B,{\mathbf{Z}}_{B_k^{\ast }})\right],$$

(11)

where $\alpha$ and ${\beta }_k$ are hyper-parameters controlling the importance of each term in the loss. In our experiments, we set $\alpha =1$ and ${\beta }_k=1$.

The final loss function is a sum of three losses:

$$\mathcal{L}\left({\varphi }_z\right)={\mathcal{L}}_1\left({\varphi }_z\right)+{\mathcal{L}}_2\left({\varphi }_z\right)+{\mathcal{L}}_3\left({\varphi }_z\right).$$

(12)

5. Applications of the Proposed Model

In this section, we demonstrate the application of the proposed model to (a) self-supervised pretraining of the model for classification and (b) self-supervised model distillation.

5.1. Self-Supervised Pretraining for Classification

The proposed method can be used for efficient self-supervised model pretraining for classification. Once pretrained, the model is finetuned for classification. We report our results on the STL10 [15] and ImageNet-1K [18] datasets on linear evaluation and semi-supervised finetuning, with 1% and 10% of label pipelines in Table 1 and Table 2. In a linear evaluation pipeline, the pretrained encoder is used as is, without further training, while only a one-layer linear classifier is trained with labeled data. In the semi-supervised finetuning pipeline, the classifier head is attached to the pretrained encoder and the full model is finetuned on the labeled data.

5.2. Knowledge Distillation

The proposed method can also be used for efficient knowledge distillation. This is performed by using the pretrained model (teacher) as the feature extractor and computing the distance correlation between the embeddings of the trainable encoder (student) and the pretrained teacher encoder. In practice, this can be used to match the performance of the big pretrained models with smaller models or match the performance of the models that have been trained on proprietary datasets. In contrast to the standard knowledge distillation approaches [39], our approach does not use any labels or require a latent space of the same shape. As a practical example, we demonstrate that, by using the proposed knowledge distillation approach, we are able to match the performance of the CLIP [13] based on ViT-B-16 [14] with 86.2 M parameters pretrained on 400 M images from the LAION-400M [44] dataset, using ResNet50 with only 23.5 M parameters pretrained on the STL10 and ImageNet-1K datasets, as shown in Section 6.2. Then, the lower-complexity distilled model is used for downstream tasks such as classification.

6. Results

In this section, we demonstrate the performance of the proposed method for two downstream tasks: (a) SSL-based classification and (b) knowledge distillation-based classification.

6.1. SSL-Based Classification

We evaluate the representations obtained after pretraining the ResNet50 backbone with MV–MR on the ImageNet-1K and STL10 datasets for 1000 epochs using the loss function described above. The model pretrained on ImageNet-1K is evaluated with a linear protocol and a semi-supervised protocol with 1% and 10% of images labeled.

6.1.1. Evaluation on ImageNet-1K

Linear evaluation protocol: A linear classifier is trained on top of the frozen representations of the ResNet50 [12] pretrained using MV–MR for 100 epochs with the cross-entropy loss.

Semi-supervised evaluation protocol: The pretrained ResNet50 is fine-tuned with a fraction of the ImageNet-1K dataset—1% or 10% of sampled labels for 100 epochs with the cross-entropy loss.

The results on the validation set of ImageNet-1K for linear and semi-supervised evaluation protocols of the model are shown in Table 1. The main advantage of the MV–MR is that it presents a new way to regularize latent space for self-supervised pretraining by using distance correlation between the embeddings from the model and hand-crafted image features. Due to the lack of computational resources, we did not run the parameter optimization for ImageNet-1K pretraining, so we think that the results could be further improved.

Table 1. Classification accuracy. Evaluation on ImageNet-1K. Evaluation of the representations from ResNet50 non-contrastive backbones pretrained with MV–MR on (1) linear evaluation protocol on top of frozen representations from ImageNet; (2) semi-supervised classification on top of fine-tuned representations, with 1% and 10% of ImageNet samples labeled. Top-1 refers to the accuracy of a classifier by determining if the highest-probability prediction is correct, and Top-5 refers to whether the correct answer is among the five highest probability predictions.

Method	Linear		Semi-Supervised
	Top 1	Top 5	Top 1		Top 5
			1%	10%	1%	10%
Supervised	76.5	-	25.4	56.4	48.4	80.4
PIRL [45]	63.6	-	-	-	-	-
SimSiam [6]	71.3	-	-	-	-	-
InfoMin Aug [46]	73.0	91.1	-	-	-	-
OBoW [47]	73.8	-	-	-	-	-
BYOL [37]	74.3	91.6	53.2	68.8	78.4	89.0
Barlow Twins [7]	73.2	91.0	55.0	69.7	79.2	89.3
VICReg [11]	73.2	91.1	54.8	69.5	79.4	89.5
MV–MR (ours)	74.5	92.1	56.1	69.9	79.4	89.5

Table 1. Classification accuracy. Evaluation on ImageNet-1K. Evaluation of the representations from ResNet50 non-contrastive backbones pretrained with MV–MR on (1) linear evaluation protocol on top of frozen representations from ImageNet; (2) semi-supervised classification on top of fine-tuned representations, with 1% and 10% of ImageNet samples labeled. Top-1 refers to the accuracy of a classifier by determining if the highest-probability prediction is correct, and Top-5 refers to whether the correct answer is among the five highest probability predictions.

Method	Linear		Semi-Supervised
	Top 1	Top 5	Top 1		Top 5
			1%	10%	1%	10%
Supervised	76.5	-	25.4	56.4	48.4	80.4
PIRL [45]	63.6	-	-	-	-	-
SimSiam [6]	71.3	-	-	-	-	-
InfoMin Aug [46]	73.0	91.1	-	-	-	-
OBoW [47]	73.8	-	-	-	-	-
BYOL [37]	74.3	91.6	53.2	68.8	78.4	89.0
Barlow Twins [7]	73.2	91.0	55.0	69.7	79.2	89.3
VICReg [11]	73.2	91.1	54.8	69.5	79.4	89.5
MV–MR (ours)	74.5	92.1	56.1	69.9	79.4	89.5

Evaluation of Small Datasets

In this study, we demonstrate the self-supervised learning model performance on small-scale datasets. The model is trained on the STL10 and CIFAR20 [16] datasets with hand-crafted features: (i) flattened original images, (ii) augmented images, (iii) ScatNet features, (iv) HOG features, and (v) LSD features. The proposed model achieves state-of-the-art results in the linear evaluation protocol on the STL10 and CIFAR20 datasets compared to all other self-supervised methods. The results for STL10 are reported in Table 2, and those for CIFAR20 are in

Table 3. Evaluation on CIFAR20. Classification accuracy for the linear evaluation protocol on top of frozen representations from the CIFAR20 dataset.

Method	CIFAR20
IIC [49]	25.7
TSUC [50]	35.3
SCAN [16]	50.7
RUC [51]	54.3
LFER Ensemble [52]	56.1
ScatSimCLR [30]	63.8
MV–MR (ours)	73.2

.

Table 2. Evaluation on STL10. Classification accuracy for the linear evaluation protocol on top of frozen representations from the STL10 dataset.

Method	STL10
ADC [48]	53.0
IIC [49]	61
TSUK [50]	66.5
SCAN [16]	80.9
ScatSimCLR [30]	85.1
RUC [51]	86.7
MV–MR (ours)	89.7

Table 2. Evaluation on STL10. Classification accuracy for the linear evaluation protocol on top of frozen representations from the STL10 dataset.

Method	STL10
ADC [48]	53.0
IIC [49]	61
TSUK [50]	66.5
SCAN [16]	80.9
ScatSimCLR [30]	85.1
RUC [51]	86.7
MV–MR (ours)	89.7

Transfer Learning

To evaluate the pretrained representation of multiclass classification on the VOC07 [53] dataset, we train a linear classifier on top of the frozen representations from the pretrained encoder for 100 epochs. The mAP on the VOC07 dataset is reported in

Table 4. Transfer learning on multiclass classification on the VOC07 [53] dataset. Evaluation of the non-contrastive representations from the pretrained model on multiclass classification using the linear classifier on top of frozen representations. We report mAP.

Method	Linear Classification
	VOC07
Supervised	87.5
PIRL [45]	81.1
BYOL [37]	86.6
OBoW [47]	89.3
Barlow Twins [7]	86.2
VICReg [11]	86.6
MV–MR(ours)	87.1

, along with results from other non-contrastive state-of-the-art SSL methods with a ResNet50 backbone.

6.2. Knowledge Distillation-Based Classification

To evaluate the proposed approach on the knowledge distillation-based classification task, we have used a pretrained CLIP [13] model based on the ViT-B-16 [14] encoder as the teacher and ResNet50 [12] as the student model. The CLIP model is trained based on the contrastive loss between the image and text embeddings. To proceed with the knowledge distillation in the same way as the SSL training, we use the default projector 8192-8192-8192 after the ResNet50 encoder. The pretrained CLIP ViT model uses images of shape $224\times 224\times 3$ as an input and outputs a latent vector of shape 512, as shown in Figure 2. When reporting the results, the teacher model is evaluated using zero-shot evaluation on the ImageNet-1k dataset and a linear evaluation pipeline on other datasets. The student model is evaluated using a linear evaluation pipeline on all datasets.

The goal of the experimental validation is to demonstrate whether the ResNet50 model with 23.5 M parameters trained only on a smaller dataset can provide similar performance to the CLIP based on the ViT model with 86.2 M parameters and trained on 400 M images. It is important to point out that the training is performed without any additional labels, according to the proposed knowledge distillation framework. In

Table 5. Knowledge distillation experiment with CLIP based on ViT-B-16 as teacher model and ResNet50 as a student model.

Approach	Parameters	STL10	ImageNet-1K	CIFAR100
CLIP ViT-B-16
(zero-shot)	86.2 M	-	67.1	-
CLIP ViT-B-16
(linear evaluation)	86.2 M	98.5	77.4	82.2
MV–MR ResNet50
(linear evaluation)	23.5 M	95.6	75.3	78.6

, we report results of knowledge distillation, where CLIP based on ViT-B-16 is used as a teacher model and ResNet50 is used as a student model. The model is trained for only 200 epochs on a single NVIDIA RTX2080ti GPU using the proposed knowledge distillation approach on the STL10, CIFAR100, and ImageNet-1K datasets. The obtained results confirm that the convolutional ResNet50 model with 4× fewer parameters in comparison to the transformer ViT teacher model and trained on a considerably smaller amount of unlabeled data can closely approach the performance of the teacher model without any special labeling, clustering, additional augmentations, or complex contrastive losses. Remarkably, the proposed knowledge distillation largely preserved this performance and achieved 95.6% versus the best SSL MV–MR result of 89.7%, as indicated in Table 2 and Table 5. The proposed distillation method outperforms all other distillation methods on the CIFAR100 dataset: 78.6% vs. current state-of-the-art 78.08% [54]. Thus, both the proposed MV–MR SSL training and knowledge distillation achieve state-of-the-art results on the STL10 and CIFAR100 datasets and demonstrate competitive results for the ImageNet-1K among all non-contrastive and clustering-free SSL methods.

6.3. Ablation Studies

In this subsection, we describe the ablation studies on the proposed losses (

Table 6. Ablation studies on the combination of losses. We check the importance of each loss term for the training of the model. It is shown that using loss terms ${\mathcal{L}}_1$, ${\mathcal{L}}_2$, and ${\mathcal{L}}_3$ provides the best classification performance. Also, we observe a phenomenon in which loss terms ${\mathcal{L}}_2$ and ${\mathcal{L}}_3$ work the best when applied jointly with the loss ${\mathcal{L}}_1$. However, it is interesting to point out that a disjoint usage of these losses does not lead to reasonable performance enhancement. The exact nature of this phenomenon is not completely clear and additional investigation should be performed.

${\mathcal{L}}_{\mathbf{1}}$	${\mathcal{L}}_{\mathbf{2}}$	${\mathcal{L}}_{\mathbf{3}}$	Accuracy
			Top 1	Top 5
1 loss
✓			50.86	93.95
	✓		46.71	92.18
		✓	44.1	92.08
2 losses
✓	✓		50.76	93.83
✓		✓	47.39	92.54
	✓	✓	40.06	89.31
3 losses
✓	✓	✓	69.38	98.85

). In each of the experiments, we use the same training and evaluation setup: dataset—STL10, epochs—100, batch size—64, 16-bit precision, batch accumulation—1 batch. We use a linear evaluation pipeline. We demonstrate the impact of representation learning based on the maximization of the considered upper bound on the mutual information and the maximization of distance covariance in various settings. In this ablation, we show that the best results are achieved when using three loss terms: ${\mathcal{L}}_1$, ${\mathcal{L}}_2$, and ${\mathcal{L}}_3$ (Appendix A).

7. Implementation Details

The architecture of the MV–MR is similar to ones used in other SSL methods such as BarlowTwins [7], VICReg [11], and others. The model $f_{{\varphi }_z}$, shown in Figure 1, consists of two main parts: (i) the encoder, which is used for downstream tasks, and (ii) the projector, which is used for the mapping of encoder outputs to the embeddings used for the training loss functions in (Figure 1). In our experiments, we use standard ResNet50 [12], available in the $torchvision$ library [55], as the encoder and projector, which consists of two linear layers of size 8192, followed by batch normalization, ReLU, and output linear layer.

We use computer vision feature-extraction methods applied to the original data: original RGB image (that is being flattened into a feature vector), ScatNet features of the image [56], randomly augmented images, flattening into a feature vector, histogram of oriented gradients (HOG), and local standard deviation filter (LSD filter) [20].

ScatNet transform: ScatNet [19,56] is a class of Convolutional Neural Networks (CNNs) that have a set of useful properties: (i) deformation stability, (ii) fixed weights, (iii) sparse representations, (iv) interpretable representation.

Randomly augmented image: In our experiments, we have applied the following augmentations to the image: random cropping, horizontal flipping, random color augmentations, grayscale, and Gaussian blur. Then, the image is flattened into a one-dimensional feature vector.

HOG: Histogram of oriented gradients (HOG) [21] is a feature description that is based on the counting of occurrences of gradient orientation in the localized portion of an image.

LSD filter: A local standard deviation filter [20] is a filter that computes a standard deviation in a defined image region over the image. The region is usually of a rectangular shape of size $3\times 3$ or $5\times 5$ pixels.

We use the PyTorch framework [55] for the implementation of the proposed approach. We use ScatNet with the following parameters: $J=2$ and $L=8$. We use the HOG feature extractor with the following parameters: number of bins—24 and pool size—8. We use a kernel of size $3\times 3$ in the STD filter. As augmentations, for both image representation and as the input to the encoder, we use randomly resized cropping; random horizontal flipping with probability $0.5$; random color-jittering augmentation with brightness $0.8$, contrast $0.8$, saturation $0.8$, hue $0.2$, and probability $0.8$; random grayscale with probability $0.2$; and Gaussian blur with a kernel size of $0.1$ of the image size, mean 0, and sigma in the range $[0.1,2]$.

For the losses, the margin parameter $\gamma$ is set to 1, and $ϵ$ is set to $1\times {10}^{-4}$ in (7).

During the self-supervised pretraining experiments that are presented in Table 1 and Table 2, we train models for 1000 epochs, with batch size 256, gradient accumulation every 4 steps, base learning rate $1\times {10}^{-4}$, Adam [57] optimizer, cosine learning rate schedule, and 16-bit precision. During linear evaluation, we train a single-layer linear model for 100 epochs with batch size 256, learning rate $1\times {10}^{-4}$, and Adam optimizer. During semi-supervised evaluation on ImageNet-1K, we train a model for 100 epochs with batch size 128, learning rate $1\times {10}^{-4}$, and Adam optimizer. During the knowledge distillation, we train the model for 200 epochs, with batch size 512, base learning rate $1\times {10}^{-4}$, Adam optimizer, cosine learning rate schedule, and 16-bit precision.

When training, weight parameters $\lambda =1$ and $\mu =1$ in in ${\mathcal{L}}_1$, $\alpha =1$ in ${\mathcal{L}}_2$, and ${\beta }_k=1,k=1\dots K$ in ${\mathcal{L}}_3$.

8. Conclusions

In this paper, we introduce novel self-supervised MV–MR learning and knowledge distillation approaches, which are based on the maximization of several dependency measures between two embeddings obtained from views with and without augmentations and multiple representations extracted from non-augmented views. The proposed methods use an upper bound on mutual information and a distance correlation for the dependence estimation for the representations of different dimensions. We explain the intuition behind the proposed method of upper bound on the mutual information and the usage of distance correlation as a dependence measure. Our method achieves state-of-the-art self-supervised classification on the STL10 and CIFAR20 datasets and comparable state-of-the-art results on ImageNet-1K datasets in linear evaluation and semi-supervised evaluations. We show that ResNet50 pretrained using knowledge distillation on CLIP ViT-B-16 achieves comparable performance with far fewer parameters (23.5 M with ResNet50 vs. 86.2 M parameters with CLIP ViT-B-16) and a relatively small training set on multiple datasets: STL10 and ImageNet-1k. The proposed disillation method also achieves state-of-the-art peformance on the CIFAR100 dataset, 78.6% vs. previous state-of-the-art of 78.08%.

In our paper, we exclusively focus on the image-classification downstream task using ResNet architecture, limiting ourselves to its standard augmentation techniques. Future efforts should extend beyond ResNets to encompass transformers and other advanced deep learning architectures, exploring their applicability not just in classification but also in other vision downstream tasks, such as object-detection and -segmentation tasks, by using the pretrained backbone with a proper head that is finetuned for the selected downstream task. This expansion would allow for a broader range of augmentation strategies, such as Masked Image Modeling (MIM), and provide insights into the performance of different hand-crafted features across various architectures, enhancing the versatility of self-supervised learning approaches in computer vision. The code is available at: github.com/vkinakh/mv-mr.