ProMatch: Semi-Supervised Learning with Prototype Consistency

Abstract

Recent state-of-the-art semi-supervised learning (SSL) methods have made significant advancements by combining consistency-regularization and pseudo-labeling in a joint learning paradigm. The core concept of these methods is to identify consistency targets (pseudo-labels) by selecting predicted distributions with high confidence from weakly augmented unlabeled samples. However, they often face the problem of erroneous high confident pseudo-labels, which can lead to noisy training. This issue arises due to two main reasons: (1) when the model is poorly calibrated, the prediction of a single sample may be overconfident and incorrect, and (2) propagating pseudo-labels from unlabeled samples can result in error accumulation due to the margin between the pseudo-label and the ground-truth label. To address this problem, we propose a novel consistency criterion called Prototype Consistency (PC) to improve the reliability of pseudo-labeling by leveraging the prototype similarities between labeled and unlabeled samples. First, we instantiate semantic-prototypes (centers of embeddings) and prediction-prototypes (centers of predictions) for each category using memory buffers that store the features of labeled examples. Second, for a given unlabeled sample, we determine the most similar semantic-prototype and prediction-prototype by assessing the similarities between the features of the unlabeled sample and the prototypes of the labeled samples. Finally, instead of using the prediction of the unlabeled sample as the pseudo-label, we select the most similar prediction-prototype as the consistency target, as long as the predicted category of the most similar prediction-prototype, the ground-truth category of the most similar semantic-prototype, and the ground-truth category of the most similar prediction-prototype are equivalent. By combining the PC approach with the techniques developed by the MixMatch family, our proposed ProMatch framework demonstrates significant performance improvements compared to previous algorithms on datasets such as CIFAR-10, CIFAR-100, SVHN, and Mini-ImageNet.

1. Introduction

In the past few decades, machine leaning has demonstrated remarkable success across various visual tasks [1,2,3,4,5,6,7,8]. This success can be attributed to advancements in learning algorithms and the availability of extensive labeled datasets. However, in real-world scenarios, the construction of large labeled datasets can be costly and often impractical. Therefore, finding ways to effectively learn from a limited number of labeled data points has become a major concern. This is where semi-supervised learning (SSL) [9,10,11] comes into play. SSL is an important branch of the machine learning theory and its algorithms, which has emerged as a promising solution to address this challenge by leveraging the abundance of unlabeled data. It has proven to be a remarkable achievement in the field.

The goal of SSL is to enhance the generalization performance by leveraging the potential of unlabeled data. One widely accepted assumption, known as the Low-density Separation Assumption [12], posits that the decision boundary should typically reside in low-density regions in order to improve generalization. Building upon this assumption, two prominent paradigms have emerged: pseudo-labeling [11] and consistency regularization [13]. These approaches have gained significant popularity in the field as effective methods for leveraging unlabeled data in the pursuit of a better generalization performance. Consistency-regularization based methods have become widely adopted methods in SSL. These methods aim to maintain the stability of network outputs when presented with noisy inputs [13,14]. However, one limitation of consistency-regularization based methods is their heavy reliance on extensive data augmentations, which may restrict their effectiveness in certain domains such as videos and medical images. On the other hand, pseudo-labeling based methods are alternative approaches that have gained popularity in SSL. These methods select unlabeled samples with high confidence as training targets (pseudo-labels) [11]. One notable advantage of pseudo-labeling based methods is their simplicity, as they do not require multiple data augmentations and can be easily applied to various domains.

In recent trends, a combination of pseudo-labeling and consistency regularization has shown promising results [15,16,17,18]. The underlying idea of these methods is to train a classifier using labeled samples and use the predicted distribution as pseudo-labels for unlabeled samples. These pseudo-labels are typically generated by weakly augmented views [16,19], or by averaging predictions from multiple strongly augmented views [9]. The objective is then constructed by applying the cross-entropy loss between the pseudo-labels and the predictions obtained from different strongly augmented views. It is worth noting that the pseudo-labels are often sharpened or processed using argmax, and each instance is assigned to a specific category to further refine the learning process.

Building upon this concept, several methods have been proposed. MixMatch [9] adopts a sharpened averaged prediction from multiple strongly augmented views as the pseudo-label and incorporates the mix-up trick [15] to enhance the quality of pseudo-labels. ReMixMatch [16] further improves upon this idea by generating pseudo-labels with weakly augmented views. Additionally, it introduces a distribution alignment strategy, which promotes alignment between the pseudo-label distribution and the distribution of ground-truth class labels. FixMatch [19] simplifies these ideas by employing a confidence-based threshold to select high-confidence pseudo-labels. This method has achieved state-of-the-art performance among augmentation anchoring-based methods. SimPLE [20] enhances previous approaches by incorporating a novel unsupervised objective called Pair Loss. The primary objective is to minimize the statistical distance between high-confidence pseudo-labels that surpass a specific similarity threshold. SimMatch [21], on the other hand, simultaneously matches similarity relationships in both semantic and instance spaces for different augmentations. It further interacts between the semantic and instance pseudo-labels by employing a memory buffer for instantiation. These methods represent significant advancements in semi-supervised learning by effectively utilizing pseudo-labels and incorporating various strategies to enhance the performance of the models.

The previously mentioned methods have one thing in common: they estimate the pseudo-label depending upon the high confident predicted distributions from a single unlabeled sample. One major issue with these methods is their heavy reliance on the quality of pseudo-labeling, which limits their effectiveness when pseudo-labeling is unreliable. This problem stems from the following facts: (1) pseudo-labels evaluated from a single sample may be inaccurate, especially when the model is poorly calibrated. Erroneous high confidence predictions from a single sample can result in many incorrect pseudo-labels, leading to noisy training. (2) Propagating pseudo-labels from unlabeled samples is unstable, since there is a margin between the pseudo-label and the ground-truth label. This fact is likely to cause the error accumulation in the pseudo-labeling process.

In response to the aforementioned challenges, we aim to leverage the characteristics of prototypes (i.e., centers of representations) from labeled samples to generate more reliable pseudo-labels. To achieve this, we introduce a novel loss called the Prototype Consistency (PC) Loss that ensures the stability of label propagation and enhances the accuracy of pseudo-labels. By incorporating the PC Loss with the techniques developed by the MixMatch family [9,16,19], we propose the ProMatch algorithm, which offers an effective training framework for semi-supervised learning (SSL). The framework of ProMatch is depicted in Figure 1. Initially, we construct two sets of prototypes from the semantic space and the prediction space using weakly augmented labeled data. Next, for a given unlabeled sample, we generate the pseudo-label by selecting the most similar prediction-prototype based on the PC criterion. The PC criterion requires that the categories of the most similar semantic-prototype and prediction-prototype align with the predicted category of the most similar prediction-prototype. Finally, we formulate the PC Loss by combining the unlabeled data with strongly augmented views, and then integrate supervised and unsupervised losses from MixMatch family methods to establish the ProMatch training framework. This approach allows us to improve the reliability of the pseudo-labeling process and the accuracy of the pseudo-label by leveraging the stability and consistency of prototypes. It is worthy noting that compared to the state-of-the-art SSL methods, the principal characteristics of our proposal lie in the fact that we use prototypes of labeled samples to exploit and propagate the pseudo-labels.

Our contributions can be summarized as follows:

• We introduce a novel loss component called PC Loss, which addresses the limitations of traditional pseudo-labeling methods. By deriving the pseudo-label from the prediction-prototype of labeled data, the PC Loss ensures more precise and stable label propagation in semi-supervised learning.
• By integrating the PC Loss with the techniques employed in the MixMatch family methods, we establish the ProMatch training framework. This framework combines the benefits of PC Loss and the existing approaches, resulting in an improved performance in semi-supervised learning tasks.
• Extensive experimental results demonstrate that ProMatch achieves significant performance gains over previous algorithms on popular benchmark datasets such as CIFAR-10, CIFAR-100, SVHN, and Mini-ImageNet. These results validate the effectiveness and superiority of our proposed approach in the field of SSL.

2. Related Work

2.1. Consistency Regularization

Consistency regularization is a commonly used technique in machine learning to improve the generalization ability and stability of models. Specifically, it often employs input perturbation techniques [10,22]. For example, in image classification, it is common to elastically deform or add noise to an input image, which can dramatically change the pixel content of an image without altering its label. In other words, it can artificially expand the size of a training set by generating a near-infinite stream of new, modified data. Up to now, many methods based on the pseudo-label have been proposed. For instance, in [22], it increases the variability of the data by incorporating stochastic transformations and perturbations in deep semi-supervised learning. Simultaneously, it minimizes the discrepancy between the predictions of unlabeled samples and their true labels. Temporal Ensembling [23] meets the consistency requirements by minimizing the mean square difference between the predicted probability distributions of the two data-augmented views. Mean Teacher [10] further extends this concept by replacing the aggregated predictions with the output of an exponential moving average (EMA) model. In VAT [24], consistency regularization is implemented through the introduction of virtual adversarial loss. It perturbs samples in the input space with a small noise and maximizes the adversarial loss, forcing the model to generate consistent predictions for these perturbed samples. To encapsulate, in semi-supervised learning, a classifier should output the same class distribution for an unlabeled example, whether it was augmented or not. For unlabeled points x, in the simplest case, it is achieved by adding a regularization term to the loss function as follows:

$${∥{\mathrm{p}}_{\mathrm{model}}(y\mid Augment\left(x\right);\theta )-{\mathrm{p}}_{\mathrm{model}}(y\mid Augment\left(x\right);\theta )∥}_2^2.$$

(1)

Note that Augment(x) is a stochastic transformation, so the two terms in Equation (1) are not identical. VAT [24] computes an additive perturbation to apply to the input, which maximally changes the output class distribution. MixMatch [9] utilizes a form of consistency regularization through the use of standard data augmentation for images (random horizontal flips and crops). FixMatch [19] distinguishes between two degrees of data augmentation (weak and strong): weak augmentation uses standard data augmentation, and strong augmentation may include greater random clipping, rotation, scaling, affine transformation, etc.

2.2. Pseudo-Labeling

Pseudo-labels are artificial labels generated by the model itself; they aid the model in learning more robust and generalized representations. However, one should be cautious when using pseudo-labels, as prediction results may include errors or uncertainty, potentially introducing noise. Among pseudo-labeling-based approaches, such as [11], they conduct entropy minimization implicitly by constructing hard (one-hot) labels from high-confidence predictions on unlabeled samples. TSSDL [25] introduces confidence scores which are determined based on the density of a local neighborhood surrounding each unlabeled sample to measure the reliability of pseudo-labels. In [26], it involves training a teacher model on labeled data to generate pseudo-labels for unlabeled data. Then, a noisy student model is trained using the unlabeled data with pseudo-labels. R2-D2 [27] attempts to update the pseudo-labels through an optimization framework. It generates pseudo-labels by the decipher model during the repetitive prediction process on unlabeled data. In summation, appropriate measures can ensure the reliability of the pseudo-labeled data. It encourages us to focus on the sample with a high confidence (low entropy) that is away from the decision boundaries.

2.3. The Combination of Consistency Regularization and Pseudo-Labeling

Some methods [28,29,30,31] propose to integrate both approaches in a unified framework, which is often called the holistic approach. As one of the pioneering works, FixMatch [19] first generates a pseudo-label from the model’s prediction on the weakly-augmented instance and then encourages the prediction from the strongly-augmented instance to follow the pseudo-label. Their success inspired many variants that use, e.g., curriculum learning [30,31]. FlexMatch [30] dynamically adjusts the pre-defined threshold in a class-specific manner based on the estimated learning status of each class, which is determined by the number of confident unlabeled data samples. Dash [31] dynamically selects the unlabeled data whose loss value does not exceed a dynamic threshold at each optimization step to train learning models. These methods show high accuracy, comparable to supervised learning in a fully-labeled setting.

3. Methods

In this section, we will give a detailed explanation of our newly proposed ProMatch framework. In Section 3.1, we will give a brief definition of the semi-supervised learning (SSL) task. Then, we will introduce the generation of prototypes in Section 3.2. Next, we present the Prototype Consistency Loss (PC Loss) in Section 3.3. Finally, we combine the PC Loss into the MixMatch family methods and introduce the total objective in Section 3.4.

3.1. Preliminaries

We consider the K-class semi-supervised classification setting that define both labeled data $\mathcal{X}=\{(x_b,y_b);b\in (1,\dots ,B_x)\}$ and unlabeled data $\mathcal{U}=\{u_b;b\in (1,\dots ,B_u)\}$ to train the model f. The model f is define as $f=f_{\theta }^s\circ f_{\varphi }^p$ which is composed of a semantic encoder $f_{\theta }^s$ followed by a prediction classifier $f_{\varphi }^p$. Note that $\theta$ and $\varphi$ are the set of parameters of $f_{\theta }^s$ and $f_{\varphi }^p$. The weak augmentation (i.e., random crop and flip) and strong augmentation (i.e., RandAugment [32]) are represented by $\omega (·)$ and $\Omega (·)$.

3.2. Prototype Generation

We create a set of semantic-prototypes ${\left\{c_k^s\right\}}_{k=1}^K$ and a set of prediction-prototypes${\left\{c_k^p\right\}}_{k=1}^K$ from $\mathcal{X}$. The process can be seen in Figure 2. In more specific terms, we construct two buffers of memory queues, ${\left\{Q_k^s\right\}}_{k=1}^K$ and ${\left\{Q_k^p\right\}}_{k=1}^K$, where each key corresponds to the class. Both $Q_k^s$ and $Q_k^p$ denote a memory queue for class k with the fixed size $|Q_k^p|$ (which is equivalent to $|Q_k^s|$) and store the feature points from $f_{\theta }^s$ and $f_{\varphi }^p$. Each $c_k^s$ and $c_k^p$ is calculated by averaging the feature points which are in $Q_k^s$ and $Q_k^p$. Meanwhile, we update $Q_k^s$ and $Q_k^p$ for all k at every step by pushing new features from labeled data in the batch and discarding the eldest ones when the queue is full. The prototype represents the center of clustering for each category from labeled data, which shows more stable propagation than a single sample.

3.3. Prototype Consistency Loss

In this section, we propose the PC Loss to help select the most dependable pseudo-label. First, for a given unlabeled sample, we obtain the semantic feature $z_b^s$ and prediction feature $z_b^p$ by the equation $z_b^s=f_{\theta }^s\left(\omega \left(u_b\right)\right)$ and $z_b^p=\left(f_{\varphi }^p\left(z_b^s\right)\right)$ over its weakly augmented version.

Second, we construct two similarity vectors ${\left\{s_i^s\right\}}_{i=1}^K$ and ${\left\{s_i^p\right\}}_{i=1}^K$. Each element $s_i^s$/$s_i^p$ represents the similarity between the semantic/prediction feature and the semantic/prediction prototype of each category, which can be represented as

$$s_i^s=\frac{z_b^s·c_i^s}{|z_b^s|·|c_i^s|}$$

(2)

$$s_i^p={\sqrt{z_b^p}}^{\top }·\sqrt{c_i^p}$$

(3)

In Equation (2), $z_b^s$ is the semantic feature of the weakly augmented unlabeled sample, and $c_i^s$ is the semantic-prototype of the i-th category. $\left\{s_i^s\right\}$ is the i-th element of similarity vector ${\left\{s_i^s\right\}}_{i=1}^K$, which represents the similarity between $z_b^s$ and $c_i^s$. In Equation (3), $z_b^p$ is the prediction feature of the weakly augmented unlabeled sample, $c_i^p$ is the prediction-prototype of the i-th category. $\left\{s_i^p\right\}$ is the i-th element of similarity vector ${\left\{s_i^p\right\}}_{i=1}^K$, which represents the similarity between $z_b^p$ and $c_i^p$. Note that we employ the cosine measure and Bhattacharyya coefficient [33] to calculate the semantic similarity $s_i^s$ and prediction similarity $s_i^p$, respectively.

Third, based on the similarity vectors, we select the most similar semantic-prototype ${\widehat{c}}_i^s$ and most similar prediction-prototype${\widehat{c}}_j^p$ which are regarded as the closest semantic-prototype and prediction-prototype to the semantic and prediction feature, respectively. Where $i=argmax\left({\left\{s_i^s\right\}}_{i=1}^K\right)$ and $j=argmax\left({\left\{s_i^p\right\}}_{i=1}^K\right)$, i and j represent the categories corresponding to the most similar semantic-prototype and most similar prediction-prototype; ${\left\{s_i^s\right\}}_{i=1}^K$ and ${\left\{s_i^p\right\}}_{i=1}^K$ represent the similarity vectors over the semantic and prediction space.

Finally, we formulate the PC loss as

$${\mathcal{L}}_{\mathcal{PC}}=\frac{1}{B_u}\sum _{b=1}^{B_u}\mathbb{1}\left(i=j=argmax\left({\widehat{c}}_j^p\right)\right)\mathcal{H}\left({\widehat{c}}_j^p,q_b^{\left(s\right)}\right)$$

(4)

where $\mathcal{H}(·)$ is the cross-entropy, $\mathbb{1}[·]$ is a filter function, ${\widehat{c}}_j^p$ is the most similar prediction-prototype, $argmax\left({\widehat{c}}_j^p\right)$ returns the category having a maximum prediction probability, and $q_b^{\left(s\right)}$ is the prediction of the unlabeled sample with strong augmentation, i.e., $q_b^{\left(s\right)}=f(\Omega \left(u_b\right))$. Equation (4) is the core of our approach, which highlights the prototype consistency criterion and the method to construct the loss function based on this criterion. Equation (4) demonstrates that the most similar prediction-prototype ${\widehat{c}}_j^p$ can be considered as a pseudo-label only if the following condition is true: the category of the most similar semantic-prototype (i.e., i), the category of the most similar prediction-prototype (i.e., j), and the predicted category of the most similar prediction-prototype (i.e., $argmax\left({\widehat{c}}_j^p\right)$) are equivalent. This condition includes two aspects of constraints: (1) $i=j$ indicates that the features regarding with the unlabeled samples and the prototypes regarding the labeled samples are supported to be consistent both on the semantic and prediction space, ensuring the reliability for the pseudo-labeling process. (2) $j=argmax\left({\widehat{c}}_j^p\right)$ indicates that the ground-truth category and the predicted category of the prediction-prototype should be consistent, guaranteeing the accuracy of the pseudo-label. The details are shown in Figure 3.

3.4. Total Objective

The proposed PC loss is a generic loss which can easily be coupled with other SSL algorithms. In this paper, we combine the PC loss with the MixMatch [9,16,19] family methods and present a ProMatch framework, in which the final objective can be represented as:

$${\mathcal{L}}_{total}={\mathcal{L}}_x+{\lambda }_u{\mathcal{L}}_u+{\lambda }_{\mathcal{PC}}{\mathcal{L}}_{\mathcal{PC}}$$

(5)

In Equation (5), ${\mathcal{L}}_x$ is the supervised loss computed by ${\mathcal{L}}_x=\frac{1}{B_x}{\sum }_{b=1}^{B_x}\mathcal{H}\left(f\left(\omega \left(x_b\right)\right),y_b\right)$. ${\mathcal{L}}_u$ is the unsupervised loss formulated as ${\mathcal{L}}_u=\frac{1}{B_u}{\sum }_{b=1}^{B_u}\mathbb{1}\left(max(z_b^p\right)\ge \tau )H\left(z_b^p,q_b^{\left(s\right)}\right)$. $\mathbb{1}(·>\tau )$ is the filter function for confidence-based thresholding, $\tau$ is the threshold, and $z_b^p=f\left(w\left(u_b\right)\right)$, $q_b^{\left(s\right)}=f\left(\Omega \left(u_b\right)\right)$. ${\mathcal{L}}_{\mathcal{PC}}$ is the PC Loss. ${\lambda }_u$ and ${\lambda }_{\mathcal{PC}}$ are the hyperparameters that control the weights of the two losses.

The concrete algorithm of our method is shown in Algorithm 1.

4. Experiments

This section validates the effectiveness of ProMatch for semi-supervised learning tasks on three popular benchmarks of CIFAR-10 [34], CIFAR-100 [34], SVHN [35], and Mini-ImageNet. First, we introduce the standard datasets in Section 4.1. Second, the baselines related to our task are introduced in Section 4.2. Third, we list the implementation details of the total experiments in Section 4.3. Fourth, we conduct experiments on the standard datasets with different amounts of labeled data in Section 4.4. Finally, we perform ablation experiments on each component. All experiments are based on the PyTorch deep learning framework, implemented on a Linux server configured with Intel(R) Core(TM) i7-10700F CPU @ 2.90 GHz and GeForce RTX 3090. The version of CUDA is 11.2 and the version of CuDNN is 8.

4.1. Datasets

CIFAR-10 [34]: CIFAR-10 is a widely used benchmark dataset in the field of computer vision and machine learning. The dataset consists of 60,000 color images and each image size is 32 × 32 pixels. It covers a total of 10 different categories and each category contains 6000 images. CIFAR-10 is split into two subsets: a training set and a test set. The training set contains 50,000 images and the test set contains 10,000 images. We randomly select 1000 and 4000 samples from the training set as labeled data, while the rest of the training set is used as unlabeled data.

CIFAR-100 [34]: CIFAR-100 is an extension of the CIFAR-10 dataset. It consists of 60,000 color images and each image size is 32 × 32 pixels. These images cover 100 diverse categories with 600 images per class. Similar to CIFAR-10, CIFAR-100 is split into a training set and a test set. The training set contains 50,000 images and each category contains 500 images. The rest of the 10,000 images form the test set and each category contains 100 images. We randomly select 10,000 samples from the training set as labeled data, while the rest of the training set is used as unlabeled data.

SVHN [35]: SVHN consists of the images of house numbers captured from Google Street View. It contains over 600,000 color images. Each image represents a cropped image patch containing a single digit from 0 to 9. SVHN consists of 10 classes. The training set contains 73,257 images and the test set contains 26,032 images. The image size in SVHN is 32 × 32. We randomly selected 1000 and 4000 samples from the training set as labeled data.

Mini-ImageNet: Mini-ImageNet is a sub-dataset of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It is firstly used for few-shot learning. It consists of 100 randomly selected categories from ImageNet-1K, where each category of the training set contains 600 labeled images of size 84 × 84. For the SSL evaluation, we select 500 images from each class to form the training set and the rest of the 100 images are utilized for the test set.

4.2. Baseline

We compare ProMatch with six typical baseline methods.

• VAT [24]: maintains the consistency of the model’s output under adversarial perturbations.
• MeanTeacher [10]: keeps the consistency between model parameters and a moving average teacher model.
• MixMatch [9]: guesses low-entropy labels for data-augmented unlabeled examples and mixes labeled and unlabeled data using Mix-up.
• PLCB [36]: proposes to learn from unlabeled data by generating soft pseudo-labels using the network predictions.
• ReMixMatch [16]: introduces the distribution alignment and augmentation anchoring to upgrade MixMatch.
• FixMatch [19]: simplifies its previous works by introducing a confidence threshold into its unsupervised objective function. For the same unlabeled sample, FixMatch encourages consistency between weak and strong augmented images.
• SimPLE [20]: introduces a similarity threshold and focuses on the similarity among unlabeled samples.
• FlexMatch [30]: proposes Curriculum Pseudo Labeling, a curriculum learning approach to utilize unlabeled samples according to the status of model learning.
• DoubleMatch [37]: combines the pseudo-labeling technique with a self-supervised loss, enabling the model to utilize all unlabeled data in the training process.
• NP-Match [38]: adjusts neural processes (NPs) to semi-supervised learning and proposes an uncertainty-guided skew-geometric JS divergence to replace the original KL divergence in NPs.
• Bad GAN [39]: a generative-model-based SSL method, which is built up upon the assumption that good semi-supervised learning requires a bad generator.
• Triple-GAN [40]: also a generative-model-based SSL method, which is formulated as a three-player minimax game consisting of a generator, a classifier, and a discriminator.

4.3. Implementation Details

For most experiments in ProMatch, Wide ResNet [41] is adopted as the backbone (WRN-28-2 for CIFAR10 and SVHN, WRN-28-8 for CIFAR-100) following [19]. For Mini-ImageNet, we use ResNet-18 as the backbone. We train the model by using SGD [42] with a momentum of 0.9 or AdamW [43]. The initial learning rate is 0.03 with a cosine learning rate decay schedule of $\eta ={\eta }_{0}cos\left(\frac{7\pi s}{16S}\right)$, where ${\eta }_0$ is the initial learning rate, s is the current training step, and S is the total training step. For CIFAR-100 on WRN-28-2 and Mini-ImageNet on ResNet-18, we use AdamW without learning rate scheduling. In addition, we use the exponential moving average (EMA) of the network parameter for evaluation and label guessing. Note that we use an identical set of hyperparameters for all datasets: ${\lambda }_u=1,{\lambda }_{\mathcal{PC}}=1,\tau =0.95,\alpha =0.75$. More specific implementation details are shown in

Table 1. Hyperparameters of the experiments for CIFAR-10, SVHN, CIFAR-100, and Mini-ImagNet, \  represents no use.

	CIFAR-10	SVHN	CIFAR-100	CIFAR-100	Mini-ImageNet
${\tau }_c$	0.95
${\lambda }_u$	1
${\lambda }_{pc}$	1
$\left|Q_k^p\right|$	4
$lr$	0.003			0.02
K	7		4	2	7
T	0.5
$\alpha$	0.75
weight decay	0.0005		0.001	0.04	0.02
batch size	64				16
EMA decay	0.999
backbone network	WRN 28-2		WRN 28-8	WRN 28-2	ResNet 18
optimizer	SGD			AdamW
momentum	0.9			\
lr scheduler	cosine decay			\
lr decay rate	$\frac{7\pi }{16}$			\

.

4.4. Results

For all datasets, we perform labeled and unlabeled set segmentation by randomly sampling an equal number of images from all classes without replacement.

CIFAR-100: We evaluate the performance of ProMatch on CIFAR-100 with 10,000 labels under two different settings. First, we utilize Wide ResNet 28-8 [41] as the backbone and set the number of weak augmentation K to 4. We use the SGD [42] optimizer and set the weight decay to 0.001. As shown in

Table 2. Top-1 test accuracy of the experiments for CIFAR-100. ${}^{*}$: using SimPLE [20]’s implementation. ${}^{†}$: reported in FixMatch [19].

CIFAR-100
Method	10,000 Labels	Backbone
MixMatch [9] ${}^{*}$	64.01%	WRN 28-2
MixMatch Enhanced	67.12%	WRN 28-2
SimPLE [20]	70.82%	WRN 28-2
ProMatch	72.71%	WRN 28-2
MixMatch ${}^{†}$ [9]	71.69%	WRN 28-8
ReMixMatch ${}^{†}$ [16]	76.97%	WRN 28-8
FixMatch [19]	77.40%	WRN 28-8
SimPLE [20]	78.11%	WRN 28-8
FlexMatch [30]	78.10%	WRN 28-8
DoubleMatch [37]	78.78%	WRN 28-8
NP-Match [38]	78.78%	WRN 28-8
ProMatch	78.85%	WRN 28-8

, compared to FixMatch [19], ProMatch achieves an accuracy rate improvement from 77.40% to 78.85%. Next, we evaluate the performance of ProMatch on CIFAR-100 by using the Wide ResNet 28-2 [41] network. We change the optimizer from SGD [42] to AdamW [43]. The different hyperparameters of this experiment are set at $K=2, lr=0.02$, with EMA decay = 0.04. The experimental result shows that ProMatch significantly outperforms the baseline methods by a large margin. Compared to SimPLE [20], our method achieves an accuracy rate improvement from 70.82% to 72.71%. Moreover, ProMatch achieves nearly the best accuracy rate at about 500 epochs. It indicates that our approach is more stable and has a faster convergence rate.

CIFAR-10: We utilize the Wide ResNet 28-2 [41] network to evaluate the accuracy of ProMatch on CIFAR-10 with 1000 and 4000 labels. We use SGD [42] with a momentum of 0.9 as the optimizer. As shown in

Table 3. Top-1 test accuracy rate of the experiments for CIFAR-10 and SVHN. Bad GAN and Triple-GAN-V2 use CNN-13 as the backbone, while others use Wide ResNet 28-2. ${}^{†}$: The accuracy is reported in ReMixMatch [16] and using its own implementation. ${}^{‡}$: Fully supervised baseline using all the labels and simple augmentation (flip-and-crop).

	CIFAR-10		SVHN
Method	1000 Labels	4000 Labels	1000 Labels	4000 Labels
VAT ${}^{†}$ [24]	81.36%	88.95%	94.02%	95.80%
MeanTeacher ${}^{†}$ [10]	82.68%	89.64%	96.25%	96.61%
Bad GAN [39]	79.37%	85.59%	95.75%	96.03%
MixMatch [9]	92.25%	93.76%	96.73%	97.11%
ReMixMatch [16]	94.27%	94.86%	97.17%	97.58%
Triple-GAN-V2 [40]	85.00%	89.99%	96.55%	96.92%
FixMatch [19]	−	95.69%	97.64%	−
SimPLE [20]	94.84%	94.95%	97.54%	97.31%
FlexMatch [30]	-	95.81%	93.28%	-
DoubleMatch [37]	-	95.35%	97.90%	-
ProMatch	95.01%	95.83%	97.79%	97.88%
Fully Supervised ${}^{†‡}$	96.23%		98.17%

, the accuracy rate of ProMatch reaches 95.01% and 95.83% on 1000 and 4000 labeled samples, respectively. In contrast, the accuracy rates of Bad GAN are 79.37% and 85.59% on 1000 and 4000 labeled samples, while the accuracy rates of Triple-GAN-V2 are 85.00% and 89.99% on 1000 and 4000 labeled samples. It indicates that our method significantly outperforms the classical generative-model-based SSL methods. Moreover, the performance of ProMatch is superior to the recent state-of-the-art methods, such as SimPLE, FlexMatch, and DoubleMatch. Note that compared to the increment of accuracy for CIFAR-100, ProMatch increases at a relatively small accuracy rate for CIFAR-10. This is because CIFAR-10 is a comparatively simple dataset compared to CIFAR-100. Many mainstream semi-supervised algorithms such as FixMatch and SimPLE have already achieved desirable results on CIFAR-10. The prediction accuracy rates of these methods using partial labeled samples (i.e., semi-supervised learning) are pretty close to the accuracy rates of the methods using the whole labeled samples (i.e., supervised learning). This means that the advancement space of the semi-supervised learning methods for the CIFAR-10 dataset is limited, since the accuracy rate derived by the fully supervised benchmark can be considered as the theoretical upper bound for the performance of the semi-supervised learning. All the same, the accuracy rate obtained by our proposal has reached 95.83% under 4000 labeled samples of CIFAR-10, which is not only the highest among the compared methods and but is also very near the upper bound.

SVHN: We use the Wide ResNet 28-2 [41] network and set the same hyperparameters as CIFAR-10 to evaluate the performance of ProMatch on SVHN with 1000 and 4000 labels. As shown in Table 3, ProMatch achieves 97.79% and 97.88% Top-1 accuracy rates for 1000 and 4000 labeled samples, respectively. Compared to Bad GAN and Triple-GAN-V2, our method achieves an accuracy rate improvement distinctly. Despite the outstanding performance of the baseline methods on the SVHN dataset, ProMatch achieves a higher accuracy rate compared to them.

Mini-ImageNet: We use the ResNet-18 network to assess the performance of ProMatch on a more complex dataset Mini-ImagNet. The number of labeled samples is 4000. We set the learning rate and weight decay to 0.02. Moreover, to elucidate the effect of erroneous high confident pseudo-labels, we recorded the error rates of pseudo-labels whose confidence threshold is $\tau$. The results are shown in

Table 4. Experimental results on Mini-ImageNet with 4000 labels. We report the Top-1 test accuracy rate and the error rate of pseudo-labels whose confidence $\ge \tau$.

Mini-ImageNet
Method	4000 Labels	Error Rate of Pseudo-Labels
Mean Teacher [10]	27.49%	38.87%
PLCB [36]	43.51%	22.63%
MixMatch [9]	48.46%	18.32%
FixMatch [19]	50.21%	14.77%
SimPLE [20]	49.39%	15.35%
ProMatch	50.83%	13.21%

. From the table, we can observe that the error rates of pseudo-labels display inverse correlations with the test accuracy rates of the semi-supervised learning model. Among the comparisons, our method has the lowest error rate $13.21\%$ of pseudo-labels and the highest accuracy rate $50.83\%$ of the test dataset, indicating that the proposed ProMatch can effectively increase the accuracy of pseudo-labels and boost the performance of semi-supervised learning. Note that compared to SimPLE, the accuracy rate of our method increases by $1.44\%$, which demonstrates the effectiveness of our proposal for the complex dataset.

4.5. Ablation Study

We present extensive ablation studies to verify the effect of different components. We conduct these ablation studies on CIFAR-10 with 10,000 labels and use Wide ResNet 28-2 as the backbone. The reasons are as follows: (1) CIFAR-100 covers a certain number of classes and the image size is small; (2) the Wide ResNet 28-2 network is fast enough to train compared to Wide ResNet-28-8.

The Effect of the Weak Augmentation Number K. We exhibit the result of the predicted accuracy rate under different numbers of weak augmentation. The results are shown in

Table 5. Ablation experiment on CIFAR-100. All experiments use WRN 28-2. w/o represents without PC Loss.

Ablations: CIFAR-100
Ablation	Augmentation Type	${\mathit{\lambda }}_{\mathcal{PC}}$	${\mathit{\tau }}_{\mathit{c}}$	$\mathit{K}$	$\left|{\mathit{Q}}_{\mathit{k}}^{\mathit{p}}\right|$	Accuracy
ProMatch	RandAugment	1	0.95	2	4	72.71%
ProMatch	RandAugment	1	0.95	7	4	73.38%
w/o ${\mathcal{L}}_{\mathcal{PC}}$	RandAugment	0	0.95	2	4	69.07%
w/o ${\mathcal{L}}_{\mathcal{PC}}$	RandAugment	0	0.95	7	4	69.94%
w/o RandAugment	fixed	1	0.95	2	4	67.66%
w/o RandAugment, w/o ${\mathcal{L}}_{\mathcal{PC}}$	fixed	0	0.95	2	4	67.41%
$\tau$ = 0	RandAugment	1	0	2	4	71.54%
$\tau$ = 0.5	RandAugment	1	0.5	2	4	71.53%
$\tau$ = 0.75	RandAugment	1	0.75	2	4	71.89%
${\lambda }_{\mathcal{PC}}$ = 0.5	RandAugment	0.5	0.95	2	4	72.55%
${\lambda }_{\mathcal{PC}}$ = 2	RandAugment	2	0.95	2	4	71.81%
${\lambda }_{\mathcal{PC}}$ = 3	RandAugment	3	0.95	2	4	72.42%
${\lambda }_{\mathcal{PC}}$ = 4	RandAugment	4	0.95	2	4	72.76%
$\left|Q_k^p\right|$ = 3	RandAugment	1	0.95	2	3	71.83%
$\left|Q_k^p\right|$ = 5	RandAugment	1	0.95	2	5	72.01%
$\left|Q_k^p\right|$ = 10	RandAugment	1	0.95	2	10	71.82%
$\left|Q_k^p\right|$ = 20	RandAugment	1	0.95	2	20	72.13%

. As shown from the first row and the second row, we compare the accuracy rate by varying the number of K in ProMatch. Compared to ProMatch on $K=2$, ProMatch on $K=7$ achieves an accuracy rate improvement from 72.71% to 73.38%. As shown from the third row and the fourth row, we change the number of K in ProMatch without the PC Loss. ProMatch on $K=7$ and without the PC Loss reaches an accuracy rate of 69.94%, while the accuracy rate of ProMatch on $K=2$ and without the PC Loss is only 69.07%. These experimental results demonstrate that increasing K from 2 to 7 can improve the accuracy rate.

The Effect of PC Loss. To demonstrate the effect of the PC Loss, we conducted three experiments. The results are shown in Table 5. First, as shown from the first row and the third row, we validate the performance of ProMatch and ProMatch without the PC Loss. ProMatch without the PC Loss only reaches an accuracy of 69.07%. Compared to ProMatch, the accuracy rate of ProMatch without the PC Loss decreases by 3.61%. Second, as shown from the second row and the fourth row, we increase the number of K from 2 to 7. In this case, we compare the accuracy rate between ProMatch and ProMatch without the PC Loss. The accuracy rate of ProMatch without the PC Loss is 69.94%, which is lower than ProMatch by 3.44%. Finally, as shown from the fifth row and the sixth row, we change the augmentation type from RandAugment [32] to the fixed augmentation. Based on this, we compare the accuracy rate between ProMatch and ProMatch without the PC Loss. ProMatch reaches an accuracy rate of 67.66%. The accuracy rate of ProMatch without the PC Loss is 67.41%. We find that the accuracy rate of ProMatch is improved as relatively small.

To further demonstrate the effect of the PC Loss, we evaluate the recall rate, Top-1 accuracy rate, and the F1 score on CIFAR-100. Due to the large number of categories in CIFAR-100, we randomly select seven categories to present their results. As shown in Figure 4a,b, ProMatch improves the precision of the classes while achieving an enhanced recall rate. As shown in Figure 4c, ProMatch reaches a higher F1 Score, which further demonstrates that the introduction of the PC Loss has improved the algorithm’s overall performance. Besides, we visualize the t-SNE of the features on ProMatch without the PC Loss and ProMatch. As shown in Figure 5, ProMatch exhibits a better clustering performance and enhances the accuracy of classification. Finally, we visualize the Top-1 and Top-5 accuracy rates on the test dataset and validation dataset. As shown in Figure 6, ProMatch exhibits a rapid convergence speed on both the test dataset and validation dataset. Compared to ProMatch without the PC Loss, ProMatch obtains a high accuracy rate in a short period of time.

Algorithm 1 ProMatch algorithm.
1: Input: A batch of labeled data $\mathcal{X}={\left\{\left(x_b,y_b\right)\right\}}_{b=1}^{B_x}$ and unlabeled data $\mathcal{U}{\left\{u_b\right\}}_{b=1}^{B_u}$, network for semantic encoder $f_{\theta }^s\left(·\right)$, and prediction classifier $f_{\varphi }^p\left(·\right)$, buffer of memory queue ${\left\{Q_k^s\right\}}_{k=1}^K$ and ${\left\{Q_k^p\right\}}_{k=1}^K$. Weak-augmentation $\omega \left(·\right)$, strong-augmentation $\Omega \left(·\right)$, Cosine similarity function $\sigma \left(·\right)$, Bhattacharyya coefficient $bha\left(·\right)$, the number of weak-augmentation K, cross-entropy loss function $\mathcal{H}(·)$.
2: Prototype Generation:
3:    $f_{\theta }^s\left(x_b\right)⟹Q_k^s⟹{\left\{c_k^s\right\}}_{k=1}^K$	▹ Generate semantic-prototype and store by label k.
4:    $f_{\theta }^s\left(f_{\varphi }^p\left(x_b\right)\right)⟹Q_k^p⟹{\left\{c_k^p\right\}}_{k=1}^K$	▹ Generate prediction-prototype and store by label k.
5: Prototype Consistency:
6: for$b=1$ to $B_u$ do
7:     ${\tilde{u}}_b=\omega \left(u_b\right)$	▹ Apply weak data augmentation to $u_b$
8:     ${\widehat{u}}_b=\Omega \left(u_b\right)$	▹ Apply strong data augmentation to $u_b$
9:     $z_b^s=f_{\theta }^s\left({\tilde{u}}_b\right)$	▹ Compute semantic feature across ${\tilde{u}}_b$ using EMA
10:     $z_b^p=\frac{1}{K}{\sum }_{k=1}^K{f}_{\varphi }^p\left(z_b^s\right)$	▹ Compute average predictions across ${\tilde{u}}_b$ using EMA
11:     $z_b^p=Sharpen(z_b^p,T)$	▹ Apply temperature sharpening to the average prediction
12:     $q_b^{\left(s\right)}=f_{\varphi }^p\left(f_{\theta }^s\left({\widehat{u}}_b\right)\right)$	▹ Compute predictions across ${\widehat{u}}_b$ using EMA
13:     ${\left\{s_i^s\right\}}_{i=1}^K=\sigma (z_b^s,{\left\{c_k^s\right\}}_{k=1}^K)$	▹ Similarity between $z_b^s$ and ${\left\{c_k^s\right\}}_{k=1}^K$
14:     ${\left\{s_i^p\right\}}_{i=1}^K=bha(z_b^p,{\left\{c_k^p\right\}}_{k=1}^K)$	▹ Similarity between $z_b^p$ and ${\left\{c_k^s\right\}}_{k=1}^K$
15: end for
16:    ${\widehat{c}}_i^s=c_{argmax\left({\left\{s_i^s\right\}}_{i=1}^K\right)}^s$	▹ Obtain the most similar semantic-prototype
17:    ${\widehat{c}}_j^p=c_{argmax\left({\left\{s_i^p\right\}}_{i=1}^K\right)}^p$	▹ Obtain the most similar prediction-prototype
18: Loss:
19:    ${\mathcal{L}}_x=\frac{1}{B_x}{\sum }_{b=1}^{B_x}\mathcal{H}(y_b,f\left(\omega \left(x_b\right)\right))$
20:    ${\mathcal{L}}_u=\frac{1}{B_u}{\sum }_{b=1}^{B_u}\mathbb{1}\left(max\left(z_b^p\right)\ge \tau \right)·\mathcal{H}\left(z_b^p,q_b^{\left(s\right)}\right)$
21:    ${\mathcal{L}}_{\mathcal{PC}}=\frac{1}{B_u}{\sum }_{b=1}^{B_u}\mathbb{1}\left(i=j=argmax\left({\widehat{c}}_j^p\right)\right)\mathcal{H}\left({\widehat{c}}_j^p,q_b^{\left(s\right)}\right)$
22: return    ${\mathcal{L}}_x+{\lambda }_u{\mathcal{L}}_u+{\lambda }_{\mathcal{PC}}{\mathcal{L}}_{\mathcal{PC}}$	▹ Total loss

The Effect of Different Augmentation Strategy. To verify the effect of the different augmentation strategies, we simply replace RandAugment [32] with the fixed augmentation. The results are shown in Table 5. First, as shown from the first row and the fifth row, we compare the accuracy rate between ProMatch with fixed augmentation and ProMatch with RandAugment. ProMatch with fixed augmentation only reaches an accuracy rate of 67.66%. The accuracy rate is much lower than ProMatch with RandAugment. Second, as shown from the third row and the sixth row, we evaluate the effect of different augmentation strategies on ProMatch without the PC Loss. The accuracy rate of ProMatch with RandAugment and without the PC Loss is 69.07%, while ProMatch with fixed augmentation and without the PC Loss only achieves a 67.41% accuracy rate. These results demonstrate that RandAugment [32] improves the model’s robustness to different samples and alleviates the overfitting issues.

The Effect of Hyperparameters. In this section, we evaluate the effect of several hyperparameters such as the confidence threshold $\tau$, PC loss coefficient ${\lambda }_{\mathcal{PC}}$, and memory buffer size $\left|Q_k^p\right|$ under different values. The results are shown in Table 5 and Figure 7.

• The first row and 7th–9th row of Table 5 show the accuracy rates of our proposal ProMatch under different values of the confidence threshold $\tau =0,0.5,0.75,0.95$. We also plot these results in Figure 7a. From these results, we can observe that the accuracy rates of our method have a positive correlation with the confidence threshold. Specifically, ProMatch achieves the optimal accuracy rate of 72.71% at the confidence threshold $\tau =0.95$.
• The first row and 11th–13th row of Table 5 show the accuracy rates of ProMatch under different PC loss coefficients ${\lambda }_{\mathcal{PC}}=0.5,1,2,3,4$. We also plot these results in Figure 7b. From these results, we can observe that ProMatch achieves the optimal accuracy rate at ${\lambda }_{\mathcal{PC}}=4$.
• The first row and 14th–17th row of Table 5 show the accuracy rates of ProMatch under different buffer sizes $\left|Q_k^p\right|=3,4,5,10,20$. We also plot these results in Figure 7c. These results demonstrates that ProMatch achieves the optimal accuracy rate at $\left|Q_k^p\right|=4$.

5. Conclusions

This paper proposed a new framework ProMatch to improve the performance of SSL. ProMatch enhances the reliability and accuracy of pseudo-labeling by considering the PC, which maintains the consistency among the ground-truth category of the semantic-prototype, the ground-truth category of the prediction-prototype, and the predicted category of the prediction-prototype. According to the PC criterion, we formulated a new PC loss by selecting the most-similar prediction-prototype that is closest to the predicted distribution for the unlabeled sample as the pseudo-label. Extensive experiments are conducted on CIFAR-10, CIFAR-100, SVHN, and Mini-ImageNet datasets. The results demonstrate the effectiveness of our proposal. In addition, it should be noted that the performance of ProMatch is mainly dependent upon the reliability of the prototypes regarding the labeled samples. Thus, our proposal may not be adept at dealing with some specific tasks such as the imbalanced category task and fine-grained recognition task, where the prototypes of different categories are extremely indistinct. In our future work, we will devote to making our method more robust against various specific circumstances.