Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning

1 Introduction

Over the past decade, Semi-Supervised Learning (SSL) algorithms [Zhu and Goldberg, 2009] have demonstrated remarkable performance across multiple tasks, even when presented with a meagre number of labeled training samples [Jia et al., 2023]. These algorithms delve into the underlying data distribution by harnessing numerous unlabeled samples. Among the representative methods employed for this purpose are pseudo-labeling [Lee, 2013] and consistency regularization [Laine and Aila, 2016; Sajjadi et al., 2016]. Pseudolabeling involves utilizing model predictions as target labels, while consistency regularization encourages similar predictions for distinct views of an unlabeled sample. However, the majority of current approaches operate under the assumption that unlabeled data exclusively comprises samples belonging to seen classes, as observed within the labeled data [Bendale and Boult, 2015; Boult et al., 2019]. In contrast, the presence of samples from novel classes in the unlabeled data is common, as it is challenging for human annotators to discern such instances amidst an extensive pool of unlabeled samples [Oliver et al., 2018].

† Corresponding author

To aid this challenge, Open-World Semi-Supervised Learning, denoted as OpenSSL, has gained recent attention, leading to the proposition of several effective methodologies [Cao et al., 2022; Guo et al., 2022; Liu et al., 2023]. Figure 1 demonstrates the problem setting of OpenSSL as an intuitive example. To tackle this issue, extant methods adopt a two-pronged strategy. On one front, they endeavour to identify unlabeled samples pertaining to seen classes and allocate pseudo-labels accordingly. On the other front, they automatically cluster unlabeled samples belonging to novel categories. Notably, OpenSSL shares an affinity with Novel Class Discovery (NCD) [Han et al., 2019; Han et al., 2020], particularly concerning the clustering of novel class samples. However, NCD methodologies presuppose that unlabeled samples originate exclusively from novel classes. OpenSSL relaxes this assumption to mirror real-world scenarios more accurately. Evidently, the central challenge of effectively clustering novel class samples hinges upon the acquisition of discriminative feature representations, given the absence of supervisory information. To mitigate this quandary, existing methods harness self-supervised learning paradigms (e.g., SimCLR [Chen et al., 2020a]) which circumvent the need for labeled data during the training of feature extractors within deep neural networks. Subsequently, a linear classifier is cultivated by optimizing the cross-entropy loss for labeled data, in conjunction with specifically tailored unsupervised objectives for the unlabeled counterpart. Widely employed unsupervised objectives include entropy regularization and pairwise loss, both of which effectively enhance performance.

This paper introduces a novel OpenSSL algorithm. An initial observation reveals that the model exhibits faster learning of seen classes compared to novel classes. This discrepancy is intuitive because of accurate supervision within labeled data for seen classes, whereas novel classes are learned through unsupervised means. Figure 2 depicts a demonstration of the learning speed discrepancy between seen and novel classes. Motivated by this intrinsic problem, we propose an adaptive distribution-aware margin mechanism, designed to steer the model's attention towards novel class learning. Notably, this margin diminishes as the model's predicted class distribution approaches the underlying (class-balanced) distribution. To learn robust representations and facilitate the clustering of novel classes, we introduce pseudo-label contrastive clustering. This technique aggregates unlabeled samples sharing the same class, guided by model predictions. Importantly, we exploit multiple positive and negative pairs as supervisory signals, in contrast to the reliance on a single positive pair as seen in previous works. For unlabeled samples exhibiting low confidence, we integrate unsupervised contrastive learning to facilitate the acquisition of informative representations. This unsupervised contrastive objective operates as a complement to the pseudo-label contrastive clustering. Combining the aforementioned modules, we present, LPS, to address the OpenSSL challenge. Figure 2 showcases the efficacy of LPS compared with existing state-of-the-art approaches. Notably, we reveal that the conventional practice of freezing the feature extractor, previously trained via self-supervised learning in prior research, falls short of optimal.

In summary, our main contributions are:

• We propose a novel and simple method, LPS, to effectively synchronize the learning pace of seen and novel

classes for open-world semi-supervised learning.

• We conduct extensive experiments to verify the effectiveness of the proposed method against the previous state-of-the-art. Particularly, LPS achieves over 3% average increase of accuracy on the ImageNet dataset.
• We examine the effectiveness of the key components of the proposed method. Different from previous works, we discover that fine-tuning the pre-trained backbone allows the model to learn more useful features, which can significantly improve the performance.

2 RelatedWork

Semi-Supervised Learning. Within the realm of SSL, pseudo-labeling [Lee, 2013] and consistency regularization [Laine and Aila, 2016; Sajjadi et al., 2016; Wei et al., 2022a; Wei and Gan, 2023] are two widely used techniques. Pseudolabeling converts model predictions on unlabeled samples into either soft labels or hard labels, subsequently employed as target labels. Consistency regularization strives to ensure model outputs exhibit a high degree of consistency when applied to perturbed samples. Recent advancements [Berthelot et al., 2019; Sohn et al., 2020; Xu et al., 2021] combine pseudo-labeling with consistency regularization to yield further performance enhancements. In addition to the above techniques, the application of contrastive learning into SSL has also received substantial interest. For example, TCL [Singh et al., 2021] introduces contrastive learning as a tool to enhance representation learning. TCL maximizes agreement between different views of the same sample while minimizing agreement for distinct samples. In consonance with this paradigm, we design a new complementary contrastive loss to explore the consistency of all samples effectively.

Novel Class Discovery. The setting of NCD aligns closely with the scenario investigated in this paper. NCD assumes a scenario where the labeled data consists of samples of seen classes, while the unlabeled data exclusively comprises samples of novel classes. [Han et al., 2019] initially raised the NCD problem. Subsequent research such as [Han et al., 2020; Zhong et al., 2021a; Zhong et al., 2021b] predominantly adopted multi-stage training strategies. The underlying principle is capturing comprehensive high-level semantic information from labeled data, subsequently propagated to unlabeled counterparts. The majority of NCD methods involve preliminary model pre-training, wherein several objective functions are invoked to minimize inter-sample distances for each class. However, in real-world scenarios, the assumption of unlabeled data solely comprising novel classes is unrealistic [Wei et al., 2022b], as seen classes also significantly populate the unlabeled dataset. Our experimentation reveals that NCD algorithms struggle to match the performance of other leading methods in the context of OpenSSL.

Open-World Semi-Supervised Learning. While conventional SSL methods operate under the assumption of labeled and unlabeled data being associated with a predefined set of classes, recent advancements [Oliver et al., 2018; Chen et al., 2020b; Guo et al., 2020; Saito et al., 2021] challenge this notion and acknowledge the potential emergence of novel classes within the unlabeled data. Further, ORCA [Cao et al., 2022] delves into the OpenSSL problem and attempts to cluster novel classes from unlabeled data. Both ORCA and NACH [Guo et al., 2022] notice the discrepancy in learning pace between seen and novel classes. They propose to use an entropy regularizer and pair-wise loss to ameliorate this issue. More recently, OpenNCD [Liu et al., 2023] introduces prototypes-based contrastive learning for recognizing seen classes and discovering novel classes. This paper introduces an adaptive margin loss and pseudo-label contrastive clustering mechanism to synchronize the learning pace and enhance novel class discovery.

3 The ProposedMethod

Notations. The training dataset is composed of the labeled data Dl = {(xi, yi)}ni=1 and the unlabeled data Du = {xi}n+m i=n+1 . Within the context of OpenSSL, the classes in Dl are designated as seen classes, constituting the set denoted as Cs. The scenario of interest acknowledges a distribution mismatch, leading to Du comprising instances from both seen and novel classes. The collection of these novel classes is represented as Cn. Additionally, we adopt the premise of a known number of novel classes akin to prior OpenSSL methodologies [Cao et al., 2022; Guo et al., 2022; Liu et al., 2023]. The goal of OpenSSL is to classify samples originating from the Cs and cluster samples emanating from the Cn.

Overview. The fundamental challenge in OpenSSL arises from the pronounced discrepancy in learning paces between seen and novel classes, primarily due to the precise supervisory guidance for seen classes. This discrepancy results in a bias towards seen classes in the model's predictions, adversely impacting both the accurate classification of seen class samples and the effective clustering of novel class samples. To circumvent this challenge, we introduce Learning Pace Synchronization (LPS), a methodology with adaptive synchronizing loss and pseudo-label contrastive clustering as in Figure 3. The adaptive synchronizing loss aims to achieve a balance between the learning pace of seen and novel classes, and the pseudo-label contrastive clustering exploits pseudolabels to group unlabeled data from the same class together in the output space.

3.1 Adaptive Synchronizing

To start with, we describe the proposed adaptive marginal loss which regularizes the learning pace of seen classes to synchronize the learning pace of the model. Conventionally, the margin is defined as the minimum distance of the data to the classification boundary. For a sample (x, y), we have:

Instead of employing a fixed margin, LDAM [Cao et al., 2020] introduces a class-specific margin, where the margin between rare classes and other classes is larger than the margin for frequent classes, for tackling class-imbalanced data. Specifically, it sets the margin of class j as:

The constant C controls the intensity and nj denotes the frequency of class j in the training data. Motivated by this, we propose a new variant of the margin loss to synchronize the learning pace of seen and novel classes. We apply adaptive margin loss to demand a larger margin between the novel and other classes, so that scores for seen classes, towards which the model highly biased, do not overwhelm the novel classes. For each sample (x, y), the adaptive margin loss is defined as follows:

In this formulation, K represents the total number of classes, zj signifies the model output for the j-th class, z = f(x; θ), and ̂π denotes the estimated class distribution by the model. Additionally, we introduce an approximation of the true class distribution π, which is naturally inaccessible during training. In line with prior studies, we assume a uniform distribution for π, leaving the exploration of arbitrary distributions for future investigations. The hyper-parameter C is introduced to control the maximum margin, and we empirically set C = 10 across all experiments. We conduct a series of studies on the value of C in the supplementary material.

For the sake of simplicity, we assume that the mini-batch is comprised of labeled data Bl and unlabeled data Bu. Given that the computation of Eq. (3) relies on the class distribution, we proceed to estimate the complete class distribution through labeled data and unlabeled data exhibiting high predictive confidence. Specifically, we endeavour to achieve this estimation through:

Here, ̂ y = softmax(z). In view of the tendency for novel class samples to exhibit underconfidence, we empirically introduce a progressively evolving confidence threshold λnovel = 0.4 + 0.4 × tT , where t and T signify the current training iteration and the total training iterations, respectively. For seen classes, a fixed confidence threshold λseen = 0.95 is employed.

Given that the estimated class distribution ̂π mirrors the model's confidence in class predictions, we harness this insight to regulate the learning pace of both seen and novel classes. Notably, in the early training phases, the model is inclined towards seen classes, with the logit adjustment term ∆j assuming a larger negative margin for seen classes, thereby attenuating their learning pace. As training progresses, the model attains a more balanced capability across both seen and novel classes, as reflected by a diminishing Kullback-Leibler (KL) divergence between ̂π and π.

In summary, the adaptive margin loss LAM for both labeled and pseudo-labeled data is defined as follows:

In this context, zw and zs correspond to the output logits stemming from the weak and strong augmented versions of sample x, respectively. The symbol | · | denotes the set cardinality operation. Additionally, we utilize ̂ p = argmax(softmax(zw )) to represent the pseudo-label associated with the sample.

Distinctions and Connections with Alternatives. It is worth noting that the concept of adaptive margin has been used in prior literature [Li et al., 2020; Ha and Blanz, 2021; Cao et al., 2022]. Different from LPS, [Li et al., 2020] leverages the semantic similarities between classes to generate adaptive margins with the motivation to separate similar classes in the embedding space, and [Ha and Blanz, 2021] utilizes the ground-truth distance between different samples to generate adaptive margins with the motivation to adapt to rating datasets. In OpenSSL, ORCA [Cao et al., 2022] also integrates an adaptive margin mechanism based on the model's predictive uncertainty, which can only equally suppress the learning pace of seen classes. However, there are still differences in the learning paces of different classes among seen classes. Our proposed adaptive margin is based on the current estimated distribution to reflect the learning pace of different classes, which offers increased flexibility for regulating the learning pace across classes by generating the class-specific negative margin. Furthermore, the inclusion of the KL divergence term effectively guards against the model converging to a trivial solution where all samples are arbitrarily assigned to a single class.

3.2 Pseudo-Label Contrastive Clustering

The basic idea of discovering novel classes is to explore the correlations between different samples and cluster them into several groups. Prior OpenSSL approaches often transform the clustering task into a pairwise similarity prediction task, wherein a modified form of binary cross-entropy loss is optimized. Different from existing works, we introduce a new clustering method to fully exploit reliable model predictions as supervisory signals.

Our approach involves the construction of a multi-viewed mini-batch by using weak and strong augmentations. Within each mini-batch, we group the labeled and confident unlabeled samples, which is denoted as Bl′ . Concurrently, unlabeled samples exhibiting predicted confidence levels failing below the threshold λ are denoted as Bu′ . Pseudo-label contrastive clustering only takes Bl′ as inputs. For each sample in Bl′ , the set of the positive pairs contains samples with the same given label or pseudo-label. Conversely, the set of negative pairs contains samples of other classes. Formally, the objective of pseudo-label contrastive clustering is defined as follows:

where zi denotes the output logits, P(xi) denotes the set of positives of xi and A(xi) ≡ Bl′\{xi}. τ is a tunable temperature parameter and we set τ = 0.4 in all experiments.

In contrast to existing methods such as ORCA [Cao et al., 2022] and NACH [Guo et al., 2022], which establish a single positive pair for each sample by identifying its nearest neighbour, the objective in Eq. (6) adopts pseudo-labels to form multiple positive pairs. This approach offers dual advantages: firstly, the alignment of samples within the same class is more effectively harnessed through the utilization of multiple positive sample pairs; secondly, it leverages the consistency of distinct views of the same sample to mitigate the negative impact of erroneous positive pairs, while concurrently imparting a repulsion effect to samples from different classes through negative pairs.

Since Eq. (6) augments the labeled dataset by unlabeled samples of high predictive confidence, we ask whether unlabeled samples of low confidence can be used to enrich representation learning. In pursuit of this, we incorporate unsupervised contrastive learning [Wang and Isola, 2020] to encourage similar predictions rather than embeddings between a given sample and its augmented counterpart. This helps to signify the uniformity among unlabeled samples, ultimately leading to clearer separations. In detail, for each sample in the low-confidence set Bu′ , the unsupervised contrastive learning couples it with its augmented view to constitute a positive pair. Simultaneously, a set of negative pairs is formulated, containing all the samples within the mini-batch except the sample itself. The unsupervised contrastive learning loss LUC is formulated as follows:

Here, xp is the positive sample of xj and A(xj) ≡ Bu′ ∪ Bl′\{xj} for the sample xj.

In essence, unsupervised contrastive learning complements the pseudo-label contrastive clustering by fully exploiting the unlabeled samples. In the experiments, the ablation studies underscore the pivotal role played by both types of contrastive losses in our approach.

Lastly, we incorporate a maximum entropy regularizer to address the challenge of converging during the initial training phases, when the predictions are mostly wrong (e.g., the model tends to assign all samples to the same class) [Arazo et al., 2020]. Specifically, we leverage the KL divergence between the class distribution predicted by the model and a uniform prior distribution. It is worth noting that the integration of an entropy regularizer is a widespread practice in dealing with the OpenSSL problem, including approaches such as ORCA, NACH, and OpenNCD. The final objective function of LPS is articulated as follows:

where REntropy denotes the entropy regularizer, η1 and η2 are hyper-parameters set to 1 in all our experiments. We provide detailed analyses on the sensitivity of hyperparameters in the supplementary material.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate our method on three commonly used datasets, i.e., CIFAR-10, CIFAR-100 [Krizhevsky, 2009], and ImageNet [Russakovsky et al., 2015]. Following prior works [Cao et al., 2022; Guo et al., 2022; Liu et al., 2023], we assume that the number of novel classes is known. Specifically, we randomly select 50% of the classes as seen classes, and the remaining classes are regarded as novel classes, e.g., the number of novel classes is 50 for CIFAR-100. On each dataset, we consider two types of labeled ratios, i.e., only 10% or 50% of data in seen classes are labeled. For the ImageNet dataset, we subsample 100 classes to form the ImageNet-100 dataset for fair comparisons with existing works.

Following prior works [Cao et al., 2022; Guo et al., 2022; Liu et al., 2023], we evaluate our method with respect to the accuracy of seen classes, novel classes, and all classes. For seen classes, the accuracy is calculated as the normal classification task. For novel classes, we first utilize the Hungarian algorithm [Kuhn, 1955] to solve the optimal prediction-target class assignment problem and then calculate the accuracy of novel classes. For overall accuracy, we also solve the optimal assignment in the entire unlabeled dataset to calculate the novel class accuracy, measuring the overall performance.

Implementation Details. Following [Cao et al., 2022; Guo et al., 2022; Liu et al., 2023], we utilize the self-supervised learning method SimCLR [Chen et al., 2020a] to pre-train the backbone and fix the first three blocks. In LPS, the weak augmentation contains random crop and horizontal flip, and the strong augmentation is RandAugment [Cubuk et al., 2019]. For CIFAR-10 and CIFAR-100, we utilize ResNet-18 as our backbone which is trained by the standard SGD with a momentum of 0.9 and a weight decay of 0.0005. We train the model for 200 epochs with a batch size of 512. For the ImageNet dataset, we opt for ResNet-50 as our backbone. This choice also undergoes training via the standard SGD, featuring a momentum coefficient of 0.9 and a weight decay of 0.0001. The training process spans 90 epochs, with a batch size of 512. and The cosine annealing learning rate schedule is adopted on CIFAR and ImageNet datasets. These experiments are conducted on a single NVIDIA 3090 GPU.

4.2 Comparing with ExistingMethods

Baselines. We compare LPS with SSL methods, open-set SSL methods, NCD methods, and OpenSSL methods. The NCD methods consider that the labeled data only has disjoint classes compared with the unlabeled data and aim at clustering novel classes without recognizing seen classes. For novel classes, clustering accuracy can be obtained directly. For seen classes, we first regard them as novel classes and leverage the Hungarian algorithm to match some of the discovered classes with seen classes, and then calculate the classification accuracy. We select two competitive NCD methods DTC [Han et al., 2019] and RankStats [Han et al., 2020] in the experiments. Moreover, we include GCD [Vaze et al., 2022] for comparison, which is an extended NCD method.

For the SSL and open-set SSL methods, we leverage their capability in estimating out-of-distribution samples to extend to the OpenSSL setting. For comparison, we select FixMatch[Sohn et al., 2020], which assigns pseudo-labels to unlabeled samples based on confidence. The classification accuracy of seen classes can be reported directly according to pseudo-labels. For novel classes, we first estimate samples without pseudo-labels as novel classes and then utilize k-means to cluster them. The open-set SSL methods maintain the classification performance of seen classes by rejecting novel classes. We compare with DS3L [Guo et al., 2020] and calculate its accuracy in the same way as FixMatch.

For the OpenSSL methods, we compare with ORCA [Cao et al., 2022], NACH [Guo et al., 2022], and OpenNCD [Liu et al., 2023]. We also compare the self-supervised pre-trained model SimCLR and conduct K-means on the primary features to calculate the accuracy.

Results. The results on three datasets are reported in Table 1. The mean accuracy is computed over three runs for each method. Although the non-OpenSSL methods perform well on their original tasks, their overall performance is unsatisfactory in the OpenSSL setting. The results of SimCLR are obtained by the pre-trained model without extra fine-tuning, and the OpenSSL methods are based on the pre-trained model. It is obvious that the OpenSSL methods achieve significant performance improvements compared to non-OpenSSL methods. Compared with the state-of-the-art OpenSSL methods, our method LPS achieves the best overall performance across all datasets. On the CIFAR-10 dataset, LPS outperforms NACH by 1.2% in novel class accuracy. Likewise, on the CIFAR-100 dataset, LPS demonstrates superiority, yielding a substantial 3.2% improvement. Particularly concerning the ImageNet-100 dataset, LPS has the capacity to surpass existing state-of-the-art methods, resulting in a 3.8% increase in overall accuracy. Experimental results demonstrate that LPS can effectively balance the learning of seen and novel classes.

Distribution Analysis. For further validation of our approach, we present a comprehensive analysis of the KL divergence trend between the estimated and prior class distributions, along with the estimated class distributions at the

(a) KL Divergence.

(b) Class Distribution.

Figure 4: (a) The KL divergence between the estimated and prior class distributions. (b) The estimated class distribution of the final training iteration. Experiments are conducted on the CIFAR-100 dataset with 50% seen classes (50% labeled) and 50% novel classes.

concluding training iteration for LPS, NACH, and ORCA in Figure 4. It can be seen that LPS is able to give more attention to novel classes in the early training stage, in contrast to ORCA and NACH, which exhibit a preference for learning seen classes primarily. This becomes particularly evident in the estimated class distribution at the end of the training, where ORCA and NACH falter in effectively distinguishing novel classes, leading to the erroneous assignment of certain novel class samples. In contrast, the estimated distribution obtained by LPS closely aligns with the prior distribution, consequently fostering effective novel class differentiation.

Feature Visualization. We employ t-distributed stochastic neighbour embedding (t-SNE) [Van der Maaten and Hinton, 2008] to visualize the learned feature representations through LPS and prior methods. The results depicted in Figure 5 demonstrate that the representations obtained by LPS yield clearer classification boundaries and exhibit fewer instances of misclassifications. Conversely, ORCA and NACH manifest a higher frequency of misclassifications and unclear boundaries, resulting in comparatively inferior performance in contrast to LPS. In addition, we utilize the metric of normalized mutual information (NMI) to further assess the clustering performance and representations of novel classes. We report the results in Table 2 which clearly demonstrate the superiority of LPS. Given that all methods share the same foundation of SimCLR pre-trained backbone, the visualization, and NMI results highlight the efficacy of our approach in enhancing representation learning.

Fine-tuning the Pre-trained Backbone. Furthermore, it is noteworthy that all previous OpenSSL methods adopt a practice of freezing the parameters within the first three blocks of the backbone, solely fine-tuning the last block, with the intention of mitigating overfitting. However, such an approach constrains the extent of performance enhancement, as the backbone's parameters remain unmodifiable and unoptimized to better suit downstream tasks. To establish that our method is not susceptible to the overfitting dilemma, we conducted a series of experiments on the CIFAR dataset employing stateof-the-art OpenSSL methods while fine-tuning the backbone. The results are reported in Table 3. The experimental results reveal that existing OpenSSL methods manifest modest performance improvement, if any, in comparison to their initial frozen counterparts. In contrast, our proposed method, unaffected by overfitting concerns, consistently yields substantial performance gains across both seen and novel classes. Specifically, the overall accuracy for CIFAR-10 experiences a notable improvement of 2.9%, while an impressive 6.3% increase is observed for CIFAR-100. These results underscore the effectiveness of LPS in harnessing the additional learnable parameters for further enhancing model performance.

Ablation Analysis. Moreover, we conduct a comprehensive analysis of the contributions of distinct components in our approach. The objective function of LPS comprises the adaptive margin loss (LAM), the pseudo-label contrastive clustering loss (LPC), the unsupervised contrastive learning loss (LUC), and the entropy regularizer (REntropy). Concretely, the ablation study is mainly conducted by removing each term individually from the objective function except for the adaptive margin which is replaced by a standard cross-entropy. As observed in Table 4, the removal of any components leads to performance degradation. The substantial drop in novel performance after removing the entropy regularizer highlights its significant role in the process of novel class discovery. Moreover, the utilization of both pseudo-label contrastive loss and adaptive margin loss substantially improves the accuracy of novel classes.

5 Conclusion

In this study, we present Learning Pace Synchronization (LPS), a potent solution tailored to the OpenSSL problem. LPS introduces an adaptive margin loss to effectively narrow the learning pace gap that exists between seen and novel classes. Moreover, we formulate a pseudo-label contrastive clustering loss to augment the process of novel class discovery. Extensive evaluation is conducted across three benchmark datasets with distinct quantities of labeled data. We also discover that the conventional practice of freezing the self-supervised pre-trained backbone hinders the generalization performance. We hope our work can inspire more efforts towards this realistic setting.

Acknowledgments

This work was supported by the National Science Foundation of China (62206049, 62225602), and the Big Data Computing Center of Southeast University.