Masked Contrastive Learning for Anomaly Detection

1 Introduction

Over the past few years, machine learning has achieved immense success surpassing human-level performance in many tasks, such as classification, segmentation, and object detection [Tan and Le, 2019; Tan et al., 2020; Chen et al., 2020a]. However, such a well-trained model assigns arbitrary high probability [Hein et al., 2019] on the unfamiliar test samples, since most machine learning systems generally depend on the closed-set assumption (i.e., i.i.d. assumption). This phenomenon may lead to a fatal accident in safety-critical applications like medical-diagnosis or autonomous driving. Anomaly detection 1 is a research area that aims to circumvent such symptoms by identifying whether the test samples

1 also termed out-of-distribution detection, novelty detection, or outlier detection in the contemporary machine learning context.

(a) SimCLR

(b) SupCLR

(c) MCL without SPA

come from in-distribution or not. A flurry of recent deeplearning based models, including reconstruction based [Oza and Patel, 2019; Li et al., 2018], density estimation based [Malinin and Gales, 2018], post-processing methods [Lee et al., 2018; Liang et al., 2017], and self-supervised learning methods [Golan and El-Yaniv, 2018; Hendrycks et al., 2019; Tack et al., 2020; Winkens et al., 2020], have been proposed for the task and have shown noticeable progress.

Among the numerous approaches mentioned, selfsupervised learning (SSL) is in the limelight and validating its superiority over previous methods in various research areas

[Devlin et al., 2018; Chen et al., 2020a]. Since it is unfeasible to access out-of-distribution (OOD) data in most real-world scenarios, the ability of SSL to learn complex and diverse representations without additional labels is receiving much attention from anomaly detection lately. [Golan and El-Yaniv, 2018] is one of the earlier works to identify the potential of SSL and has proposed a simple yet effective technique that aims to learn intrinsic features within in-distribution (IND) samples via auxiliary tasks (e.g., predicting flip, rotation, or translation of input data). Furthermore, [Hendrycks et al., 2019] confirmed that using such auxiliary tasks not only helps to determine anomalous samples but also helps to defend against adversarial attacks. More recent works [Tack et al., 2020; Winkens et al., 2020] exploit contrastive learning (CL), especially SimCLR [Chen et al., 2020a], that learns individual data representations in a task-agnostic way by maximizing the agreement between differently augmented views of the same image while repelling others in the batch.

SimCLR obtains effective individual representation for each data point, as well as clustered representations for each class, even without any human label or supervision. (See Fig. 1a.) However, its task-agnostic feature results in blurry boundaries between each cluster, so it requires a fine-tuning process used for some downstream tasks (e.g., multi-class classification). Such process undermines expression ability well-learned through SimCLR, given that most of the finetuning procedure leverages cross-entropy loss and it solely considers class labels of the data without taking into account unique characteristics of the data or similarity between them. Consequently, the fine-tuned model often assigns high confidence probabilities to OOD input, reducing the distributional discrepancy between IND and OOD [Hein et al., 2019].

Our foremost insight is that forming dense clusters without fine-tuning while preserving individual representations by inheriting the advantages of SimCLR will shape a more meaningful visual representation contrary to the pre-train then tune paradigm, thus contributing to the effective detection of anomalous data. To this end, we propose a task-specific variant of contrastive learning called masked contrastive learning (MCL), which can shape more clear boundaries between each class. (See Fig.1d) The core idea of MCL is to generate a mask that can adjust the repelling-ratio properly by considering class labels in the batch. Experimental results show that MCL is more befitted to anomaly detection then SimCLR or its other task-specific variant (i.e., SupCLR), which still exhibits blurry decision boundaries (See Figure 1.1b).

Moreover, contrary to the previous belief that the auxiliary self-supervision task (e.g., predicting flip, rotation, or translation of input data) does not substantially improve label classification accuracy [Hendrycks et al., 2019], we observe that it is possible to considerably improve both IND and OOD performance with a proper inference method. To this end, we propose self-ensemble inference (SEI) that fully exploits ability learned from simple auxiliary self-supervision task in the inference phase. SEI enhances model performance in all situations without losing generality and can be used in any classifier. By combining our models, we can outperform previous state-of-the-art methods.

Our main contributions are summarized as below:

• We propose a novel extension to contrastive learning dubbed masked contrastive learning which can shape dense class-conditional clusters.
• We also propose an inference method called selfensemble inference that fully leverages ability learned from auxiliary self-supervision tasks in test time. Selfensemble inference can further boost both IND and OOD performance.
• We validate our approaches on various image benchmark datasets, where we obtain significant performance gain over the previous state-of-the-art.

The source code for our model is available online. 2

2 Masked Contrastive Learning

As the name implies, our method adopts contrastive learning, particularly SimCLR, with two additional components: class-conditional mask and stochastic positive attraction; see Fig. 2. In this section, we provide detailed explanations of each component in MCL. (See Section 5 for further details regarding contrastive learning.)

2.1 Background: Contrastive Learning

Recent contrastive learning algorithms (e.g., SimCLR) learn representations by maximizing the agreement between differently augmented views of the same image while repelling others in the batch. Specifically, each image xk from randomly sampled batch B = {(xk, yk)}Nk=1 is augmented twice, generating an independent pair of views (¯x2k-1, ¯x2k) and augmented batch ¯ B = {(¯xk, ¯ yk)}2N k=1 , where labels of augmented views ¯ y2k-1, ¯ y2k are equal to original label yk. The augmented pair of views, ¯x2k-1 = t(xk) and ¯x2k = t′ (xk), are generated via independent transformation instance t and t′ , drawn from pre-defined augmentation function family T . (¯x2k-1, ¯x2k), then are passed sequentially through encoder network fθ and projection head gφ, yielding latent vectors (z2k-1, z2k) that are utilized for the contrastive loss (i.e., NTXent):

where simi,j = z⊤ i zj/(‖zi‖‖zj‖) denotes cosine similarity between pair of latent vectors in (zi, zj) and τ stands for temperature hyper-parameter. The final objective is to minimize Eq. 1 over positive pairs, which maps the input into effective individual representation in a task-agnostic way:

2.2 Class-Conditional Mask

The benefit of CL in anomaly detection has been reported recently [Winkens et al., 2020; Tack et al., 2020]. Nonetheless, we found that well-formed representations from CL, which facilitate distinguishing anomalous data, are lost during the fine-tuning procedure. Due to the task-agnostic nature of CL, however, the fine-tuning steps are essential, making it difficult to avoid the aforementioned phenomenon. MCL mitigates such symptoms by injecting task-specific characteristics to existing CL, resulting in fine-tuning procedure inessential. One key component in MCL is class-conditional mask (CCM) which is a simple yet effective masking technique that adaptively determines the repelling-ratio considering the label information in each ¯ B. CCM can be defined as follows:

2 https://github.com/HarveyCho/MCL

where 0 < α < 1/τ. CCM alters the temperature for the same label views to a smaller value α, so that query view repels views with the same label relatively small amount compared to views with different labels. The generated CCM is then multiplied to the similarity score in Eq. 1, modifying the previous SimCLR loss to the following equation.

Penalizing a small ratio α to positive views restrains respective representation in the same cluster from being too similar to each other, making individual data representation more distinctive.

2.3 Stochastic Positive Attraction

As can be seen in Fig. 1c, CCM promotes a more label-wise cluster compared to SimCLR. Even in CCM, however, the core operating principle is still identical to SimCLR in that it only attracts the view from the same image while it repels remaining views within the batch. Due to this repulsive nature, each data representation gets more distant as training continues, and it leads to the formation of unsatisfactory scattered clusters. To alleviate this phenomenon, we add another component named stochastic positive attraction (SPA), an additional attraction with the stochastically sampled view in the batch. Specifically, SPA attracts query ¯xi with stochastic positive sample (¯xj, ¯ yj) ∼ U( ¯ B+ i ) in the positive augmented batch ¯ B+ i for query ¯xi, where U refers to the discrete uniform distribution. The positive augmented batch ¯ B+ i for query ¯xi contains views with the same label except views from its parent image x(i-1)\2, where the symbol \ denotes integer quotient operator:

CCM is also used for negative views with the additional constraint which excludes views from its parent image. SPA for query view ¯xi now can be defined as follows:

The complete version of MCL is acquired by combining CCM and SPA, where the overall loss term being as follows:

where λ denotes weight hyper-parameter for SPA loss.

2.4 Training Auxiliary Task inMCL

Training simple auxiliary self-supervision task along with the main downstream task is possible in MCL by adding constraint in CCM. Let Tmain be the main task with Cmain number of classes, Taux be an auxiliary task with Caux number of classes and corresponding augmented batch be ¯ B = {(¯xi, ¯ ymain i , ¯ yaux i )}2N i=1 with additional auxiliary task label. Then CCM with auxiliary self-supervision task can be defined as follows:

By simply setting β = 1/τ, it is possible to train Cmain × Caux distinctive clusters for each (¯ ymain j , ¯ yaux j ) pairs. Since the auxiliary task plays a complementary role to the main task, it is more plausible to form grouped clusters for respective main labels and to have distinctive clusters for each auxiliary label inside them. With appropriate constraint (i.e., 0 < α < β < 1/τ ), MCL forms hierarchical clusters by dint of CCM.

3 Inference

3.1 Scoring Function inMCL

Since there is no task-specific final layer in MCL, classification or anomaly detection are conducted via class-wise density estimation analogous to [Lee et al., 2018], utilizing negative Mahalanobis distance -dM as a scoring function s:

where z = gφ(fθ(x)), and µi, Σi refer to mean and covariance matrix of n-dimensional multivariate normal distribution (MND) N(µi,Σi) for class i ∈ I = {1, 2, · · · ,Cmain}. Note that calculating MNDs for each class is a one-time operation acquired from training data. The vector S(x) contains scores of each label for image x and the class label with highest score i∗ = argmaxn∈I Sn(x) is selected as a predictive label, where Sn(x) denotes n-th element of S(x). The corresponding IND score for predictive label si∗(x) measures the confidence for predictive label i∗ which are used to distinguish OOD data, following the binary decision function hδ from below:

where δ denotes anomaly threshold.

3.2 Self Ensemble Inference

The key idea of SEI is to exploit the model's ability to discriminate within IND, learned through an auxiliary selfsupervision task, in the inference phase. For example, consider predicting 4-directional rotations (from 0◦ , 90◦ , 180◦ , to 270◦ ) is employed for an auxiliary task. Then SEI ensembles the results from corresponding the 4-rotated test images and derives calibrated index i∗ and corresponding score si∗. Specifically, let i ∈ I = {1, 2, ...,Cmain} be the main task label, and j ∈ J = {1, 2, ...,Caux} be the auxiliary task label. Then, Cmain × Caux number of MNDs N(µ(j) i ,Σ(j) i ) are calculated for every label combinations. The test image x is augmented Caux times, yielding {x(j) }Caux j=1 . Each augmented test image x(m) with the class label yaux = m is fed into the corresponding MND N(µ(m) i ,Σ(m) i ), where i ∈ I and j = m, yielding a score vector S(m) (x):

Our goal is to aggregate {S(j) (x)}Caux j=1 properly to make model more robust and reliable. We considered 3 different aggregation methods to extract predictive label i∗ . The foremost intuitive way is averaging the main label scores across {S(j) (x)}Caux j=1 .

Another variation is to select label index from the highest IND score:

The last aggregation is the weighted-average, which gives adaptive weights to each score in S. Weights for each score are computed per j using the harmonic mean to penalize exceptionally low scores for better calibration:

Corresponding score to predictive label i∗ is used to distinguish OOD data following Eq. 14. Depending on the aggregation method, the effect that the model can achieve varies. Further details are elaborated in Section 4.2 with experimental results.

4 Experiment

In this section, we demonstrate the effectiveness of MCL and SEI on several multi-class image classification datasets.

Experiment configurations. In the following experiments, we adopt ResNet-34 [He et al., 2016] with a single projection head, following structure used to train CIFAR-10 in SimCLR. We also fixed hyper-parameters related to contrastive learning following SimCLR, which include transformation T = {color jittering, horizontal flip, grayscale, inception crop}, the strength of color distortion to 0.5, batch size to 1024, and temperature τ to 0.2, to keep our experiment tractable. For MCL hyper-parameters, we set α to 0.05, β to 2.5, and λ to 1 which meets certain condition for MCL (See Appendix A for details). Unlike SimCLR, we used SGD optimizer with learning rate 1.2 (0.3 × batch size / 256), decay 1e-6, and momentum 0.9. Furthermore, we use a cosine annealing scheduler without any warm-up.

Table 2: Performance comparison between MCL and SupCLR. Models are trained on CIFAR-10 dataset with ResNet-18.

Evaluation metrics. To evaluate IND detection performance, we measured the label classification accuracy. For OOD detection performance, we used the area under the receiver operating characteristic curve (AUROC), which is a threshold (δ in Section 3.1) free metric and the most common metric in anomaly detection literature.

Selecting self-supervision tasks. Since the complex auxiliary task is not our primary concern, we considered rotation, horizontal flip, and translation as candidates for auxiliary tasks, which are simple and commonly used in the area of anomaly detection [Golan and El-Yaniv, 2018]. However, MCL contains inception crop [Szegedy et al., 2015] and horizontal flip in contrastive transformation T , so using translations or horizontal flip as an auxiliary task only confuses the model. To this end, we employed predicting 4-directional rotations (0◦ , 90◦ , 180◦ , and 270◦ ) as our auxiliary task.

4.1 Multi-Class Anomaly Detection

We trained our model on CIFAR-10 [Krizhevsky et al., 2009] as IND, and used CIFAR-100, SVHN [Netzer et al., 2011], ImageNet [Deng et al., 2009], and LSUN [Yu et al., 2015] datasets for OOD. Note that all the classes in OOD datasets are disjoint with CIFAR-10. In particular, for ImageNet and LSUN dataset, we use ImageNet-Fix and LSUN-Fix datasets [Tack et al., 2020] which are fixed versions of the previously released dataset [Liang et al., 2017]. The previous version contains unintended artifacts caused by resizing large images.(See Appendix I in [Tack et al., 2020] for more details.)

Main Results

We compared MCL's performance with several other methods in OOD detection. Table 1 summarizes our experimental results. MCL with SEI shows significant performance gain over previous methods in all setting; moreover, it outperforms supervised method [Hendrycks et al., 2018], which utilizes additional explicit OOD data.

Comparison with SupCLR

We compared MCL's performance with SupCLR [Khosla et al., 2020], another task-specific variant of CL. Table 2 summarizes performance comparison with SupCLR. In terms of test accuracy, SupCLR performs slightly better than MCL, while in terms of AUROC, MCL shows better performance. The superiority of MCL in detecting anomalies comes from several differences between the two frameworks. Specifically, SupCLR additionally attracts all same label views from ¯ B+ i in the augmented batch ¯ B. Since SupCLR forces all positive views to have a high similarity to the query view, the ability to distinguish data with the same label diminishes. Unlike SupCLR, MCL penalizes a small amount α to positive views, which endows an ability to discern each data with same label.

4.2 Ablation Study

In this section, we perform an ablation study on our proposed methods, along with baselines. In all experiments, we treated CIFAR-10 as IND and CIFAR-100 as OOD.

Masked Contrastive Learning

We conduct ablation experiments to explore the effectiveness of the main components (CCM, SPA, and auxiliary 4-way rotation task) in MCL. Tab. 3 reports our ablation experiments along with baseline models. For SimCLR based models, we fine-tuned the pre-trained model in two ways. One way is to use cross-entropy loss which is a traditional finetuning method in the classification task, and the other way is to use cross-entropy loss along with SimCLR loss (Eq.2) jointly [Winkens et al., 2020]. The two methodologies show almost the same accuracy, while there is a slight difference in AUROC. Our justification for this phenomenon is due to the nature of the cross-entropy loss, which solely reflects the class label leading diminish in the distributional discrepancy between IND and OOD. We conjectured that the additional SimCLR loss in joint fine-tuning mitigates this phenomenon and shows better AUROC performance. This phenomenon is more evident when applied to MCL, showing substantial performance degradation in AUROC. Therefore, we used MCL with neither a fine-tuning procedure nor any task-specific layer on the top level of the network.

Self-Ensemble Inference 1

In this part, we share our findings on SEI with its variations (average, maximum, and weighted-average). Since we employed a 4-directional rotation prediction as our auxiliary task, SEI is done in a 4-way correspondingly. It is also possible to add additional 4-way SEI with a horizontally flipped image, which we dubbed 8-way SEI. (Fig. 3 provides a visual explanation.) 8-way SEI follows the same strategy introduced earlier. The only difference is the number of augmented images to ensemble. As claimed in [Hendrycks et al., 2019], learning the auxiliary task alone can not improve accuracy. But with the help of SEI, we were able to achieve performance gains in both accuracy and AUROC regardless of its variations as can be seen in Tab. 4. Interestingly, the effect of each ensemble differs depending on the aggregation method. For example, average SEI makes the model robust to input variation which is a commonly known benefit of the ensemble, bringing noticeable gain in AUROC contrary to accuracy. On the other hand, maximum SEI yields a significant gain in accuracy, which indicates MCL's prediction with high confidence score is quite precise. The weighted-average SEI absorbs the advantages of both ensembles by assigning adaptive weights to its score. As a side note, the SEI can be used for the general classifier trained with an auxiliary task, which also yields significant performance gains as can be seen in Tab. 1.

Self-Ensemble Inference 2

We also conduct ablation experiments on SEI and the relationship with the auxiliary task. We applied SEI to 3 differently trained model as follows:

• MCL without auxiliary task: MCL is trained neither with auxiliary task nor data augmentation
• MCL with data augmentation: MCL trained with additional 4-way rotated images without rotation labels.
• MCL with auxiliary task: MCL is trained with additional 4-way rotated images with rotation labels.

Tab. 5 summarizes our extended ablation study for SEI. Applying SEI to MCL without auxiliary task only confuses the model and undermines both IND and OOD performance substantially, as rotated images are unseen during training phase. If the rotated image is augmented without a label, the model performance is degraded but the SEI shows a slight

• (a) OOD samples with high IND scores.

(b) Wrongly classified samples from IND.

Figure 5: Histogram and PDF for correctly classified IND, wrongly classified IND and OOD distribution.

improvement. Finally, when the auxiliary task is explicitly trained with the main downstream task, SEI shows significant gain without any performance degradation. Such experimental results reveal that it is necessary to additionally train the auxiliary task to use SEI properly.

4.3 Qualitative Results

In this section, we analyze the data distribution along with several failure cases of our model. As can be seen in Fig. 5, both OOD data distribution and wrongly classified data distribution have lower scores compared to correctly classified samples, which indicates that our model can measure predictive uncertainty quite precisely. Furthermore, to analyze our failure cases in detail, we conducted a case study on OOD samples with high confidence and wrongly classified IND samples. In MCL, representation is learned based on semantic similarity, so it is feasible to conjecture model's decision-making process via observing nearest neighbors of the input. For most failure cases, their predicted label and ground-truth label (GT) belongs to the same super-class regardless of the aforementioned failure types. For example, MCL predicts a bear (OOD) as a dog (IND); likewise, a cat as a dog (both IND), which belongs to the same super-class, mammal. As a side note, few data were mislabeled, as can be seen in Fig. 4.(a).(GT: Fox)

5 RelatedWork

Anomaly detection. Recent approaches in OOD can be categorized as follows: reconstruction based [Oza and Patel, 2019; Li et al., 2018], density estimation based [Malinin and Gales, 2018], post-processing based [Lee et al., 2018; Liang et al., 2017], and self-supervised learning based. Selfsupervised learning based methods can be split again into auxiliary self-supervision based [Golan and El-Yaniv, 2018; Hendrycks et al., 2019] and contrastive learning based [Tack et al., 2020; Winkens et al., 2020]. Our method belongs to self-supervised learning, which exploits both auxiliary selfsupervision task and contrastive learning.

Contrastive learning. Contrastive learning is a specific framework of self-supervised learning, which has shown impressive results in visual representation learning tasks [Chen et al., 2020a; Chen et al., 2020b]. Most recent work in OOD [Tack et al., 2020; Winkens et al., 2020] report that employing CL improves OOD performance. Our work goes further from the previous papers and proposes a task-specific variant of CL. [Khosla et al., 2020] proposed SupCLR, another task-specific variant of SimCLR, which is a noteworthy work. Similar to MCL, SupCLR also leverages label information in the batch while training and shows superior accuracy over SimCLR. Despite its performance in IND accuracy, representation from SupCLR which is not entirely appropriate for discerning anomalous data, as it attracts all same label views which discrepancy in each class disappears.

6 Conclusion

In this paper, we propose a novel training method called masked contrastive learning (MCL) and an inference method called self-ensemble inference (SEI). MCL can shape classconditional clusters by inheriting advantages of CL and SEI fully leverages trained features from auxiliary self-supervised tasks in the inference phase. By combining our methods, our model reaches the new state-of-the-art performance.

Acknowledgements

We thank Taeuk Kim, Ye-seul Song and the anonymous reviewers for their thoughtful feedback and comments.