Novelty Detection via Contrastive Learning with Negative Data Augmentation

1 Introduction

Novelty detection can be described as a one-class classification, which aims to detect the samples whether drawing far away from the learned distribution of training samples from the target class. Generative adversarial networks (GANs) [Sabokrou et al., 2018; Perera et al., 2019; Chen et al., 2020a] have been a common choice for novelty detection. Generator and discriminator compete mutually while collaborating to learn a representative latent space for the target class. In this latent space, the reconstructed features of novelty samples (outliers) have higher reconstruction errors than normal samples (inliers), which can be used for distinguishing between normal and novelty classes. Recently, self-supervised learning [Komodakis and Gidaris, 2018; Ji et al., 2019] holds great for improving representations when labeled data are scarce. In the training process, the network learns useful feature representation by solving some specialized pretext tasks, such as geometric transformations prediction [Hendrycks et al., 2019]. During inference, the model is transferred to the downstream task, like novelty detection.

∗These authors contributed equally to this work.

†Corresponding authors.

However, some weaknesses are still existing in the previous work. For the GAN based methods, it suffers from three critical problems: mode-dropping, instable training, and low discriminative ability. First, various GAN based methods (e.g. ALOCC [Sabokrou et al., 2018], OCGAN [Perera et al., 2019] and DualGAN [Chen et al., 2020a]), only learn the partial modes of target distributions, which causes the problem of mode dropping [Arora et al., 2018]. Second, the imbalance capacity of generator and discriminator causes the training of model unstable [Zhao et al., 2017], which affects the learning of latent representation of normal samples. Third, the latent features with low discriminative ability [Liu et al., 2020] are generated in self-representation, since the decoder of GANs tends to learn more structive representation than discriminative characteristics.

The existing self-supervised learning based novelty detection requires specialized implementation for the pretext tasks, such as design supervised labels, loss functions, and network architectures. Rotation prediction [Komodakis and Gidaris, 2018] and transformations prediction [Hendrycks et al., 2019] could capture the semantic information of object shapes that are useful for target tasks. However, it lacks other semantic information such as object textures and colors, which leads to the low discriminative ability of features that fails to effectively detect novelty samples. Besides, some [Lim et al., 2018; Sinha et al., 2021] use data augmentation as an additional source of data in the GAN. Inspired by these works, contrasting shifted instance (CSI) [Tack et al., 2020] contrasts distributionally-shifted augmentations with an auxiliary softmax classifier to learn the feature representation of encoder based on SimCLR [Chen et al., 2020b]. However, the augmented images as negative samples only use one random augmentation, which cannot generate effective negative samples. Beside, the detection score function of CSI is complex with high computation and memory cost.

To address these issues, we propose a novel decoderencoder framework for one-class novelty detection to learn more discriminative latent representation by contrastive learning. Our framework consists of three parts: a generative network, a contrastive network, and a mutual information estimator. First, the generative network (decoder) aims to learn the representation of target class by mapping each initialized latent vector to each target image; The initialization of the latent vectors is obtained by encoding the entire distribution of training data to alleviate the problem of modedropping. Then, we employ the contrastive network (encoder) to extract both local feature maps from positive and negative samples. It also captures the global latent features from positive samples. In particular, we select the normal training data as positive samples and use negative data augmentation strategy to generate hard negative samples (a.k.a. novelty-like samples) from their corresponding positive samples by using multiple random transformations. Different from outliers, these novelty-like samples are more close to the normal ones. Therefore, it can better help to separate the distribution of normal and novelty samples by learning the disjointness between positive and novelty-like samples, as illustrated in Fig.1. Finally, mutual information estimator is adopted to generate discriminative latent features through contrastive learning on the features of input pairs by maximizing the local, global, and prior mutual information. Our decoder-encoder framework is presented in Fig. 2.

We summarize our main contributions of this paper:

• We propose a novel effective decoder-encoder framework for the novelty detection task. Contrastive learning is introduced to learn more discriminative latent features for distinguishing between positive and negative augmentation samples.
• The mutual information estimator is trained in a nonadversarial way by distinguishing between features of local parts and global context constituted by only positive and novelty-like samples, which helps the model to train more stably with faster convergence than GANbased methods.
• Extensive experiments demonstrate the superior performance of our approach for novelty detection in several challenging datasets. For instance, our method achieves the highest mean AUC of 0.843 and 0.899 on CIFAR-10 and DCASE, compared to state-of-the-art methods.

2 Methodology

2.1 Our Decoder-encoder Framework

Our framework consists of three components: a generative network, a contrastive network and a mutual information estimator, as it shown in Fig. 2. The generative network learns a mapping from latent space to high-dimensional image space, while the contrastive network maps a positive/negative image to local feature maps and a global latent feature. The mutual information estimator is used for distinguishing between the features from the target samples and their corresponding hard negative samples to effectively learn more discriminative latent feature presentation.

Notations. Let X = {x1, · · · , xN} denote the original input images of target class with N samples, Z = {z1, · · · , zN} denote their corresponding global latent features where zi = φθc (xi) ∈ Rd is learned by the contrastive network C with parameters θc. d is the dimension of latent features. Ai = φθfp (xi) ∈ RH×W×S is the local feature maps extracted from contrastive network C, where θfp ⊂ θc and H ×W × S is the dimension of feature maps. x′i = ψθg (z′i ) presents the reconstructed image by generative network G with parameters θg, where z′i is the initialized latent vector. In mutual information estimator, we denote the global estimator, local estimator and prior estimator as GE, LE and PE with parameters θge, θle and θpe, respectively.

Generative network. The input latent vectors z′1 , , · · · , z′N of generative network are first initialized by the PCA projection of all input images, which helps to alleviate the problem of mode dropping. Like a decoder, the generative network outputs ψθg (z′i ) should be regained close to the input image xi. Instead of MSE loss, we employ the Laplacian pyramid loss [Ling and Okada, 2006] as the reconstruction loss; MSE is easy to yield the blurry image, while the Laplacian pyramid loss can generate better reconstructed images by Laplacian pyramid representation. Therefore, the reconstruction loss between the reconstructed output ψθg (z′i ) and the input image xi is formulated as:

where Lapj (·) is the j-th level of Laplacian pyramid representation.

Contrastive network. To improve the discriminative ability of the latent features of the target class, the contrastive network uses a pair of positive and negative samples as an input to extract both local feature maps (Api , Ani ) and a global latent vector zi of the positive sample, where the samples from different classes compete each other through mutual information estimator. As shown in Fig. 2 (blue box), a positive samples xi is an original image and its negative sample is generated by the negative data augmentation M as M(xi) (called noveltylike samples). Our transformations of augmentation include random resized crop, random color jitter, random grayscale, and random horizontal flip. We employ the combination of multiple random transformations to capture more rich semantic information, including object shape, textures, and colors. In addition, this augmentation can better help to separate the distribution of normal and novelty samples by learning the disjointness between positive and novelty-like samples, as the distribution of novelty-like samples is more close to positive ones than outliers. In our framework, we employ cooperative learning between the generative network and contrastive network to generate better reconstructed images and provide more discriminative global latent vectors. It also motivates us to use the decoder-encoder framework instead of the conventional encoder-decoder frameworks with unilateral learning [Vincent et al., 2010; Marchi et al., 2017]. Therefore, we need to minimize the distance between the initialized latent vector z′i and the global latent vector φθc (xi) as:

Inspired by [Sabokrou et al., 2018], the reconstruction space is more effective for distinguishing the positive and novelty samples compared to the latent space. Therefore, a test example x first go through contrastive network and then generative network in a encoder-decoder manner during testing. The constraint by Eq. (2) makes the testing feasible. Without this constraint, the reconstructed images from positive samples will have significantly large reconstructed error even with the constraint of Eq. (1).

Mutual information estimator. Mutual information measures the essential relevance of two instances. The larger mutual information is, the more similar two variables have. Given two random variables x and y, mutual information [Belghazi et al., 2018] can be estimated by the JS divergence between the joint p(y | x)p(x) and the product of the margins p(x)p(y). According to the definition of the variational estimation of JS divergence [Nowozin et al., 2016], the maximization of the mutual information between variables X and Y can be formulated as:

where σ denotes the sigmoid function and T(x) = log 2p(x) p(x)+q(x) . Here p(z|x)p(x) and p(z)p(x) are utilized to replace p(x) and q(x).

In our paper, we divide mutual information estimator into three parts: global estimator, local estimator and prior estimator. The goal of the mutual information estimator is to generate discriminative global latent features by contrastive learning between positive samples and negative samples. We first consider the global mutual information (see Fig. 3(a)). Based on Eq. (3), the maximization of global mutual information is also equivalent to minimize Eq. (4) by introducing a global estimator τθge ([Az, z]), which can be formulated as:

where Apz and Anz are the vectors downscaled from the feature maps Ap and An from the positive sample and the negative sample, respectively. They all have the same dimension to z. [Az, z] is the concatenation between the downscaled feature Az and z as an input pair. β is a hyperparameter. Similar to [Hjelm et al., 2019], the optimization of Eq. (4) is to estimate the global latent feature distribution from positive samples by distinguishing the input from positive samples or negative samples.

Second, we consider local mutual information (see Fig. 3(b)), and also construct the relationship between the local feature map and the global latent feature. The process of estimation is the same as global mutual information. Thus, the objective function of local mutual information loss can be formulated as:

where τθle ([Aij, zA]) is a local estimator to output the reverent probability between the local feature map A and latent representation zA, which consists of a wide range of replicated feature vectors from z with the same dimension to A. An input pair is either from a positive sample [Apij , zA] or a negative sample [Anij , zA] at coordinates (i, j). H and W represent the height and width of the feature map.

Third, we employ the KL-divergence between the global latent feature and the prior distribution (e.g., normal distribution) to encourage the latent feature to be more regular. Thus we can construct the following objective function:

where q(z) is a prior distribution (e.g., normal distribution) and p(z|A) is the output of prior estimator τθpe . By combining the aforementioned three loss functions (i.e., Eqs (3),

(4) and (5)), we obtain the final mutual information estimator loss function:

By minimizing the above function, we generate discriminative latent features z that helps to make the normal and novelty samples separable.

2.2 Overall Loss Function

According to the above discussion, we can construct the overall loss function for our decoder-encoder framework as:

where λ1, λ2 and λ3 are the hyperparameters for balancing these three different terms. For solver, SGD optimizer can be directly used to minimize Eq. (8) in an end-to-end manner.

2.3 Implementation Details

We adopt the structure of DCGAN [Radford et al., 2016] as the structure of the generative network and contrastive network. The global mutual information estimator is a fullyconnected network with two 512-unit hidden layers. A 1 x 1 convnet with two 512-unit hidden layers is regarded as the local mutual information estimator. The prior mutual information estimator is a fully-connected network with two hidden layers of 1000 and 200 units.

Our image augmentation M contains cropping, horizontal flip, color jitter, rotation, and grayscale for random augmentations. During inference, the test sample x first goes through the contrastive network to be encoded into a latent vector. The generative network then upscales the latent vector to reconstruct the image from the learned discriminative latent feature space. Finally, the abnormal score is calculated by image reconstruction error between test image sample x and the corresponding generated image x′ . The test sample x is regarded as a novelty instance if the image reconstruction error is larger than a predefined threshold T, and a normal instance otherwise. We use PyTorch [Paszke et al., 2019] to implement our method. For training parameters, the learning rate and the number of total epochs are set to 0.002 and 100, respectively. SGD optimizer with momentum is adopted to

√

√ √

√

With mutual information estimation Without mutual information estimation

Figure 4: The visualization of latent space learned from target class (Frog class) by using proposed method with/without mutual information estimation.

optimize the parameters of our framework. Batch size, momentum and weight decay are set to 128, 0.9 and 0.005, respectively. For hyperparameters, β and γ are set to 0.5 and 0.1, respectively. λ1, λ2 and λ3 are all set to 1.

3 Experiments

3.1 Experimental Setting

Public novelty detection dataset. We select CIFAR-10 [Krizhevsky and Hinton, 2009], COIL 100 [Nene et al., 1996], MNIST [Lecun and Bottou, 1998], fMNIST [Xiao et al., 2017] and DCASE [Mesaros et al., 2017] as the standard evaluation dataset. COIL100 contains 100 objects with multiple different poses, where each class has less than 100 images. MNIST contains 60K/10K 28 × 28 training/test grayscale handwritten digit images from 0 to 9. fMNIST contains 60K/10K 28 × 28 training/test images of fashion apparels/accessories. CIFAR-10 contains 50K/10K 32 × 32 training/test images with diverse content, background and complexity from 10 classes. In DCASE [Mesaros et al., 2017] dataset, all abnormal event audios are artificially mixed with background audios (e.g. home, bus, and train) which contains 15 different acoustic background scenes. We use the original mixed audio files from the challenge, which contains 491, 496 and 500 audio files of roughly 30 seconds in the training, validation and test dataset, respectively. This task aims to distinguish abnormal acoustic signals from the normal ones.

Face anti-spoofing detection dataset. Replay-Attack [Chingovska et al., 2012] dataset and CASIA-MFSD [Zhang et al., 2012] dataset contain different attacks e.g. printed paper face, replaying a video and wearing a mask. ReplayAttack dataset contains 1.2K videos (200 real access videos and 1K attack videos). CASIA-MFSD dataset contains 50 subjects where each subject has 12 videos under 3 different image resolutions and varied lightings. The goal of this task is to detect whether a face is "alive" or just a fraudulent reproduction.

Evaluation methodology. The protocol in the literature is proposed for one-class novelty detection [Perera et al., 2019]. All of in-class training samples from only one class are used for training, and all samples in test set are used for testing. We use Area Under Curve (AUC) to evaluate the performance in novelty detection and acoustic anomaly detection. We also use Half Total Error Rate (HTER) [Bengio and Mari´ethoz, 2004] for spoofing face detection.

3.2 Ablation Study

In this part, we evaluate the effect of each mutual information loss in Eq. (7), the effect of information fusion between feature maps and latent feature, the effect of Laplacian pyramid loss and the effect with/without PCA initialization. We conduct the experiments on CIFAR-10 for ablation study.

The effect of mutual information estimation. We first evaluate the effect of each component in mutual information estimation. The results are summarized in Tab. 1. Obviously, our model achieves a significantly higher AUC score by 0.843 with all mutual information estimation compared to that without any mutual information estimation (i.e. 0.75 AUC). This is due to the improvement of discriminative ability for the latent feature learned by the mutual information estimators. We also observe that global mutual information estimation (i.e. global loss Eq. (4)) achieves the best performance when using only one mutual information estimation loss. Note that training our model with two mutual information estimation losses achieves comparable results to the full three losses (see the 6th, 7th and 8th column in Tab. 1). Overall, three mutual information estimation losses indeed help to improve the performance of our model.

We further evaluate the mutual information estimation can help the model to obtain more discriminative feature representation by visualization. To this end, we randomly select 500 in-class samples (frog class) and 500 out-of-class samples from the dataset for testing 1 . As illustrated in Fig. 4, two global latent features are generated by our framework with/without mutual information estimation on CIFAR-10 by using t-SNE [van der Maaten and Hinton, 2008]. The normal samples (indicated by blue dots in Fig. 4 left) are significantly separated from novelty samples (indicated by yellow dots in Fig. 4 right) by using mutual information estimation, compared to that without this estimator.

1 We run data selection 5 times and obtain a similar visualization result. For simplicity, we select one of them for visualization.

The effect to remove zA or z. In local and global mutual information estimation, the concatenation of feature maps and latent vector are used as the inputs of their estimators during training. The discriminative feature representation is learned by distinguishing the concatenated features all from positive samples and part from negative samples. To evaluate the effectiveness of the concatenated features, we remove the information of latent matrix zA in the local estimator or latent vector z in the global estimator. In the first part of Tab. 2, the (first 3 rows) combination without any removal in our estimators achieves the highest AUC, compared to the removals of zA or z.

The effect of Laplacian pyramid loss. As shown in the second part of Tab. 2, the framework without any reconstruction loss results in the worst average AUC score of 0.53. We further compare the Laplacian pyramid loss with the MSE loss. As presented in the second part of Tab. 2, the performance of Laplacian pyramid loss significantly outperforms the MSE loss (i.e. 0.843 AUC vs. 0.810 AUC). To explain, Laplacian pyramid loss is a perception-level error, which is more effective for novelty detection than MSE loss.

The effect with/without PCA initialization. We compared our PCA initialization with normal distribution initialization and contrastive prior. Contrastive prior only uses the output z of the contrastive network at the 100th epoch training as initialization. As shown in the third part of Tab. 2, PCA initialization achieves the best performance, compared to other initialization methods. This is because PCA encodes the information of the entire training data, which alleviates the problem of mode dropping.

3.3 Comparison with State-of-the-art Methods

Novelty Detection

We first evaluate the effectiveness of our method on CIFAR10. As shown in Tab. 3, our method achieves the highest mean AUC of 0.843, compared to other SOTA methods. We also found that the self-supervised learning methods based on the pretext tasks (e.g., RotNet [Komodakis and Gidaris, 2018], Geometric [Hendrycks et al., 2019]) achieves higher performance, compared with GAN-based methods (e.g., OCGAN [Perera et al., 2019] and DualGAN [Chen et al., 2020a]). For the individual class, our method also shows the best results, except the class car, dog, horse and truck. The semantic information of these classes has high related to the object shape in CIFAR-10. Since the RotNet focuses on the semantic information of object shape, it achieves high performance in these classes. However, it lack capturing other semantic feature(e.g. object color and object texture).

Following [Perera et al., 2019], we also evaluate the performance of novelty detection methods on COIL100, MNIST, fMNIST by using the following evaluation setting: The 80% of in-class samples are regarded as a normal class for training, while the rest of 20% of in-class samples is adopt for testing. Out-of-class test samples have the same number of in-class test samples, which are randomly selected from the test set. As shown in Tab. 4, our method achieves the best performance across different datasets, compared to state-of-theart methods. For example, in MNIST dataset and fMNIST dataset, we achieve the improvement of average AUC score 0.1% and 0.4% respectively.

Acoustic Anomaly Detection

The original WaveNet [Oord et al., 2016] has been successfully applied into raw audio generation and music synthesis, which benefits from its powerful convolutional autoregressive architecture. Recently, [Rushe and Mac Namee, 2019] has extended its structure for anomaly detection in raw audio. Thus, we also make a comparison with it on the DCASE dataset, which is denoted by WaveNet for convenience. As shown in Tab. 5, we present the results of different methods across 15 classes/scenarios. Obviously, our method achieves the best performance in most of the scenarios, except the restaurant and home scenarios, compared to CAE, WaveNet, and DualGAN. For example, our method outperforms the best DualGAN by 8% AUC on library scenario. Note that DualGAN achieves amazing performance with 0.9 AUC. We conjecture this is due to the randomness of the DualGAN method in the home background. Interestingly, self-supervised learning based methods (e.g. RotNet [Komodakis and Gidaris, 2018] and Geometric [Hendrycks et al., 2019]) shows worse performance on DCASE. This is due to the failure to presentation of object colors and object textures. Actually, as shown in Fig. 5, the main discriminative features between normal and novelty audio in spectrogram are from object colors and textures. Rich semantic information of object shapes obtained by these methods cannot separate the normal samples from novelty samples.

Spoofing Face Detection

Our method also works for spoofing face detection. Actually, we formulate face anti-spoofing detection as a novelty detection task by only using the normal (live faces) samples for training. We intuitively take the optical flow obtained from consecutive video frames as input, since the data source of spoof face detection is from the video. To this end, we first extract the optical flow using FlowNet2.0 [Ilg et al., 2017] from each video with 30 fps.

Following [Tu et al., 2020], the detection models select the training set from one of the training set in CASIA-MFSD and Replay-Attack dataset, and the test set from the other test dataset. The results are summarized in Tab. 6. The proposed method achieves the best performance (HTER = 0.175) on the Replay-Attack test set of which includes different types of spoofing attacks. On the other dataset setting, our method achieves a competitive performance (HTER = 0.308) on the testing set of the CASIA-MFSD dataset. Actually, the HTER achieved by our unsupervised method significantly lower than supervision methods except auxiliary method [Liu et al., 2018]. This is probably due to the help of the additional depth information and rPPG signal. Nevertheless, our method achieves better performance by simultaneously evaluating these two tasks on the average HTER. We also found that the generalization of these models trained on the CASIA-MFSD dataset is better than the models trained on the Replay-Attack dataset. We speculate that the detection scenario on the CASIA-MFSD dataset is more complex than the Replay-Attack dataset, which leads to the easy learning of more knowledge during training.

3.4 Stability and Convergence

We further evaluate the effectiveness of our method during training. Our decoder-encoder framework achieves more stable training and fast convergence speed in CIFAR-10 dataset, compared to other encoder-decoder frameworks, e.g. OCGAN, ALOCC and DualGAN. As shown in Fig. 6, the proposed decoder-encoder framework tends to be convergent after the 25-th epoch, while the convergent value of other methods is significantly larger (e.g. about 40 epochs in OCGAN). In addition, the image reconstruction error is reduced steadily using the decoder-encoder framework, while the significant large fluctuations occur in other methods. This is due to the usage of the mutual information estimator that adversarial optimization is removed in our framework.

4 Conclusion

In this paper, we propose a novel decoder-encoder framework for novelty detection. To alleviate mode dropping of GANs, each latent initialized vector is mapped to an image by the PCA initialization in the generative network. To learn a more discriminative latent feature representation, we introduce a contrastive network to learn to compare the different input pairs through mutual information estimation. Specifically, our framework is trained without adversarial optimization, which benefits fast convergence and stable training of our model. We have comprehensively evaluated the performance of our method on a variety of novelty detection tasks over different datasets, which demonstrates the superior performance gains over the state-of-the-art methods.

Acknowledgments

We thank for the support from National Natural Science Foundation of China (61772524, 61902129, 61972157, 61876161, 61701235, 61373077), Shanghai Pujiang Talent Program (19PJ1403100), the Science and Technology Commission of Pudong (NO. PKJ2018-Y46) and Shanghai Jiaotong University Translational Medicine Cross Foundation (ZH2018ZDA25), Shanghai Sailing Program (21YF1411200), Natural Science Foundation of Shanghai (20ZR1417700), the Fundamental Research Funds for the Central Universities, CAAI-Huawei MindSpore Open Fund.