{"type":"ScholarlyArticle","title":"P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection","authors":[{"name":"Zhiwei Zhang","slug":"zhiwei-zhang","given":"Zhiwei","family":"Zhang","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences , China; School of Information and Electronics , Beijing Institute of Technology , China; Multimedia Laboratory of SIAT."},{"name":"Shifeng Chen","slug":"shifeng-chen","given":"Shifeng","family":"Chen","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences , China"},{"name":"Lei Sun","slug":"lei-sun","given":"Lei","family":"Sun","affiliation":"School of Information and Electronics , Beijing Institute of Technology , China"}],"abstract":"One-class novelty detection is to identify anomalous instances that do not conform to the expected normal instances. In this paper, the Generative Adversarial Networks (GANs) based on encoder-decoder-encoder pipeline are used for detection and achieve state-of-the-art performance. However, deep neural networks are too over-parameterized to deploy on resource-limited devices. Therefore, Progressive Knowledge Distillation with GANs (P-KDGAN) is proposed to learn compact and fast novelty detection networks. The P-KDGAN is a novel attempt to connect two standard GANs by the designed distillation loss for transferring knowledge from the teacher to the student. The progressive learning of knowledge distillation is a two-step approach that continuously improves the performance of the student GAN and achieves better performance than single step methods. In the first step, the student GAN learns the basic knowledge totally from the teacher via guiding of the pre-trained teacher GAN with fixed weights. In the second step, joint fine-training is adopted for the knowledgeable teacher and student GANs to further improve the performance and stability. The experimental results on CIFAR-10, MNIST, and FMNIST show that our method improves the performance of the student GAN by 2.44%, 1.77%, and 1.73% when compressing the computation at ratios of 24.45:1, 311.11:1, and 700:1, respectively.","doi":"10.24963/ijcai.2020/448","url":"https://ijcai.org/proceedings/2020/448","pdf":"https://ijcai.org/proceedings/2020/0448.pdf","markdown":"https://ijcai.org/proceedings/2020/448.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/448","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":448,"pages":{"first":3237,"last":3243},"keywords":["Machine Learning: Adversarial Machine Learning","Machine Learning: Deep Learning","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25559/file_25559.m3u8","length_sec":293,"provider":"hls"}],"references":[{"position":1,"raw":"[Abati et al., 2019] Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregres- sion for novelty detection. CVPR, pages 481-490, 2019.","title":"Latent space autoregression for novelty detection","authors":"Davide Abati, Angelo Porrello, Simone Calderara, Rita Cucchiara","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akc ¸ay et al., 2018] Samet Akc ¸ay, Amir Atapour Abarghouei, and Toby P. Breckon. Ganomaly: Semi- supervised anomaly detection via adversarial training. ACCV, 2018.","title":"Ganomaly: Semisupervised anomaly detection via adversarial training","authors":"Amir Samet Akc ¸ay, Toby P Atapour Abarghouei, Breckon","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chandola et al., 2009] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection : A survey. ACM Comput. Surv., 41(3):1-72, 2009.","title":"Anomaly detection : A survey","authors":"Varun Chandola, Arindam Banerjee, Vipin Kumar","year":"2009","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2017] Y. Cheng, D. Wang, P. Zhou, and T. Zhang. A survey of model compression and acceler- ation for deep neural networks. ArXiv, abs/1710.09282, 2017.","title":"A survey of model compression and acceleration for deep neural networks","authors":"Y Cheng, D Wang, P Zhou, T Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue et al., 2017] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erfani et al., 2016] Sarah M. Erfani, Sutharshan Ra- jasegarar, Shanika Karunasekera, and Christopher Leckie. High-dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recognition, 58:121-134, 2016.","title":"High-dimensional and large-scale anomaly detection using a linear one-class svm with deep learning","authors":"Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, Christopher Leckie","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[García-Teodoro et al., 2009] Pedro García-Teodoro, Jesús E. Díaz-Verdejo, Gabriel Maciá-Fernández, and Enrique Vázquez. Anomaly-based network intrusion detection: Techniques, systems and challenges. Comput. Secur., 28(1-2):18-28, 2009.","title":"Anomaly-based network intrusion detection: Techniques, systems and challenges","authors":"Pedro García-Teodoro, Jesús E Díaz-Verdejo, Gabriel Maciá-Fernández, Enrique Vázquez","year":"2009","venue":"Comput. Secur","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton et al., 2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural net- work. ArXiv, abs/1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey E Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun and Cortes, 2005] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.","title":"The mnist database of handwritten digits","authors":"Lecun, Cortes, Yann Lecun, Corinna Cortes","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018] Peiye Liu, Wu Liu, Huadong Ma, Tao Mei, and Mingoo Seok. Ktan: knowledge transfer adversarial network. ArXiv, abs/1810.08126, 2018.","title":"Ktan: knowledge transfer adversarial network","authors":"Peiye Liu, Wu Liu, Huadong Ma, Tao Mei, Mingoo Seok","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Parzen, 1962] Emanuel Parzen. On estimation of a proba- bility density function and mode. Annals of Mathematical Statistics, 33(3):1065-1076, 1962.","title":"On estimation of a probability density function and mode","authors":"Emanuel Parzen","year":"1962","venue":"Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NeurIPS, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perera et al., 2019] Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection us- ing gans with constrained latent representations. CVPR, pages 2893-2901, 2019.","title":"Ocgan: One-class novelty detection using gans with constrained latent representations","authors":"Perera","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2016] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2016.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. ICLR, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ruff et al., 2018] Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. ICML, 2018.","title":"Deep one-class classification","authors":"Lukas Ruff, Nico Görnitz, Lucas Deecke, Ahmed Shoaib, Robert A Siddiqui, Alexander Vandermeulen, Emmanuel Binder, Marius Müller, Kloft","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schlegl et al., 2017] Thomas Schlegl, Philipp Seeböck, Se- bastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. IPMI, 2017.","title":"Unsupervised anomaly detection with generative adversarial networks to guide marker discovery","authors":"Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, Georg Langs","year":"2017","venue":"IPMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schölkopf et al., 2001] Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alexander J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471, 2001.","title":"Estimating the support of a high-dimensional distribution","authors":"Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alexander J Smola, Robert C Williamson","year":"2001","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Srivastava et al., 2008] Abhinav Srivastava, Amlan Kundu, Shamik Sural, and Arun K. Majumdar. Credit card fraud detection using hidden markov model. IEEE Transactions on Dependable and Secure Computing, 5(1):37-48, 2008.","title":"Credit card fraud detection using hidden markov model","authors":"Abhinav Srivastava, Amlan Kundu, Shamik Sural, Arun K Majumdar","year":"2008","venue":"IEEE Transactions on Dependable and Secure Computing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extract- ing and composing robust features with denoising autoen- coders. ICML, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018a] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: knowledge distillation with genera- tive adversarial networks. NeurIPS, 2018.","title":"Kdgan: knowledge distillation with generative adversarial networks","authors":"Wang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018b] Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Adversarial learning of portable student net- works. AAAI, 2018.","title":"Adversarial learning of portable student networks","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wold et al., 1987] Svante Wold, Kim H. Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3):37-52, 1987.","title":"Principal component analysis","authors":"Svante Wold, Kim H Esbensen, Paul Geladi","year":"1987","venue":"Chemometrics and Intelligent Laboratory Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning. ArXiv, abs/1708.07747, 2017.","title":"Fashion-mnist: A novel image dataset for benchmarking machine learning","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zenati et al., 2018] Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. ArXiv, abs/1802.06222, 2018.","title":"Efficient gan-based anomaly detection","authors":"Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, Vijay Ramaseshan Chandrasekhar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}