{"type":"ScholarlyArticle","title":"Classification with Rejection: Scaling Generative Classifiers with Supervised Deep Infomax","authors":[{"name":"Xin Wang","slug":"xin-wang","given":"Xin","family":"Wang","affiliation":"The University of Hong Kong"},{"name":"Siu Ming Yiu","slug":"siu-ming-yiu","given":"Siu Ming","family":"Yiu","affiliation":"The University of Hong Kong"}],"abstract":"Deep Infomax (DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs. In this paper, we propose Supervised Deep InfoMax (SDIM), which introduces supervised probabilistic constraints to the encoder outputs. The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated. Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts.  With SDIM, we could perform classification with rejection.\r\nInstead of always reporting a class label, SDIM only makes predictions when test samples' largest class conditional surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected.  Our experiments show that SDIM with rejection policy can effectively reject illegal inputs, including adversarial examples and out-of-distribution samples.","doi":"10.24963/ijcai.2020/412","url":"https://ijcai.org/proceedings/2020/412","pdf":"https://ijcai.org/proceedings/2020/0412.pdf","markdown":"https://ijcai.org/proceedings/2020/412.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/412","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":412,"pages":{"first":2980,"last":2986},"keywords":["Machine Learning: Classification","Multidisciplinary Topics and Applications: Security and Privacy","Machine Learning: Learning Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25664/file_25664.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Athalye et al., 2018] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.","title":"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples","authors":"Anish Athalye, Nicholas Carlini, David Wagner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aris- tide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Ben- gio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062, 2018.","title":"Mohamed Ishmael Belghazi, Aristide Baratin","authors":"Belghazi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brendel and Bethge, 2019] Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet. In Interna- tional Conference on Learning Representations, 2019.","title":"Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet","authors":"Wieland Brendel, Matthias Bethge","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brendel et al., 2017] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Re- liable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04248, 2017.","title":"Decision-based adversarial attacks: Reliable attacks against black-box machine learning models","authors":"Wieland Brendel, Jonas Rauber, Matthias Bethge","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carlini and Wagner, 2016] Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv:1607.04311, 2016.","title":"Defensive distillation is not robust to adversarial examples","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi and Jang, 2018] Hyunsun Choi and Eric Jang. Gen- erative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.","title":"Generative ensembles for robust anomaly detection","authors":"Jang Choi, Hyunsun Choi, Eric Jang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donsker and Varadhan, 1983] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212, 1983.","title":"Asymptotic evaluation of certain markov process expectations for large time","authors":"D Monroe, Donsker, Sr Srinivasa Varadhan","year":"1983","venue":"Communications on Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fetaya et al., 2019] Ethan Fetaya, Jacobsen, and Richard Zemel. Conditional generative models are not robust. arXiv preprint arXiv:1906.01171, 2019.","title":"Conditional generative models are not robust","authors":"Fetaya","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Geifman and El-Yaniv, 2017] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems, pages 4878-4887, 2017.","title":"Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks","authors":"Geifman, El-Yaniv","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gilmer et al., 2018] Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774, 2018.","title":"Adversarial spheres","authors":"Justin Gilmer, Luke Metz, Fartash Faghri, Maithra Samuel S Schoenholz, Martin Raghu, Ian Wattenberg, Goodfellow","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu and Rigazio, 2014] Shixiang Gu and Luca Rigazio. To- wards deep neural network architectures robust to adver- sarial examples. arXiv preprint arXiv:1412.5068, 2014.","title":"Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples","authors":"Rigazio Gu","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Hjelm","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers more robust to adver- sarial attacks? arXiv preprint arXiv:1802.06552, 2018.","title":"Are generative classifiers more robust to adversarial attacks","authors":"Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Madry et al., 2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nalisnick et al., 2018] Eric Nalisnick, Akihiro Matsukawa, and et al. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136, 2018.","title":"Do deep generative models know what they don't know","authors":"Eric Nalisnick, Akihiro Matsukawa","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nalisnick et al., 2019] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshmi- narayanan. Hybrid models with deep and invertible fea- tures. arXiv preprint arXiv:1902.02767, 2019.","title":"Hybrid models with deep and invertible features","authors":"Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Narodytska and Kasiviswanathan, 2016] Nina Narodytska and Shiva Prasad Kasiviswanathan. Simple black-box ad- versarial perturbations for deep networks. arXiv preprint arXiv:1612.06299, 2016.","title":"Simple black-box adversarial perturbations for deep networks","authors":"Nina Narodytska, Shiva Prasad Kasiviswanathan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ng and Jordan, 2002] Andrew Y Ng and Michael I Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Advances in neu- ral information processing systems, pages 841-848, 2002.","title":"On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes","authors":"Y Andrew, Michael I Ng, Jordan","year":"2002","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen et al., 2010] XuanLong Nguyen, Martin J Wain- wright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk min- imization. IEEE Transactions on Information Theory, 56(11):5847-5861, 2010.","title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","authors":"Nguyen","year":"2010","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nowozin et al., 2016] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pages 271-279, 2016.","title":"f-gan: Training generative neural samplers using variational divergence minimization","authors":"Sebastian Nowozin, Botond Cseke, Ryota Tomioka","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salimans et al., 2017] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.","title":"Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications","authors":"Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P Kingma","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Samangouei et al., 2018] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605, 2018.","title":"Defense-gan: Protecting classifiers against adversarial attacks using generative models","authors":"Pouya Samangouei, Maya Kabkab, Rama Chellappa","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schott and Rauber, 2018] Lukas Schott and Jonas et al Rauber. Towards the first adversarially robust neural net- work model on mnist. arXiv preprint arXiv:1805.09190, 2018.","title":"Towards the first adversarially robust neural network model on mnist","authors":"Rauber Schott, Lukas Schott, Jonas","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}