{"type":"ScholarlyArticle","title":"A Walkthrough for the Principle of Logit Separation","authors":[{"name":"Gil Keren","slug":"gil-keren","given":"Gil","family":"Keren","affiliation":"Chair of Embedded Intelligence for Health Care and Wellbeing , University of Augsburg , Germany"},{"name":"Sivan Sabato","slug":"sivan-sabato","given":"Sivan","family":"Sabato","affiliation":"Department of Computer Science , Ben-Gurion University of the Negev , Beer-Sheva , Israel"},{"name":"Björn Schuller","slug":"bjorn-schuller","given":"Björn","family":"Schuller","affiliation":"GLAM -Group on Language , Audio & Music , Imperial College London , UK"}],"abstract":"We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is a binary classification task of determining whether the given example belongs to a specific class. We define the Single Logit Classification (SLC) task: training the network so that at test-time, it would be possible to accurately identify whether the example belongs to a given class in a computationally efficient manner, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing loss functions that are suitable for SLC. We show that the Principle of Logit Separation is a crucial ingredient for success in the SLC task, and that SLC results in considerable speedups when the number of classes is large.","doi":"10.24963/ijcai.2019/861","url":"https://ijcai.org/proceedings/2019/861","pdf":"https://ijcai.org/proceedings/2019/0861.pdf","markdown":"https://ijcai.org/proceedings/2019/861.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/861","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":861,"pages":{"first":6191,"last":6195},"keywords":["Machine Learning: Classification","Machine Learning Applications: Big data ; Scalability","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Crammer and Singer, 2001] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learn- ing Research, 2:265-292, 2001.","title":"On the algorithmic implementation of multiclass kernel-based vector machines","authors":"Koby Crammer, Yoram Singer","year":"2001","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2014] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M. Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Lin- guistics, ACL, pages 1370-1380, Baltimore, MD, 2014.","title":"Fast and robust neural network joint models for statistical machine translation","authors":"Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M Schwartz, John Makhoul","year":"2014","venue":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Grave et al., 2017] Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. Effi- cient softmax approximation for GPUs. In Proceedings of ICML, pages 1302-1310, Sydney, Australia, 2017.","title":"Efficient softmax approximation for GPUs","authors":"Grave","year":"2017","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gutmann and Hyvärinen, 2010] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of AISTATS, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 2010.","title":"Noise-contrastive estimation: A new estimation principle for unnormalized statistical models","authors":"Hyvärinen Gutmann, Michael Gutmann, Aapo Hyvärinen","year":"2010","venue":"Proceedings of AISTATS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2013] Yan Huang, Wei Wang, Liang Wang, and Tieniu Tan. Multi-task deep neural network for multi- label learning. In Procedding of IEEE International Con- ference on Image Processing, ICIP, pages 2897-2900, Melbourne, Australia, 2013.","title":"Multi-task deep neural network for multilabel learning","authors":"Yan Huang, Wei Wang, Liang Wang, Tieniu Tan","year":"2013","venue":"Procedding of IEEE International Conference on Image Processing, ICIP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Janocha and Czarnecki, 2017] Katarzyna Janocha and Woj- ciech Marian Czarnecki. On loss functions for deep neural networks in classification. Arxiv, abs/1702.05659, 2017.","title":"On loss functions for deep neural networks in classification","authors":"Katarzyna Janocha, Wojciech Marian Czarnecki","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Keren et al., 2017] Gil Keren, Sivan Sabato, and Björn Schuller. Tunable sensitivity to large errors in neural net- work training. In Proceedings of AAAI, pages 2087-2093, San Francisco, CA, 2017.","title":"Tunable sensitivity to large errors in neural network training","authors":"Gil Keren, Sivan Sabato, Björn Schuller","year":"2017","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Keren et al., 2018a] Gil Keren, Nicholas Cummins, and Björn W. Schuller. Calibrated prediction intervals for neu- ral network regressors. IEEE Access, 6:54033-54041, 2018.","title":"Calibrated prediction intervals for neu-ral network regressors","authors":"Keren","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Keren et al., 2018b] Gil Keren, Jing Han, and Björn Schuller. Scaling speech enhancement in unseen environ- ments with noise embeddings. In In Proceedings of the CHiME Workshop on Speech Processing in Everyday En- vironments, pages 25-29, Hyderabad, India, 2018.","title":"Scaling speech enhancement in unseen environments with noise embeddings","authors":"Keren","year":"2018","venue":"Proceedings of the CHiME Workshop on Speech Processing in Everyday Environments","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Keren et al., 2018c] Gil Keren, Sivan Sabato, and Björn Schuller. Fast single-class classification and the princi- ple of logit separation. In Proceedings of the International Conference on Data Mining (ICDM), pages 227-236, Sin- gapore, 2018.","title":"Fast single-class classification and the principle of logit separation","authors":"Keren","year":"2018","venue":"Proceedings of the International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of NIPS, pages 1106-1114, Lake Tahoe, NV, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maturana and Scherer, 2015] Daniel Maturana and Sebas- tian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Proceedings of IROS, pages 922-928, Hamburg, Germany, 2015.","title":"Voxnet: A 3d convolutional neural network for real-time object recognition","authors":"Scherer Maturana, Daniel Maturana, Sebastian Scherer","year":"2015","venue":"Proceedings of IROS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih and Teh, 2012] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. In Proceedings ICML, Edinburgh, Scot- land, UK, 2012.","title":"A fast and simple algorithm for training neural probabilistic language models","authors":"Andriy Mnih, Yee Whye Teh","year":"2012","venue":"Proceedings ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS workshop on deep learning and unsupervised feature learning. Granada, Spain, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS workshop on deep learning and unsupervised feature learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Kumar Div- vala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of CVPR, pages 779-788, Las Vegas, NV, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Kumar Divvala, Ross B Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Socher et al., 2011] Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing nat- ural scenes and natural language with recursive neural net- works. In Proceedings of ICML, pages 129-136, Bellevue, WA, 2011.","title":"Parsing natural scenes and natural language with recursive neural networks","authors":"Richard Socher, Cliff Chiung-Yu Lin, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceddings of CVPR, pages 2818-2826, Las Vegas, NV, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"Proceddings of CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Jiang Wang, Yi Yang, Junhua Mao, Zhi- heng Huang, Chang Huang, and Wei Xu. CNN-RNN: A unified framework for multi-label image classification. In Proceedings of CVPR, pages 2285-2294, Las Vegas, NV, 2016.","title":"CNN-RNN: A unified framework for multi-label image classification","authors":"Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, Wei Xu","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}