{"type":"ScholarlyArticle","title":"Logit Mixing Training for More Reliable and Accurate Prediction","authors":[{"name":"Duhyeon Bang","slug":"duhyeon-bang","given":"Duhyeon","family":"Bang","affiliation":""},{"name":"Kyungjune Baek","slug":"kyungjune-baek","given":"Kyungjune","family":"Baek","affiliation":"School of Integrated Technology , Yonsei University"},{"name":"Jiwoo Kim","slug":"jiwoo-kim","given":"Jiwoo","family":"Kim","affiliation":"Department of Software , Sungkyunkwan University"},{"name":"Yunho Jeon","slug":"yunho-jeon","given":"Yunho","family":"Jeon","affiliation":""},{"name":"Jin-Hwa Kim","slug":"jin-hwa-kim","given":"Jin-Hwa","family":"Kim","affiliation":"Department of Software , Sungkyunkwan University"},{"name":"Jiwon Kim","slug":"jiwon-kim","given":"Jiwon","family":"Kim","affiliation":"Department of Software , Sungkyunkwan University"},{"name":"Jongwuk Lee","slug":"jongwuk-lee","given":"Jongwuk","family":"Lee","affiliation":"Department of Software , Sungkyunkwan University"},{"name":"Hyunjung Shim","slug":"hyunjung-shim","given":"Hyunjung","family":"Shim","affiliation":"Kim Jaechul Graduate School of AI , KAIST"}],"abstract":"When a person solves the multi-choice problem, she considers not only what is the answer but also what is not the answer. Knowing what choice is not the answer and utilizing the relationships between choices, she can improve the prediction accuracy. Inspired by this human reasoning process, we propose a new training strategy to fully utilize inter-class relationships, namely LogitMix. Our strategy is combined with recent data augmentation techniques, e.g., Mixup, Manifold Mixup, CutMix, and PuzzleMix. Then, we suggest using a mixed logit, i.e., a mixture of two logits, as an auxiliary training objective. Since the logit can preserve both positive and negative inter-class relationships, it can impose a network to learn the probability of wrong answers correctly. Our extensive experimental results on the image- and language-based tasks demonstrate that LogitMix achieves state-of-the-art performance among recent data augmentation techniques regarding calibration error and prediction accuracy.","doi":"10.24963/ijcai.2022/390","url":"https://ijcai.org/proceedings/2022/390","pdf":"https://ijcai.org/proceedings/2022/0390.pdf","markdown":"https://ijcai.org/proceedings/2022/390.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/390","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":390,"pages":{"first":2812,"last":2819},"keywords":["Machine Learning: Robustness","Machine Learning: Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986679","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984822","length_sec":301,"provider":"slideslive"}],"references":[{"position":1,"raw":"4 Proof-of-Concept Study 4.1 Toy Example We conducted a 2D classification task using the two-class spiral dataset [","title":"4 Proof-of-Concept Study 4.1 Toy Example We conducted a 2D classification task using the two-class spiral dataset","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Verma et al., 2019] to reveal the impact of LogitMix. To show the decision boundary and hidden rep- resentation of each method, we visualize the predictions in References [","title":"impact of LogitMix","authors":"Verma","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2015] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kil- ian Q Weinberger. On calibration of modern neural net- works. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.","title":"On calibration of modern neural networks","authors":"Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q Weinberger","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Heo et al., 2019] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In ICCV, pages 1921-1930, 2019.","title":"A comprehensive overhaul of feature distillation","authors":"Heo","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2014] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. In NIPS 2014 Deep Learning Workshop, 2014.","title":"Distilling the Knowledge in a Neural Network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2014","venue":"NIPS 2014 Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2020] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICML, 2020.","title":"Puzzle mix: Exploiting saliency and local statistics for optimal mixup","authors":"Jang-Hyun Kim, Wonho Choo, Hyun Oh Song","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Mas- ter's thesis, Department of Computer Science, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lakshminarayanan et al., 2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensem- bles. In NeurIPS, pages 6402-6413, 2017.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Maki, 2018] Vladimir Li and Atsuto Maki. Feature contraction: New convnet regularization in image classifi- cation. In BMVC, page 213, 2018.","title":"Feature contraction: New convnet regularization in image classification","authors":"Li, Maki ; ] Vladimir Li, Atsuto Maki","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2018] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, pages 116-131, 2018.","title":"Shufflenet v2: Practical guidelines for efficient cnn architecture design","authors":"Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, Jian Sun","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[MacKay, 1992] David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448-472, 1992.","title":"A practical bayesian framework for backpropagation networks","authors":"J C David, Mackay","year":"1992","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NeurIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Müller et al., 2019] Rafael Müller, Simon Kornblith, and Ge- offrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906.02629, 2019.","title":"When does label smoothing help","authors":"Müller","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Naeini et al., 2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.","title":"Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning","authors":"Naeini","year":"2015","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pereyra et al., 2017] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Reg- ularizing neural networks by penalizing confident output distributions. In ICLR, 2017.","title":"Regularizing neural networks by penalizing confident output distributions","authors":"Pereyra","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bern- stein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211- 252, 2015.","title":"ImageNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Seo et al., 2019] Seonguk Seo, Paul Hongsuck Seo, and Bo- hyung Han. Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. In CVPR, pages 9030-9038, 2019.","title":"Learning for single-shot confidence calibration in deep neural networks through stochastic inferences","authors":"Seo","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2020] Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S Yu, and Lifang He. Mixup- transformer: Dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394, 2020.","title":"Mixuptransformer: Dynamic data augmentation for nlp tasks","authors":"Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thulasidasan et al., 2019] Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, 2019.","title":"On mixup training: Improved calibration and predictive uncertainty for deep neural networks","authors":"Gopinath Sunil Thulasidasan, Jeff Chennupati, Tanmoy Bilmes, Sarah Bhattacharya, Michalak","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Verma et al., 2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better repre- sentations by interpolating hidden states. In ICML, pages 6438-6447. PMLR, 2019.","title":"Manifold mixup: Better representations by interpolating hidden states","authors":"Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, Yoshua Bengio","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.07461, 2018.","title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R Bowman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, pages 1492-1500, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, David Lopez-Paz","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}