{"type":"ScholarlyArticle","title":"Towards Robust ResNet: A Small Step but a Giant Leap","authors":[{"name":"Jingfeng Zhang","slug":"jingfeng-zhang","given":"Jingfeng","family":"Zhang","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Bo Han","slug":"bo-han","given":"Bo","family":"Han","affiliation":"RIKEN Center for Advanced Intelligence Project"},{"name":"Laura Wynter","slug":"laura-wynter","given":"Laura","family":"Wynter","affiliation":"IBM Research , Singapore"},{"name":"Bryan Kian Hsiang Low","slug":"bryan-kian-hsiang-low","given":"Bryan Kian Hsiang","family":"Low","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Mohan Kankanhalli","slug":"mohan-kankanhalli","given":"Mohan","family":"Kankanhalli","affiliation":"Department of Computer Science , National University of Singapore"}],"abstract":"This paper presents a simple yet principled approach to boosting the robustness of the residual network (ResNet) that is motivated by a dynamical systems perspective. Namely, a deep neural network can be interpreted using a partial differential equation, which naturally inspires us to characterize ResNet based on an explicit Euler method. This consequently allows us to exploit the step factor h in the Euler method to control the robustness of ResNet in both its training and generalization. In particular, we prove that a small step factor h can benefit its training and generalization robustness during backpropagation and forward propagation, respectively. Empirical evaluation on real-world datasets corroborates our analytical findings that a small h can indeed improve both its training and generalization robustness.","doi":"10.24963/ijcai.2019/595","url":"https://ijcai.org/proceedings/2019/595","pdf":"https://ijcai.org/proceedings/2019/0595.pdf","markdown":"https://ijcai.org/proceedings/2019/595.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/595","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":595,"pages":{"first":4285,"last":4291},"keywords":["Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Ascher, 2008] Uri M. Ascher. Numerical methods for evolu- tionary differential equations, volume 5 of Computational Science and Engineering. SIAM, 2008.","title":"Numerical methods for evolutionary differential equations","authors":"Uri M Ascher","year":"2008","venue":"Computational Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atkinson, 2008] Kendall E. Atkinson. An introduction to numerical analysis. John Wiley & Sons, 2nd edition, 2008.","title":"An introduction to numerical analysis","authors":"Kendall E Atkinson","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Butcher, 2016] John Charles Butcher. Numerical methods for ordinary differential equations. John Wiley & Sons, 2016.","title":"Numerical methods for ordinary differential equations","authors":"John Charles Butcher","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary dif- ferential equations. In Proc. NeurIPS, pages 6572-6583, 2018.","title":"Neural ordinary differential equations","authors":"Tian Qi, Chen, Yulia Rubanova, Jesse Bettencourt, David K Duvenaud","year":"2018","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proc. EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2017] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. Very deep convolutional networks for text classification. In Proc. EACL, 2017.","title":"Very deep convolutional networks for text classification","authors":"Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun","year":"2017","venue":"Proc. EACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dai et al., 2019] Zhongxiang Dai, Haibin Yu, Bryan Kian Hsiang Low, and Patrick Jaillet. Bayesian optimiza- tion meets Bayesian optimal stopping. In Proc. ICML, pages 1496-1506, 2019.","title":"Bayesian optimization meets Bayesian optimal stopping","authors":"Zhongxiang Dai, Haibin Yu, Bryan Kian Hsiang Low, Patrick Jaillet","year":"2019","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Daxberger and Low, 2017] Erik A. Daxberger and Bryan Kian Hsiang Low. Distributed batch Gaussian process op- timization. In Proc. ICML, pages 951-960, 2017.","title":"Distributed batch Gaussian process optimization","authors":"Erik A Daxberger, Bryan Kian, Hsiang Low","year":"2017","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proc. AISTATS, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proc. AISTATS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haber and Ruthotto, 2017] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.","title":"Stable architectures for deep neural networks","authors":"Eldad Haber, Lars Ruthotto","year":"2017","venue":"Inverse Problems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proc. NeurIPS, pages 8536-8546, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama","year":"2018","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proc. ECCV, pages 630-645, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hoang et al., 2018] Trong Nghia Hoang, Quang Minh Hoang, Ruofei Ouyang, and Kian Hsiang Low. Decentral- ized high-dimensional Bayesian optimization with factor graphs. In Proc. AAAI, pages 3231-3238, 2018.","title":"Decentralized high-dimensional Bayesian optimization with factor graphs","authors":"Trong Nghia Hoang, Quang, Minh Hoang, Ruofei Ouyang, Kian Hsiang Low","year":"2018","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. Proc. IEEE CVPR, pages 2261- 2269, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proc. IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In Proc. ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ketkar, 2017] Nikhil Ketkar. Deep Learning with Python: A Hands-on Introduction. Apress, 2017.","title":"Deep Learning with Python: A Hands-on Introduction","authors":"Nikhil Ketkar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey E. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Univ. Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey E Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lei Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ling et al., 2016] Chun Kai Ling, Kian Hsiang Low, and Patrick Jaillet. Gaussian process planning with Lipschitz continuous reward functions: Towards unifying Bayesian optimization, active learning, and beyond. In Proc. AAAI, 2016.","title":"Gaussian process planning with Lipschitz continuous reward functions: Towards unifying Bayesian optimization, active learning, and beyond","authors":"Chun Kai, Ling, Kian Hsiang Low, Patrick Jaillet","year":"2016","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2018] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridg- ing deep architectures and numerical differential equa- tions. In Proc. ICML, pages 3282-3291, 2018.","title":"Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations","authors":"Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong","year":"2018","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann ma- chines. In Proc. ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted Boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reed and Marks II, 1999] Russell Reed and Robert J. Marks II. Neural Smithing: Supervised Learning in Feed- forward Artificial Neural Networks. MIT Press, 1999.","title":"Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks","authors":"Reed Marks, I I, Russell Reed, Robert J Marks, I I","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ruthotto and Haber, 2018] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. arXiv:1804.04272, 2018.","title":"Deep neural networks motivated by partial differential equations","authors":"Haber Ruthotto, Lars Ruthotto, Eldad Haber","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Santurkar et al., 2018] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Proc. NeurIPS, pages 2488-2498, 2018.","title":"How does batch normalization help optimization?","authors":"Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry","year":"2018","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Weinan, 2017] E. Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.","title":"A proposal on machine learning via dynamical systems","authors":"E Weinan","year":"2017","venue":"Communications in Mathematics and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu and He, 2018] Yuxin Wu and Kaiming He. Group nor- malization. In Proc. ECCV, pages 3-19, 2018.","title":"Group normalization","authors":"He Wu, Yuxin Wu, Kaiming He","year":"2018","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proc. NeurIPS, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Yehong Zhang, Trong Nghia Hoang, Bryan Kian Hsiang Low, and Mohan Kankanhalli. Information-based multi-fidelity Bayesian optimization. In NIPS Workshop on Bayesian Optimization, 2017.","title":"Information-based multi-fidelity Bayesian optimization","authors":"Zhang","year":"2017","venue":"NIPS Workshop on Bayesian Optimization","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2019a] Jingfeng Zhang, Bo Han, Laura Wyn- ter, Kian Hsiang Low, and Mohan Kankanhalli. To- wards robust ResNet: A small step but a giant leap. arXiv:1902.10887, 2019.","title":"Towards robust ResNet: A small step but a giant leap","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2019b] Yehong Zhang, Zhongxiang Dai, and Bryan Kian Hsiang Low. Bayesian optimization with bi- nary auxiliary information. In Proc. UAI, 2019.","title":"Bayesian optimization with binary auxiliary information","authors":"Zhang","year":"2019","venue":"Proc. UAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}