{"type":"ScholarlyArticle","title":"Learning Sparse Neural Networks for Better Generalization","authors":[{"name":"Shiwei Liu","slug":"shiwei-liu","given":"Shiwei","family":"Liu","affiliation":"Eindhoven University of Technology , The Netherlands"}],"abstract":"Deep neural networks perform well on test data when they are highly overparameterized, which, however, also leads to large cost to train and deploy them. As a leading approach to address this problem, sparse neural networks have been widely used to significantly reduce the size of networks, making them more efficient during training and deployment, without compromising performance. Recently, sparse neural networks, either compressed from a pre-trained model or obtained by training from scratch, have been observed to be able to generalize as well as or even better than their dense counterparts. However, conventional techniques to find well fitted sparse sub-networks are expensive and the mechanisms underlying this phenomenon are far from clear. To tackle these problems, this Ph.D. research aims to study the generalization of sparse neural networks, and to propose more efficient approaches that can yield sparse neural networks with generalization bounds.","doi":"10.24963/ijcai.2020/735","url":"https://ijcai.org/proceedings/2020/735","pdf":"https://ijcai.org/proceedings/2020/0735.pdf","markdown":"https://ijcai.org/proceedings/2020/735.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/735","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":735,"pages":{"first":5190,"last":5191},"keywords":["Machine Learning: Cost-Sensitive Learning","Machine Learning: Deep Learning","Machine Learning: Feature Selection; Learning Sparse Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24991/file_24991.m3u8","length_sec":318,"provider":"hls"}],"references":[{"position":1,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural informa- tion processing systems, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Keskar et al., 2016] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.","title":"On large-batch training for deep learning: Generalization gap and sharp minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[LeCun et al., 1990] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neu- ral information processing systems, pages 598-605, 1990.","title":"Optimal brain damage","authors":"Yann Lecun, John S Denker, Sara A Solla","year":"1990","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Liu et al., 2019a] Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learn- ing with over one million artificial neurons on commodity hardware. arXiv preprint arXiv:1901.09181, 2019.","title":"Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Liu et al., 2019b] Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Intrinsically sparse long short- term memory networks. arXiv preprint arXiv:1901.09208, 2019.","title":"Intrinsically sparse long shortterm memory networks","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Liu et al., 2019c] Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. On improving deep learning generalization with adaptive sparse connectivity. In ICML Workshop on Understanding and Improving Generaliza- tion in Deep Learning, 2019.","title":"On improving deep learning generalization with adaptive sparse connectivity","authors":"Liu","year":"2019","venue":"ICML Workshop on Understanding and Improving Generalization in Deep Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Merity et al., 2017] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm lan- guage models. arXiv preprint arXiv:1708.02182, 2017.","title":"Regularizing and optimizing lstm language models","authors":"Stephen Merity, Nitish Shirish Keskar, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mocanu et al., 2018] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.","title":"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science","authors":"Mocanu","year":"2018","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mostafa and Wang, 2019] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Pro- ceedings of the 36th International Conference on Machine Learning-Volume 97, pages 4646-4655. JMLR. org, 2019.","title":"Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization","authors":"Hesham Mostafa, Xin Wang","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}