{"type":"ScholarlyArticle","title":"Submodular Batch Selection for Training Deep Neural Networks","authors":[{"name":"K J Joseph","slug":"k-j-joseph","given":"K J","family":"Joseph","affiliation":"Indian Institute of Technology , Hyderabad"},{"name":"Vamshi Teja R","slug":"vamshi-teja-r","given":"Vamshi","family":"Teja R","affiliation":"Indian Institute of Technology , Hyderabad"},{"name":"Krishnakant Singh","slug":"krishnakant-singh","given":"Krishnakant","family":"Singh","affiliation":"Indian Institute of Technology , Hyderabad"},{"name":"Vineeth N Balasubramanian","slug":"vineeth-n-balasubramanian","given":"Vineeth N","family":"Balasubramanian","affiliation":"Indian Institute of Technology , Hyderabad"}],"abstract":"Mini-batch gradient descent based methods are the de facto algorithms for training neural network architectures today.We introduce a mini-batch selection strategy based on submodular function maximization. Our novel submodular formulation captures the informativeness of each sample and diversity of the whole subset. We design an efficient, greedy algorithm which can give high-quality solutions to this NP-hard combinatorial optimization problem. Our extensive experiments on standard datasets show that the deep models trained using the proposed batch selection strategy provide better generalization than Stochastic Gradient Descent as well as a popular baseline sampling strategy across different learning rates, batch sizes, and distance metrics.","doi":"10.24963/ijcai.2019/372","url":"https://ijcai.org/proceedings/2019/372","pdf":"https://ijcai.org/proceedings/2019/0372.pdf","markdown":"https://ijcai.org/proceedings/2019/372.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/372","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":372,"pages":{"first":2677,"last":2683},"keywords":["Machine Learning: Classification","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Alain et al., 2015] Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduc- tion in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015.","title":"Variance reduction in sgd by distributed importance sampling","authors":"Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu, 2017] Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. The Journal of Ma- chine Learning Research, 18(1):8194-8244, 2017.","title":"Katyusha: The first direct acceleration of stochastic gradient methods","authors":"Allen-Zhu ; Zeyuan Allen-Zhu","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brahma and Othon, 2018] Pratik Prabhanjan Brahma and Adri- enne Othon. Subset replay based continual learning for scal- able improvement of autonomous systems. In CVPR Workshops, pages 1179-11798. IEEE, 2018.","title":"Pratik Prabhanjan Brahma and Adrienne Othon. Subset replay based continual learning for scalable improvement of autonomous systems","authors":"Othon Brahma","year":"2018","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chakraborty et al., 2015] Shayok Chakraborty, Vineeth Balasub- ramanian, and Sethuraman Panchanathan. Adaptive batch mode active learning. IEEE TNNLS, 26(8):1747-1760, 2015.","title":"Adaptive batch mode active learning","authors":"Shayok Chakraborty, Vineeth Balasubramanian, Sethuraman Panchanathan","year":"2015","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2017] Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, pages 1002-1012, 2017.","title":"Active bias: Training more accurate neural networks by emphasizing high variance samples","authors":"Chang","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das and Kempe, 2008] Abhimanyu Das and David Kempe. Algo- rithms for subset selection in linear regression. In ACM STOC, pages 45-54. ACM, 2008.","title":"Abhimanyu Das and David Kempe. Algorithms for subset selection in linear regression","authors":"Kempe Das","year":"2008","venue":"ACM STOC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Acceler- ating stochastic gradient descent using predictive variance reduc- tion. In NIPS, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Katharopoulos and Fleuret, 2017] Angelos Katharopoulos and Franc ¸ois Fleuret. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043, 2017.","title":"Angelos Katharopoulos and Franc ¸ois Fleuret. Biased importance sampling for deep neural network training","authors":"Fleuret Katharopoulos","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Katharopoulos and Fleuret, 2018] Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling. arXiv:1803.00942, 2018.","title":"Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling","authors":"Fleuret Katharopoulos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2014] Mu Li, Tong Zhang, Yuqiang Chen, and Alexan- der J Smola. Efficient mini-batch training for stochastic opti- mization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 661- 670. ACM, 2014.","title":"Efficient mini-batch training for stochastic optimization","authors":"Mu Li, Tong Zhang, Yuqiang Chen, Alexander J Smola","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2016] Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Fast dpp sampling for nystr\\\" om with application to kernel methods. arXiv preprint arXiv:1603.06052, 2016.","title":"Fast dpp sampling for nystr\\\" om with application to kernel methods","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin and Bilmes, 2011] Hui Lin and Jeff Bilmes. A class of sub- modular functions for document summarization. In ACL, pages 510-520. Association for Computational Linguistics, 2011.","title":"A class of submodular functions for document summarization","authors":"Hui Lin, Jeff Bilmes","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Loshchilov and Hutter, 2015] Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. ICLR Workshops, 2015. [Minoux, 1978] Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization tech- niques, pages 234-243. Springer, 1978.","title":"Online batch selection for faster training of neural networks","authors":"Hutter Loshchilov, Ilya Loshchilov, Frank Hutter","year":"1978","venue":"Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mirzasoleiman et al., 2013] Baharan Mirzasoleiman, Amin Kar- basi, Rik Sarkar, and Andreas Krause. Distributed submodu- lar maximization: Identifying representative elements in massive data. In Advances in Neural Information Processing Systems, pages 2049-2057, 2013.","title":"Distributed submodular maximization: Identifying representative elements in massive data","authors":"Mirzasoleiman","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mirzasoleiman et al., 2015] Baharan Mirzasoleiman, Ashwinku- mar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. Lazier than lazy greedy. In AAAI, 2015.","title":"Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru","authors":"Mirzasoleiman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nemhauser et al., 1978] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 14(1):265-294, 1978.","title":"An analysis of approximations for maximizing submodular set functions-i","authors":"Laurence A George L Nemhauser, Marshall L Wolsey, Fisher","year":"1978","venue":"Mathematical programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop, page 5, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS workshop","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS-W","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singh and Balasubramanian, 2018] Krishna Kant Singh and Vi- neeth N Balasubramanian. Submodular importance sampling for neural network training. Master's thesis, Indian Institute of Tech- nology Hyderabad, 2018.","title":"Krishna Kant Singh and Vineeth N Balasubramanian. Submodular importance sampling for neural network training","authors":"Balasubramanian Singh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thangarasa and Taylor, 2018] Vithursan Thangarasa and Gra- ham W Taylor. Self-paced learning with adaptive deep visual embeddings. arXiv preprint arXiv:1807.09200, 2018.","title":"Vithursan Thangarasa and Graham W Taylor. Self-paced learning with adaptive deep visual embeddings","authors":"Taylor Thangarasa","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei et al., 2014] Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset selection for large- scale speech training data. In ICASSP, pages 3311-3315. IEEE, 2014.","title":"Submodular subset selection for largescale speech training data","authors":"Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, Jeff Bilmes","year":"2014","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wei et al., 2015] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Sub- modularity in data subset selection and active learning. In ICML, pages 1954-1963, 2015.","title":"Submodularity in data subset selection and active learning","authors":"Kai Wei, Rishabh Iyer, Jeff Bilmes","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Cheng Zhang, Hedvig Kjellstrom, and Stephan Mandt. Determinantal point processes for mini-batch diversifica- tion. arXiv preprint arXiv:1705.00607, 2017.","title":"Determinantal point processes for mini-batch diversification","authors":"Cheng Zhang, Hedvig Kjellstrom, Stephan Mandt","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018] Cheng Zhang, Cengiz Öztireli, Stephan Mandt, and Giampiero Salvi. Active mini-batch sampling using repulsive point processes. arXiv:1804.02772, 2018.","title":"Active mini-batch sampling using repulsive point processes","authors":"Cheng Zhang, Cengiz Öztireli, Stephan Mandt, Giampiero Salvi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao and Zhang, 2014] Peilin Zhao and Tong Zhang. Accelerating minibatch stochastic gradient descent using stratified sampling. arXiv preprint arXiv:1405.3080, 2014.","title":"Accelerating minibatch stochastic gradient descent using stratified sampling","authors":"Peilin Zhao, Tong Zhang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao and Zhang, 2015] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss min- imization. In ICML, pages 1-9, 2015.","title":"Stochastic optimization with importance sampling for regularized loss minimization","authors":"Peilin Zhao, Tong Zhang","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou and Bilmes, 2018] Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity. In ICLR, 2018.","title":"Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity","authors":"Bilmes ; ] Tianyi Zhou, Jeff Zhou, Bilmes","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}