{"type":"ScholarlyArticle","title":"A Solver + Gradient Descent Training Algorithm for Deep Neural Networks","authors":[{"name":"Dhananjay Ashok","slug":"dhananjay-ashok","given":"Dhananjay","family":"Ashok","affiliation":"University of Toronto ,"},{"name":"Vineel Nagisetty","slug":"vineel-nagisetty","given":"Vineel","family":"Nagisetty","affiliation":"Borealis AI ,"},{"name":"Christopher Srinivasa","slug":"christopher-srinivasa","given":"Christopher","family":"Srinivasa","affiliation":"Borealis AI ,"},{"name":"Vijay Ganesh","slug":"vijay-ganesh","given":"Vijay","family":"Ganesh","affiliation":"University of Waterloo"}],"abstract":"We present a novel hybrid algorithm for training Deep Neural Networks that combines the state-of-the-art Gradient Descent (GD) method with a Mixed Integer Linear Programming (MILP) solver, outperforming GD and variants in terms of accuracy, as well as resource and data efficiency for both regression and classification tasks. \r\nOur GD+Solver hybrid algorithm, called GDSolver, works as follows: given a DNN D as input, GDSolver invokes GD to partially train D until it gets stuck in a local minima, at which point GDSolver invokes an MILP solver to exhaustively search a region of the loss landscape around the weight assignments of D’s final layer parameters with the goal of tunnelling through and escaping the local minima. The process is repeated until desired accuracy is achieved. \r\nIn our experiments, we find that GDSolver not only scales well to additional data and very large model sizes, but also outperforms all other competing methods in terms of rates of convergence and data efficiency. For regression tasks, GDSolver produced models that, on average, had 31.5% lower MSE in 48% less time, and for classification tasks on MNIST and CIFAR10, GDSolver was able to achieve the highest accuracy over all competing methods, using only 50% of the training data that GD baselines required.","doi":"10.24963/ijcai.2022/246","url":"https://ijcai.org/proceedings/2022/246","pdf":"https://ijcai.org/proceedings/2022/0246.pdf","markdown":"https://ijcai.org/proceedings/2022/246.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/246","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Constraint Satisfaction and Optimization","track":"Main Track","number":246,"pages":{"first":1766,"last":1773},"keywords":["Constraint Satisfaction and Optimization: Constraint Optimization","Constraint Satisfaction and Optimization: Constraint Programming","Constraint Satisfaction and Optimization: Constraint Satisfaction","Constraint Satisfaction and Optimization: Constraints and Machine Learning","Constraint Satisfaction and Optimization: Solvers and Tools"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987064","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985207","length_sec":676,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bunel et al., 2017] Rudy Bunel, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar. A unified view of piecewise linear neural network verification. arXiv preprint arXiv:1711.00455, 2017.","title":"A unified view of piecewise linear neural network verification","authors":"Rudy Bunel, Ilker Turkaslan, Pushmeet Philip Hs Torr, M Pawan Kohli, Kumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2017] Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. Maximum resilience of artificial neu- ral networks. In International Symposium on Automated Technology for Verification and Analysis, pages 251-268. Springer, 2017.","title":"Maximum resilience of artificial neural networks","authors":"Cheng","year":"2017","venue":"International Symposium on Automated Technology for Verification and Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gupte et al., 2013] Akshay Gupte, Shabbir Ahmed, Myun Seok Cheon, and Santanu Dey. Solving mixed integer bilinear problems using milp formulations. SIAM Journal on Optimization, 23(2):721-744, 2013.","title":"Solving mixed integer bilinear problems using milp formulations","authors":"Akshay Gupte, Shabbir Ahmed, Myun Seok Cheon, Santanu Dey","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Howard and Ruder, 2018] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text clas- sification. arXiv preprint arXiv:1801.06146, 2018.","title":"Universal language model fine-tuning for text classification","authors":"Ruder Howard, Jeremy Howard, Sebastian Ruder","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Icarte et al., 2019] Rodrigo Toro Icarte, León Illanes, Mar- garita P Castro, Andre A Cire, Sheila A McIlraith, and J Christopher Beck. Training binarized neural networks using mip and cp. In International Conference on Princi- ples and Practice of Constraint Programming, pages 401- 417. Springer, 2019.","title":"Training binarized neural networks using mip and cp","authors":"Rodrigo Toro Icarte, León Illanes, Margarita P Castro, Andre A Cire, Sheila A Mcilraith, J Christopher, Beck","year":"2019","venue":"International Conference on Principles and Practice of Constraint Programming","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kurbiel and Khaleghian, 2017] Thomas Kurbiel and Shahrzad Khaleghian. Training of deep neural networks based on distance measures using rmsprop. arXiv preprint arXiv:1708.01911, 2017.","title":"Training of deep neural networks based on distance measures using rmsprop","authors":"Thomas Kurbiel, Shahrzad Khaleghian","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li and Arora, 2019] Zhiyuan Li and Sanjeev Arora. An ex- ponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454, 2019.","title":"An exponential learning rate schedule for deep learning","authors":"Li, Arora, Zhiyuan Li, Sanjeev Arora","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Narodytska et al., 2019] Nina Narodytska, Hongce Zhang, Aarti Gupta, and Toby Walsh. In search for a sat-friendly binarized neural network architecture. In International Conference on Learning Representations, 2019.","title":"In search for a sat-friendly binarized neural network architecture","authors":"Nina Narodytska, Hongce Zhang, Aarti Gupta, Toby Walsh","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pan and Yang, 2009] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowl- edge and data engineering, 22(10):1345-1359, 2009.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2009","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversar- ial settings. In 2016 IEEE European symposium on secu- rity and privacy (EuroS&P), pages 372-387. IEEE, 2016.","title":"The limitations of deep learning in adversarial settings","authors":"Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, Ananthram Swami","year":"2016","venue":"IEEE European symposium on security and privacy (EuroS&P)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural in- formation processing systems, 32:8026-8037, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pedroso, 2011] Joo Pedro Pedroso. Optimization with gurobi and python. INESC Porto and Universidade do Porto,, Porto, Portugal, 1, 2011.","title":"Optimization with gurobi and python","authors":"Pedro Joo, Pedroso","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder, Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Thorbjarnarson and Yorke-Smith, 2020] Tómas Thor- bjarnarson and Neil Yorke-Smith. On training neural networks with mixed integer programming. arXiv preprint arXiv:2009.03825, 2020.","title":"Tómas Thorbjarnarson and Neil Yorke-Smith","authors":"Yorke-Smith Thorbjarnarson","year":"2020","venue":"On training neural networks with mixed integer programming","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tjeng et al., 2017] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356, 2017.","title":"Evaluating robustness of neural networks with mixed integer programming","authors":"Vincent Tjeng, Kai Xiao, Russ Tedrake","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Verma et al., 2019] Sunny Verma, Chen Wang, Liming Zhu, and Wei Liu. A compliance checking framework for dnn models. In Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19}. International Joint Conferences on Artificial Intelligence Organization, 2019.","title":"A compliance checking framework for dnn models","authors":"Sunny Verma, Chen Wang, Liming Zhu, Wei Liu","year":"2019","venue":"Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/924"},{"position":18,"raw":"[Zhang et al., 2018] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neu- ral network robustness certification with general activation functions. arXiv preprint arXiv:1811.00866, 2018.","title":"Efficient neural network robustness certification with general activation functions","authors":"Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, Luca Daniel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}