{"type":"ScholarlyArticle","title":"Nonlinear Hybrid Planning with Deep Net Learned Transition Models and Mixed-Integer Linear Programming","authors":[{"name":"Buser Say","slug":"buser-say","given":"Buser","family":"Say","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"},{"name":"Ga Wu","slug":"ga-wu","given":"Ga","family":"Wu","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"},{"name":"Yu Qing Zhou","slug":"yu-qing-zhou","given":"Yu Qing","family":"Zhou","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"},{"name":"Scott Sanner","slug":"scott-sanner","given":"Scott","family":"Sanner","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"}],"abstract":"In many real-world hybrid (mixed discrete continuous) planning problems such as Reservoir Control, Heating, Ventilation and Air Conditioning (HVAC), and Navigation, it is difficult to obtain a model of the complex nonlinear dynamics that govern state evolution.  However, the ubiquity of modern sensors allow us to collect large quantities of data from each of these complex systems and build accurate, nonlinear deep network models of their state transitions.  But there remains one major problem for the task of control -- how can we plan with deep network learned transition models without resorting to Monte Carlo Tree Search and other black-box transition model techniques that ignore model structure and do not easily extend to mixed discrete and continuous domains? In this paper, we make the critical observation that the popular Rectified Linear Unit (ReLU) transfer function for deep networks not only allows accurate nonlinear deep net model learning, but also permits a direct compilation of the deep network transition model to a Mixed-Integer Linear Program (MILP) encoding in a planner we call Hybrid Deep MILP Planning (HD-MILP-PLAN). We identify deep net specific optimizations and a simple sparsification method for HD-MILP-PLAN that improve performance over a naive encoding, and show that we are able to plan optimally with respect to the learned deep network.","doi":"10.24963/ijcai.2017/104","url":"https://ijcai.org/proceedings/2017/104","pdf":"https://ijcai.org/proceedings/2017/0104.pdf","markdown":"https://ijcai.org/proceedings/2017/104.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/104","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Constraints and Satisfiability","track":"Main track","number":104,"pages":{"first":750,"last":756},"keywords":["Machine Learning: Deep Learning","Constraints and Satisfiability: Modeling/Formulation","Planning and Scheduling: Planning and Scheduling","Combinatorial & Heuristic Search: Combinatorial search/optimisation"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Joze- fowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Leven- berg, Dan Mané, Rajat Monga, Sherry Moore, Derek Mur- ray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.","title":"TensorFlow: Large-scale machine learning on heterogeneous systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal et al., 2010] Yuvraj Agarwal, Bharathan Balaji, Rajesh Gupta, Jacob Lyles, Michael Wei, and Thomas Weng. Occupancy-driven energy management for smart building automation. In ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building, pages 1-6, 2010.","title":"Occupancy-driven energy management for smart building automation","authors":"Yuvraj Agarwal, Bharathan Balaji, Rajesh Gupta, Jacob Lyles, Michael Wei, Thomas Weng","year":"2010","venue":"ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bryce et al., 2015] Daniel Bryce, Sicun Gao, David Musliner, and Robert Goldman. SMT-based nonlinear PDDL+ planning. In 29th AAAI, pages 3247-3253, 2015.","title":"SMT-based nonlinear PDDL+ planning","authors":"Daniel Bryce, Sicun Gao, David Musliner, Robert Goldman","year":"2015","venue":"29th AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cashmore et al., 2016] Michael Cashmore, Maria Fox, Derek Long, and Daniele Magazzeni. A compilation of the full PDDL+ language into SMT. In ICAPS, pages 79- 87, 2016.","title":"A compilation of the full PDDL+ language into SMT","authors":"Michael Cashmore, Maria Fox, Derek Long, Daniele Magazzeni","year":"2016","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coles et al., 2013] Amanda J. Coles, Andrew Coles, Maria Fox, and Derek Long. A hybrid LP-RPG heuristic for modelling numeric resource flows in planning. JAIR, pages 343-412, 2013.","title":"A hybrid LP-RPG heuristic for modelling numeric resource flows in planning","authors":"Amanda J Coles, Andrew Coles, Maria Fox, Derek Long","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coulom, 2006] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional Conference on Computers and Games, pages 72-83. Springer Berlin Heidelberg, 2006.","title":"Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search","authors":"Coulom","year":"2006","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[D'Andreagiovanni and Gleixner, 2016] Fabio D'Andreagiovanni and Ambros M. Gleixner. Towards an Accurate Solution of Wireless Network Design Problems, pages 135-147. Springer International Publishing, 2016.","title":"Towards an Accurate Solution of Wireless Network Design Problems","authors":"Fabio D' Andreagiovanni, Ambros M Gleixner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Faulwasser and Findeisen, 2009] Timm Faulwasser and Rolf Findeisen. Nonlinear Model Predictive Path- Following Control. In Nonlinear Model Predictive Control -Towards New Challenging Applications, pages 335-343. Springer, Berlin, Heidelberg, 2009.","title":"Nonlinear Model Predictive Control -Towards New Challenging Applications","authors":"Findeisen Faulwasser, Timm Faulwasser, Rolf Findeisen","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fox and Long, 2006] Maria Fox and Derek Long. Mod- elling mixed discrete-continuous domains for planning. JAIR, pages 235-297, 2006.","title":"Modelling mixed discrete-continuous domains for planning","authors":"Maria Fox, Derek Long","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2016] Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Kilian Q Weinberger","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ivankovic et al., 2014] Franc Ivankovic, Patrik Haslum, Sylvie Thiebaux, Vikas Shivashankar, and Dana Nau. Op- timal planning with global numerical state constraints. In ICAPS, pages 145-153, 2014.","title":"Optimal planning with global numerical state constraints","authors":"Ivankovic","year":"2014","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keller and Helmert, 2013] Thomas Keller and Malte Helmert. Trial-based heuristic tree search for finite horizon MDPs. In ICAPS, pages 135-143, 2013.","title":"Trial-based heuristic tree search for finite horizon MDPs","authors":"Helmert Keller, Thomas Keller, Malte Helmert","year":"2013","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In ECML, pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Löhr et al., 2012] Johannes Löhr, Patrick Eyerich, Thomas Keller, and Bernhard Nebel. A planning based framework for controlling hybrid systems. In ICAPS, pages 164-171, 2012.","title":"A planning based framework for controlling hybrid systems","authors":"Johannes Löhr, Patrick Eyerich, Thomas Keller, Bernhard Nebel","year":"2012","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Work- shop. 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Penna et al., 2009] Giuseppe Della Penna, Daniele Maga- zzeni, Fabio Mercorio, and Benedetto Intrigila. UPMur- phi: A tool for universal planning on PDDL+ problems. In ICAPS, pages 106-113, 2009.","title":"UPMurphi: A tool for universal planning on PDDL+ problems","authors":"Giuseppe Della Penna, Daniele Magazzeni, Fabio Mercorio, Benedetto Intrigila","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Piotrowski et al., 2016] Wiktor Mateusz Piotrowski, Maria Fox, Derek Long, Daniele Magazzeni, and Fabio Merco- rio. Heuristic planning for hybrid systems. In AAAI, pages 4254-4255, 2016.","title":"Heuristic planning for hybrid systems","authors":"Mateusz Wiktor, Maria Piotrowski, Derek Fox, Daniele Long, Fabio Magazzeni, Mercorio","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Scala et al., 2016a] Enrico Scala, Patrik Haslum, Sylvie Thiébaux, and Miquel Ramírez. Interval-based relaxation for general numeric planning. In ECAI, pages 655-663, 2016.","title":"Interval-based relaxation for general numeric planning","authors":"Scala","year":"2016","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Scala et al., 2016b] Enrico Scala, Miquel Ramirez, Patrik Haslum, and Sylvie Thiebaux. Numeric planning with dis- junctive global constraints via SMT. In ICAPS, pages 276- 284, 2016.","title":"Numeric planning with disjunctive global constraints via SMT","authors":"Scala","year":"2016","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, pages 484-503, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Christopher J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, 1st edition, 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szepesvári, 2010] Csaba Szepesvári. Algorithms for Rein- forcement Learning. Morgan & Claypool, 2010.","title":"Algorithms for Reinforcement Learning","authors":"Csaba Szepesvári","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weinstein and Littman, 2012] Ari Weinstein and Michael L. Littman. Bandit-based planning and learning in continuous-action markov decision processes. In ICAPS, 2012.","title":"Bandit-based planning and learning in continuous-action markov decision processes","authors":"Littman Weinstein, Ari Weinstein, Michael L Littman","year":"2012","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yeh, 1985] William G Yeh. Reservoir management and operations models: A state-of-the-art review. Water Re- sources research, 21,12:17971818, 1985.","title":"Reservoir management and operations models: A state-of-the-art review","authors":"G William, Yeh","year":"1985","venue":"Water Resources research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}