{"type":"ScholarlyArticle","title":"Planning in Factored State and Action Spaces with Learned Binarized Neural Network Transition Models","authors":[{"name":"Buser Say","slug":"buser-say","given":"Buser","family":"Say","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"},{"name":"Scott Sanner","slug":"scott-sanner","given":"Scott","family":"Sanner","affiliation":"Department of Mechanical & Industrial Engineering , University of Toronto , Canada"}],"abstract":"In this paper, we leverage the efficiency of Binarized Neural Networks (BNNs) to learn complex state transition models of planning domains with discretized factored state and action spaces. In order to directly exploit this transition structure for planning, we present two novel compilations of the learned factored planning problem with BNNs based on reductions to Boolean Satisfiability (FD-SAT-Plan) as well as Binary Linear Programming (FD-BLP-Plan). Experimentally, we show the effectiveness of learning complex transition models with BNNs, and test the runtime efficiency of both encodings on the learned factored planning problem. After this initial investigation, we present an incremental constraint generation algorithm based on generalized landmark constraints to improve the planning accuracy of our encodings. Finally, we show how to extend the best performing encoding (FD-BLP-Plan+) beyond goals to handle factored planning problems with rewards.","doi":"10.24963/ijcai.2018/669","url":"https://ijcai.org/proceedings/2018/669","pdf":"https://ijcai.org/proceedings/2018/0669.pdf","markdown":"https://ijcai.org/proceedings/2018/669.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/669","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":669,"pages":{"first":4815,"last":4821},"keywords":["Planning and Scheduling: Other approaches to planning","Constraints and SAT: Modeling;Formulation","Planning and Scheduling: Model-Based Reasoning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Abío and Stuckey, 2014] Ignasi Abío and Peter J. Stuckey. Encoding linear constraints into sat. In Principles and Practice of Constraint Programming, pages 75-91. Springer Int Publishing, 2014.","title":"Encoding linear constraints into sat","authors":"Ignasi Abío, Peter J Stuckey","year":"2014","venue":"Principles and Practice of Constraint Programming","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amir and Chang, 2008] Eyal Amir and Allen Chang. Learning partially observable deterministic action models. JAIR, 33:349-402, 2008.","title":"Learning partially observable deterministic action models","authors":"Chang ; Amir, Eyal Amir, Allen Chang","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Audemard and Simon, 2014] Gilles Audemard and Laurent Simon. Lazy Clause Exchange Policy for Parallel SAT Solvers, pages 197-205. Springer Int. Publishing, 2014.","title":"Lazy Clause Exchange Policy for Parallel SAT Solvers","authors":"Simon ; Audemard, Gilles Audemard, Laurent Simon","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Biere et al., 2009] Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh, editors. Handbook of Satisfi- ability, volume 185 of Frontiers in Artificial Intelligence and Applications. IOS Press, 2009.","title":"Handbook of Satisfiability","authors":"Biere","year":"2009","venue":"Frontiers in Artificial Intelligence and Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boutilier et al., 1999] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretic planning: Structural as- sumptions and computational leverage. JAIR, 11(1):1-94, 1999.","title":"Decision-theoretic planning: Structural assumptions and computational leverage","authors":"Craig Boutilier, Thomas Dean, Steve Hanks","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. JMLR, 12:2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Davies et al., 2015] Toby O. Davies, Adrian R. Pearce, Pe- ter J. Stuckey, and Nir Lipovetzky. Sequencing operator counts. In 25th ICAPS, pages 61-69, 2015.","title":"Sequencing operator counts","authors":"Toby O Davies, Adrian R Pearce, Peter J Stuckey, Nir Lipovetzky","year":"2015","venue":"25th ICAPS","doi":null,"url":null,"internal_url":"/proceedings/2016/618"},{"position":8,"raw":"[Davis and Putnam, 1960] Martin Davis and Hilary Putnam. A computing procedure for quantification theory. Journal of the ACM, 7(3):201-215, 1960.","title":"A computing procedure for quantification theory","authors":"Putnam ; Davis, Martin Davis, Hilary Putnam","year":"1960","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2013] Li Deng, Geoffrey E. Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: an overview. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8599-8603, 2013.","title":"New types of deep neural network learning for speech recognition and related applications: an overview","authors":"Li Deng, Geoffrey E Hinton, Brian Kingsbury","year":"2013","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guestrin et al., 2001] Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In 17th IJCAI, pages 673-680, 2001.","title":"Max-norm projections for factored MDPs","authors":"Carlos Guestrin, Daphne Koller, Ronald Parr","year":"2001","venue":"17th IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2001/93"},{"position":11,"raw":"[Helmert, 2006] Malte Helmert. The fast downward plan- ning system. JAIR, 26(1):191-246, 2006.","title":"The fast downward planning system","authors":"Malte Helmert","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In 29th NIPS, pages 4107-4115.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":null,"venue":"29th NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[IBM, 2017] IBM. IBM ILOG CPLEX Optimization Studio CPLEX User's Manual, 2017.","title":"IBM. IBM ILOG CPLEX Optimization Studio CPLEX User's Manual","authors":null,"year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In 32nd ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"32nd ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Keller and Helmert, 2013] Thomas Keller and Malte Helmert. Trial-based heuristic tree search for finite horizon MDPs. In 23rd ICAPS, pages 135-143, 2013.","title":"Trial-based heuristic tree search for finite horizon MDPs","authors":"Helmert Keller, Thomas Keller, Malte Helmert","year":"2013","venue":"rd ICAPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In ECML, pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In 25th NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"25th NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mann and Mannor, 2014] Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Bet- ter policies with fewer iterations. In 21st ICML, volume 1, 2014.","title":"Scaling up approximate value iteration with options: Better policies with fewer iterations","authors":"Mannor Mann, Timothy Mann, Shie Mannor","year":"2014","venue":"21st ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann ma- chines. In 27th ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"27th ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Richter and Westphal, 2010] Silvia Richter and Matthias Westphal. The lama planner: Guiding cost-based anytime planning with landmarks. JAIR, 39(1):127-177, 2010.","title":"The lama planner: Guiding cost-based anytime planning with landmarks","authors":"Silvia Richter, Matthias Westphal","year":"2010","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sanner and Yoon, 2011] Scott Sanner and Sungwook Yoon. International probabilistic planning competition. 2011.","title":"International probabilistic planning competition","authors":"Yoon Sanner, Scott Sanner, Sungwook Yoon","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sanner, 2010] Scott Sanner. Relational dynamic influence diagram language (rddl): Language description. 2010.","title":"Relational dynamic influence diagram language (rddl): Language description","authors":"Scott Sanner","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Say et al., 2017] Buser Say, Ga Wu, Yu Qing Zhou, and Scott Sanner. Nonlinear hybrid planning with deep net learned transition models and mixed-integer linear pro- gramming. In 26th IJCAI, pages 750-756, 2017.","title":"Nonlinear hybrid planning with deep net learned transition models and mixed-integer linear programming","authors":"Buser Say, Ga Wu, Yu Qing Zhou, Scott Sanner","year":"2017","venue":"26th IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/104"},{"position":25,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, pages 484-503, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Christopher J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2017] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. 2017.","title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, Demis Hassabis","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sinz, 2005] Carsten Sinz. Towards an Optimal CNF Encod- ing of Boolean Cardinality Constraints, pages 827-831. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005.","title":"Towards an Optimal CNF Encoding of Boolean Cardinality Constraints","authors":"Sinz","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2007] Qiang Yang, Kangheng Wu, and Yunfei Jiang. Learning action models from plan examples using weighted max-sat. AIJ, 171(2):107-143, 2007.","title":"Learning action models from plan examples using weighted max-sat","authors":"Qiang Yang, Kangheng Wu, Yunfei Jiang","year":"2007","venue":"AIJ","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}