{"type":"ScholarlyArticle","title":"Value Iteration Networks","authors":[{"name":"Aviv Tamar","slug":"aviv-tamar","given":"Aviv","family":"Tamar","affiliation":"UC Berkeley , 2 OpenAI"},{"name":"Yi Wu","slug":"yi-wu","given":"Yi","family":"Wu","affiliation":"UC Berkeley , 2 OpenAI"},{"name":"Garrett Thomas","slug":"garrett-thomas","given":"Garrett","family":"Thomas","affiliation":"UC Berkeley , 2 OpenAI"},{"name":"Sergey Levine","slug":"sergey-levine","given":"Sergey","family":"Levine","affiliation":"UC Berkeley , 2 OpenAI"},{"name":"Pieter Abbeel","slug":"pieter-abbeel","given":"Pieter","family":"Abbeel","affiliation":"UC Berkeley , 2 OpenAI"}],"abstract":"We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiable approximation of the value-iteration algorithm, which can be represented as a convolutional neural network, and trained end-to-end using standard backpropagation.We evaluate VIN based policies on discrete and continuous path-planning domains, and on a natural-language based search task. We show that by learning an explicit planning computation, VIN policies generalize better to new, unseen domains.This paper is a significantly abridged and IJCAI audience targeted version of the original NIPS 2016 paper with the same title, available here: https://arxiv.org/abs/1602.02867","doi":"10.24963/ijcai.2017/700","url":"https://ijcai.org/proceedings/2017/700","pdf":"https://ijcai.org/proceedings/2017/0700.pdf","markdown":"https://ijcai.org/proceedings/2017/700.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/700","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":700,"pages":{"first":4949,"last":4953},"keywords":["Artificial Intelligence: automated planning","Artificial Intelligence: artificial intelligence"],"videos":[],"references":[{"position":1,"raw":"[Bellman, 1957] R. Bellman. Dynamic Programming. Prince- ton University Press, 1957.","title":"Dynamic Programming","authors":"R Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 2012] D. Bertsekas. Dynamic Programming and Optimal Control, Vol II. Athena Scientific, 4th edition, 2012.","title":"Dynamic Programming and Optimal Control","authors":"D Bertsekas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ciresan et al., 2012] D. Ciresan, U. Meier, and J. Schmidhu- ber. Multi-column deep neural networks for image classifi- cation. In Computer Vision and Pattern Recognition, pages 3642-3649, 2012.","title":"Multi-column deep neural networks for image classification","authors":"D Ciresan, U Meier, J Schmidhuber","year":"2012","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deisenroth and Rasmussen, 2011] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient ap- proach to policy search. In ICML, 2011.","title":"Pilco: A model-based and data-efficient approach to policy search","authors":"M Deisenroth, C E Rasmussen","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2016] Y. Duan, X. Chen, R. Houthooft, J. Schul- man, and P. Abbeel. Benchmarking deep reinforce- ment learning for continuous control. arXiv preprint arXiv:1604.06778, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Y Duan, X Chen, R Houthooft, J Schulman, P Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Farabet et al., 2013] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene label- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1915-1929, 2013.","title":"Learning hierarchical features for scene labeling","authors":"C Farabet, C Couprie, L Najman, Y Lecun","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2016] C. Finn, M. Zhang, J. Fu, X. Tan, Z. Mc- Carthy, E. Scharff, and S. Levine. Guided policy search code implementation, 2016. Software available from rll.berkeley.edu/gps.","title":"Guided policy search code implementation","authors":"C Finn, M Zhang, J Fu, X Tan, Z Mc-Carthy, E Scharff, S Levine","year":"2016","venue":"Software available from rll","doi":null,"url":"berkeley.edu/gps","internal_url":null},{"position":8,"raw":"[Fukushima, 1979] K. Fukushima. Neural network model for a mechanism of pattern recognition unaffected by shift in position-neocognitron. Transactions of the IECE, J62- A(10):658665, 1979.","title":"Neural network model for a mechanism of pattern recognition unaffected by shift in position-neocognitron","authors":"K Fukushima","year":"1979","venue":"Transactions of the IECE, J","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Giusti and others, 2016] A. Giusti et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 2016.","title":"A machine learning approach to visual perception of forest trails for mobile robots","authors":"A Giusti","year":"2016","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2014] X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In NIPS, 2014.","title":"Deep learning for real-time atari game play using offline monte-carlo tree search planning","authors":"Guo","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2016] X. Guo, S. Singh, R. Lewis, and H. Lee. Deep learning for reward design to improve monte carlo tree search in atari games. arXiv:1604.07095, 2016.","title":"Deep learning for reward design to improve monte carlo tree search in atari games","authors":"Guo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/218"},{"position":12,"raw":"[Ilin et al., 2007] R. Ilin, R. Kozma, and P. J. Werbos. Ef- ficient learning in cellular simultaneous recurrent neural networks-the case of maze navigation problem. In ADPRL, 2007.","title":"Efficient learning in cellular simultaneous recurrent neural networks-the case of maze navigation problem","authors":"R Ilin, R Kozma, P J Werbos","year":"2007","venue":"ADPRL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joseph et al., 2013] J. Joseph, A. Geramifard, J. W. Roberts, J. P. How, and N. Roy. Reinforcement learning with mis- specified model classes. In ICRA, 2013.","title":"Reinforcement learning with misspecified model classes","authors":"J Joseph, A Geramifard, J W Roberts, J P How, N Roy","year":"2013","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaelbling and Lozano-Pérez, 2011] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In IEEE International Conference on Robotics and Automation (ICRA), pages 1470-1477, 2011.","title":"Hierarchical task and motion planning in the now","authors":"L P Kaelbling, T Lozano-Pérez","year":"2011","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levine and Abbeel, 2014] S. Levine and P. Abbeel. Learn- ing neural network policies with guided policy search under unknown dynamics. In NIPS, 2014.","title":"Learning neural network policies with guided policy search under unknown dynamics","authors":"S Levine, P Abbeel","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levine et al., 2016] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 17, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"S Levine, C Finn, T Darrell, P Abbeel","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2015] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"J Long, E Shelhamer, T Darrell","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Ried- miller, A. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Rusu, J Veness, M Bellemare, A Graves, M Riedmiller, A Fidjeland, G Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mnih et al., 2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep re- inforcement learning. arXiv preprint arXiv:1602.01783, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A P Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Neu and Szepesvári, 2007] G. Neu and C. Szepesvári. Ap- prenticeship learning using inverse reinforcement learning and gradient methods. In UAI, 2007.","title":"Apprenticeship learning using inverse reinforcement learning and gradient methods","authors":"G Neu, C Szepesvári","year":"2007","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nogueira and Cho, 2016] R. Nogueira and K. Cho. Webnav: A new large-scale task for natural language based sequential decision making. arXiv preprint arXiv:1602.02261, 2016.","title":"Webnav: A new large-scale task for natural language based sequential decision making","authors":"R Nogueira, K Cho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross et al., 2011] S. Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"S Ross, G Gordon, A Bagnell","year":"2011","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schmidhuber, 1990] J. Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reac- tive environments. In International Joint Conference on Neural Networks. IEEE, 1990.","title":"An on-line algorithm for dynamic reinforcement learning and planning in reactive environments","authors":"J Schmidhuber","year":"1990","venue":"International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"J Schulman, S Levine, P Abbeel, M Jordan, P Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Rein- forcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tamar et al., 2016] A. Tamar, Y. WU, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. In NIPS, 2016.","title":"Value iteration networks","authors":"A Tamar, Y Wu, G Thomas, S Levine, P Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Theano Development Team, 2016] Theano Development Team. Theano: A Python framework for fast com- putation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016.","title":"Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions","authors":null,"year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Todorov et al., 2012] E. Todorov, T. Erez, and Y. Tassa. Mu- joco: A physics engine for model-based control. In Intelli- gent Robots and Systems (IROS), 2012 IEEE/RSJ Interna- tional Conference on, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"E Todorov, T Erez, Y Tassa","year":"2012","venue":"Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Watter et al., 2015] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS, 2015.","title":"Embed to control: A locally linear latent dynamics model for control from raw images","authors":"M Watter, J Springenberg, J Boedecker, M Riedmiller","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2015] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual atten- tion. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"K Xu, J Ba, R Kiros, K Cho, A Courville, R Salakhudinov, R Zemel, Y Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}