{"type":"ScholarlyArticle","title":"Value Refinement Network (VRN)","authors":[{"name":"Jan Wöhlke","slug":"jan-wohlke","given":"Jan","family":"Wöhlke","affiliation":"Bosch Center for Artificial Intelligence; UvA-Bosch Delta Lab , University of Amsterdam"},{"name":"Felix Schmitt","slug":"felix-schmitt","given":"Felix","family":"Schmitt","affiliation":"Bosch Center for Artificial Intelligence"},{"name":"Herke van Hoof","slug":"herke-van-hoof","given":"Herke","family":"van Hoof","affiliation":"AMLab , University of Amsterdam"}],"abstract":"In robotic tasks, we encounter the unique strengths of (1) reinforcement learning (RL) that can handle high-dimensional observations as well as unknown, complex dynamics and (2) planning that can handle sparse and delayed rewards given a dynamics model. Combining these strengths of RL and planning, we propose the Value Refinement Network (VRN), in this work. Our VRN is an RL-trained neural network architecture that learns to locally refine an initial (value-based) plan in a simplified (2D) problem abstraction based on detailed local sensory observations. We evaluate the VRN on simulated robotic (navigation) tasks and demonstrate that it can successfully refine sub-optimal plans to match the performance of more costly planning in the non-simplified problem. Furthermore, in a dynamic environment, the VRN still enables high task completion without global re-planning.","doi":"10.24963/ijcai.2022/494","url":"https://ijcai.org/proceedings/2022/494","pdf":"https://ijcai.org/proceedings/2022/0494.pdf","markdown":"https://ijcai.org/proceedings/2022/494.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/494","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":494,"pages":{"first":3558,"last":3565},"keywords":["Machine Learning: Deep Reinforcement Learning","Planning and Scheduling: Learning in Planning and Scheduling","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986764","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984907","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Pe- ter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in neural information processing sys- tems, pages 5048-5058, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Openai, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellman, 1957] Richard Bellman. Dynamic Programming. Princeton University Press, 1957.","title":"Dynamic Programming","authors":"Richard Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Binghong Chen, Bo Dai, Qinjie Lin, Guo Ye, Han Liu, and Le Song. Learning to plan in high dimen- sions via neural exploration-exploitation trees. In ICLR, 2019.","title":"Learning to plan in high dimensions via neural exploration-exploitation trees","authors":"Binghong Chen, Bo Dai, Qinjie Lin, Guo Ye, Han Liu, Le Song","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Christen et al., 2021] Sammy Christen, Lukas Jendele, Emre Aksan, and Otmar Hilliges. Learning functionally decomposed hierarchies for continuous control tasks with path planning. IEEE Robotics and Automation Letters, 6(2):3623-3630, 2021.","title":"Learning functionally decomposed hierarchies for continuous control tasks with path planning","authors":"Sammy Christen, Lukas Jendele, Emre Aksan, Otmar Hilliges","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Florensa et al., 2017] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Con- ference on Robot Learning, pages 482-495, 2017.","title":"Reverse curriculum generation for reinforcement learning","authors":"Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, Pieter Abbeel","year":"2017","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Francis et al., 2020] Anthony Francis, Aleksandra Faust, Hao-Tien Chiang, Jasmine Hsu, J. Chase Kew, Marek Fiser, and Tsang-Wei Edward Lee. Long-range indoor navigation with PRM-RL. IEEE Transactions on Robotics, 2020.","title":"Long-range indoor navigation with PRM-RL","authors":"Anthony Francis, Aleksandra Faust, Hao-Tien, Jasmine Chiang, J Chase Hsu, Marek Kew, Tsang-Wei Fiser, Lee Edward","year":"2020","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haber et al., 2018] Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsically-motivated, self-aware agents. In Advances in Neural Information Processing Systems, pages 8388-8399, 2018.","title":"Learning to play with intrinsically-motivated, self-aware agents","authors":"Nick Haber, Damian Mrowca, Stephanie Wang, Daniel L Li F Fei-Fei, Yamins","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ichter et al., 2018] Brian Ichter, James Harrison, and Marco Pavone. Learning sampling distributions for robot motion planning. In ICRA, pages 7087-7094. IEEE, 2018.","title":"Learning sampling distributions for robot motion planning","authors":"Brian Ichter, James Harrison, Marco Pavone","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2018] Lisa Lee, Emilio Parisotto, Deven- dra Singh Chaplot, Eric Xing, and Ruslan Salakhutdinov. Gated path planning networks. In ICML, pages 2947- 2955, 2018.","title":"Gated path planning networks","authors":"Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, Ruslan Salakhutdinov","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leurent, 2018] Edouard Leurent. An environment for au- tonomous driving decision-making. https://github.com/ eleurent/highway-env, 2018. Accessed: 2022-05-21.","title":"Edouard Leurent. An environment for autonomous driving decision-making","authors":"Leurent","year":"2018","venue":null,"doi":null,"url":"https://github.com/eleurent/highway-env,2018","internal_url":null},{"position":13,"raw":"[Levy et al., 2019] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In ICLR, 2019.","title":"Learning multi-level hierarchies with hindsight","authors":"Andrew Levy, George Konidaris, Robert Platt, Kate Saenko","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nachum et al., 2018] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierar- chical reinforcement learning. In Advances in Neural In- formation Processing Systems, pages 3303-3313, 2018.","title":"Data-efficient hierarchical reinforcement learning","authors":"Ofir Nachum, Shane Shixiang, Honglak Gu, Sergey Lee, Levine","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nardelli et al., 2019] Nantas Nardelli, Gabriel Synnaeve, Zeming Lin, Pushmeet Kohli, Philip HS Torr, and Nico- las Usunier. Value propagation networks. In ICLR, 2019.","title":"Value propagation networks","authors":"Nardelli","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2016] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. In ICLR, 2016.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tamar et al., 2016] Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration net- works. In Advances in Neural Information Processing Sys- tems, pages 2154-2162, 2016.","title":"Value iteration networks","authors":"Tamar","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. FeUdal net- works for hierarchical reinforcement learning. In ICML, pages 3540-3549, 2017.","title":"FeUdal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wöhlke et al., 2020] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-based start state curriculum framework for reinforcement learning. In Autonomous Agents and MultiAgent Systems, pages 1503-1511, 2020.","title":"A performance-based start state curriculum framework for reinforcement learning","authors":"Wöhlke","year":"2020","venue":"Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wöhlke et al., 2021] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. Hierarchies of planning and reinforcement learning for robot navigation. In ICRA, 2021.","title":"Hierarchies of planning and reinforcement learning for robot navigation","authors":"Wöhlke","year":"2021","venue":"ICRA","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}