{"type":"ScholarlyArticle","title":"Exploring the Task Cooperation in Multi-goal Visual Navigation","authors":[{"name":"Yuechen Wu","slug":"yuechen-wu","given":"Yuechen","family":"Wu","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Zhenhuan Rao","slug":"zhenhuan-rao","given":"Zhenhuan","family":"Rao","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Shijian Lu","slug":"shijian-lu","given":"Shijian","family":"Lu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University"},{"name":"Weizhi Lu","slug":"weizhi-lu","given":"Weizhi","family":"Lu","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"School of Information Science and Technology , University of Science and Technology of China"}],"abstract":"Learning to adapt to a series of different goals in visual navigation is challenging. In this work, we present a model-embedded actor-critic architecture for the multi-goal visual navigation task. To enhance the task cooperation in multi-goal learning, we introduce two new designs to the reinforcement learning scheme: inverse dynamics model (InvDM) and multi-goal co-learning (MgCl). Specifically, InvDM is proposed to capture the navigation-relevant association between state and goal, and provide additional training signals to relieve the sparse reward issue. MgCl aims at improving the sample efficiency and supports the agent to learn from unintentional positive experiences. Extensive results on the interactive platform AI2-THOR demonstrate that the proposed method converges faster than state-of-the-art methods while producing more direct routes to navigate to the goal. The video demonstration is available at: https://youtube.com/channel/UCtpTMOsctt3yPzXqe_JMD3w/videos.","doi":"10.24963/ijcai.2019/86","url":"https://ijcai.org/proceedings/2019/86","pdf":"https://ijcai.org/proceedings/2019/0086.pdf","markdown":"https://ijcai.org/proceedings/2019/86.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/86","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":86,"pages":{"first":609,"last":615},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, volume 2, 2018.","title":"Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton Van Den, Hengel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Pe- ter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In NIPS, pages 5048-5058, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Openai, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banino et al., 2018] Andrea Banino, Caswell Barry, Be- nigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation us- ing grid-like representations in artificial agents. Nature, 557(7705):429, 2018.","title":"Vector-based navigation using grid-like representations in artificial agents","authors":"Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonin-Font et al., 2008] Francisco Bonin-Font, Alberto Or- tiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey. Journal of intelligent and robotic sys- tems, 53(3):263-296, 2008.","title":"Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey","authors":"Bonin-Font","year":"2008","venue":"Journal of intelligent and robotic systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy and Koltun, 2016] Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. arXiv preprint arXiv:1611.01779, 2016.","title":"Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future","authors":"Koltun Dosovitskiy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gupta et al., 2017] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, pages 2616-2625, 2017.","title":"Cognitive mapping and planning for visual navigation","authors":"Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, Jitendra Malik","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. 2017.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kolve et al., 2017] Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2- thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.","title":"Ai2thor: An interactive 3d environment for visual ai","authors":"Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, Ali Farhadi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mirowski et al., 2016] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Ban- ino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex envi- ronments. arXiv preprint arXiv:1611.03673, 2016.","title":"Learning to navigate in complex environments","authors":"Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mirowski et al., 2018] Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Ko- ray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning to navigate in cities without a map. arXiv preprint arXiv:1804.00168, 2018.","title":"Learning to navigate in cities without a map","authors":"Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, Raia Hadsell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Mnih","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, volume 2017, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pathak et al., 2018] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In ICLR, 2018.","title":"Zero-shot visual imitation","authors":"Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, Trevor Darrell","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Savinov et al., 2018] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. 2018.","title":"Semi-parametric topological memory for navigation","authors":"Nikolay Savinov, Alexey Dosovitskiy, Vladlen Koltun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schaul et al., 2015a] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In ICML, pages 1312-1320, 2015.","title":"Universal value function approximators","authors":"Schaul","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schaul et al., 2015b] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Schaul","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016.","title":"Learning to reinforcement learn","authors":"Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2018a] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209, 2018.","title":"Building generalizable agents with a realistic and rich 3d environment","authors":"Wu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018b] Yuechen Wu, Wei Zhang, and Ke Song. Master-slave curriculum design for reinforcement learn- ing. In IJCAI, pages 1523-1529, 2018.","title":"Master-slave curriculum design for reinforcement learning","authors":"Wu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/211"},{"position":23,"raw":"[Xu et al., 2018] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. In CVPR, June 2018.","title":"Pad-net: Multi-tasks guided predictionand-distillation network for simultaneous depth estimation and scene parsing","authors":"Dan Xu, Wanli Ouyang, Xiaogang Wang, Nicu Sebe","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Wei Zhang, Ke Song, Xuewen Rong, and Yibin Li. Coarse-to-fine uav target tracking with deep reinforcement learning. TASE, 2018.","title":"Coarse-to-fine uav target tracking with deep reinforcement learning","authors":"Wei Zhang, Ke Song, Xuewen Rong, Yibin Li","year":"2018","venue":"TASE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2017] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, pages 3357-3364. IEEE, 2017.","title":"Target-driven visual navigation in indoor scenes using deep reinforcement learning","authors":"Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}