{"type":"ScholarlyArticle","title":"A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigation","authors":[{"name":"Wenqi Zhang","slug":"wenqi-zhang","given":"Wenqi","family":"Zhang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Kai Zhao","slug":"kai-zhao","given":"Kai","family":"Zhao","affiliation":"Netease Fuxi Robot Department"},{"name":"Peng Li","slug":"peng-li","given":"Peng","family":"Li","affiliation":"Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences Nanjing"},{"name":"Xiao Zhu","slug":"xiao-zhu","given":"Xiao","family":"Zhu","affiliation":"College of Mechanical Engineering , Zhejiang University of Technology"},{"name":"Yongliang Shen","slug":"yongliang-shen","given":"Yongliang","family":"Shen","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Yanna Ma","slug":"yanna-ma","given":"Yanna","family":"Ma","affiliation":"University of Shanghai for Science and Technology"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"Netease Fuxi Robot Department"},{"name":"Weiming Lu","slug":"weiming-lu","given":"Weiming","family":"Lu","affiliation":"College of Computer Science and Technology , Zhejiang University"}],"abstract":"Reliable navigation systems have a wide range of applications in robotics and autonomous driving. Current approaches employ an open-loop process that converts sensor inputs directly into actions. However, these open-loop schemes are challenging to handle complex and dynamic real-world scenarios due to their poor generalization. Imitating human navigation, we add a reasoning process to convert actions back to internal latent states, forming a two-stage closed loop of perception, decision-making, and reasoning. Firstly, VAE-Enhanced Demonstration Learning endows the model with the understanding of basic navigation rules. Then, two dual processes in RL-Enhanced Interaction Learning generate reward feedback for each other and collectively enhance obstacle avoidance capability. The reasoning model can substantially promote generalization and robustness, and facilitate the deployment of the algorithm to real-world robots without elaborate transfers. Experiments show our method is more adaptable to novel scenarios compared with state-of-the-art approaches.","doi":"10.24963/ijcai.2022/654","url":"https://ijcai.org/proceedings/2022/654","pdf":"https://ijcai.org/proceedings/2022/0654.pdf","markdown":"https://ijcai.org/proceedings/2022/654.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/654","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Robotics","track":"Main Track","number":654,"pages":{"first":4717,"last":4724},"keywords":["Robotics: Applications","Machine Learning: Deep Reinforcement Learning","Robotics: Motion and Path Planning","Robotics: Learning in Robotics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986756","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984899","length_sec":360,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Artetxe et al., 2017] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural ma- chine translation. arXiv preprint arXiv:1710.11041, 2017.","title":"Unsupervised neural machine translation","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2020] Ruisheng Cao, Su Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu Chen, and Kai Yu. Unsupervised dual paraphrasing for two-stage semantic parsing. In Proc. of ACL, pages 6806-6817, July 2020.","title":"Unsupervised dual paraphrasing for two-stage semantic parsing","authors":"Ruisheng Cao, Su Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu Chen, Kai Yu","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 1343-1350. IEEE, 2017.","title":"Socially aware motion planning with deep reinforcement learning","authors":"Michael Yu Fan Chen, Miao Everett, Jonathan P Liu, How","year":"2017","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Faust et al., 2018] Aleksandra Faust, Kenneth Oslund, Os- car Ramirez, Anthony Francis, Lydia Tapia, Marek Fiser, and James Davidson. Prm-rl: Long-range robotic nav- igation tasks by combining reinforcement learning and sampling-based planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5113-5120. IEEE, 2018.","title":"Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning","authors":"Aleksandra Faust, Kenneth Oslund, Oscar Ramirez, Anthony Francis, Lydia Tapia, Marek Fiser, James Davidson","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fox et al., 1997] Dieter Fox, Wolfram Burgard, and Sebas- tian Thrun. The dynamic window approach to colli- sion avoidance. IEEE Robotics & Automation Magazine, 4(1):23-33, 1997.","title":"The dynamic window approach to collision avoidance","authors":"Dieter Fox, Wolfram Burgard, Sebastian Thrun","year":"1997","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2021] Huiqiao Fu, Kaiqiang Tang, Peng Li, Wenqi Zhang, Xinpeng Wang, Guizhou Deng, Tao Wang, and Chunlin Chen. Deep reinforcement learning for multi- contact motion planning of hexapod robots. In Proceed- ings of the Thirtieth International Joint Conference on Ar- tificial Intelligence (IJCAI), Montréal, QC, Canada, pages 2381-2388, 2021.","title":"Deep reinforcement learning for multicontact motion planning of hexapod robots","authors":"Huiqiao Fu, Kaiqiang Tang, Peng Li, Wenqi Zhang, Xinpeng Wang, Guizhou Deng, Tao Wang, Chunlin Chen","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ha and Schmidhuber, 2018] David Ha and Jürgen Schmid- huber. Recurrent world models facilitate policy evolution. In Proc. of NeurlPS, 2018.","title":"Recurrent world models facilitate policy evolution","authors":"Schmidhuber Ha, David Ha, Jürgen Schmidhuber","year":"2018","venue":"Proc. of NeurlPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016a] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learn- ing for machine translation. In Proc. of NeurlPS, 2016.","title":"Dual learning for machine translation","authors":"He","year":"2016","venue":"Proc. of NeurlPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proc. of NeurlPS, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Proc. of NeurlPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Khatib, 1986] Oussama Khatib. Real-time obstacle avoid- ance for manipulators and mobile robots. In Autonomous robot vehicles, pages 396-404. Springer, 1986.","title":"Real-time obstacle avoidance for manipulators and mobile robots","authors":"Oussama Khatib","year":"1986","venue":"Autonomous robot vehicles","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Interna- tional Conference on Learning Representations (ICLR), Banff, Canada, April 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koenig and Likhachev, 2002] Sven Koenig and Maxim Likhachev. D* lite. Aaai/iaai, 15:476-483, 2002.","title":"D* lite","authors":"Likhachev Koenig, Sven Koenig, Maxim Likhachev","year":"2002","venue":"Aaai/iaai","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2018] Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep re- inforcement learning. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pages 6252- 6259. IEEE, 2018.","title":"Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning","authors":"Long","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pfeiffer et al., 2018] Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Sieg- wart, and Juan Nieto. Reinforced imitation: Sample effi- cient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, 3(4):4423-4430, 2018.","title":"Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations","authors":"Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, Juan Nieto","year":"2018","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qi et al., 2021] Jie Qi, Hui Yang, and Haixin Sun. Mod- rrt*: A sampling-based algorithm for robot path planning in dynamic environment. IEEE Transactions on Industrial Electronics, 68(8):7244-7251, 2021.","title":"Modrrt*: A sampling-based algorithm for robot path planning in dynamic environment","authors":"Jie Qi, Hui Yang, Haixin Sun","year":"2021","venue":"IEEE Transactions on Industrial Electronics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stentz and others, 1995] Anthony Stentz et al. The focussed d* algorithm for real-time replanning. In IJCAI, vol- ume 95, pages 1652-1659, 1995.","title":"The focussed d* algorithm for real-time replanning","authors":"Anthony Stentz","year":"1995","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1995/213"},{"position":20,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsounis et al., 2020] Vassilios Tsounis, Mitja Alge, Joonho Lee, Farbod Farshidian, and Marco Hutter. Deepgait: Planning and control of quadrupedal gaits using deep re- inforcement learning. IEEE Robotics and Automation Let- ters, 5(2):3699-3706, 2020.","title":"Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning","authors":"Vassilios Tsounis, Mitja Alge, Joonho Lee, Farbod Farshidian, Marco Hutter","year":"2020","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2020] Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion. Science Robotics, 2020.","title":"Multi-expert learning of adaptive legged locomotion","authors":"Yang","year":"2020","venue":"Science Robotics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Wenqi Zhang, Kai Zhao, Peng Li, et al. Learning to navigate in a vuca environment: Hierarchical multi-expert approach. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9254-9261, 2021.","title":"Learning to navigate in a vuca environment: Hierarchical multi-expert approach","authors":"Wenqi Zhang, Kai Zhao, Peng Li","year":"2021","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}