{"type":"ScholarlyArticle","title":"Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets","authors":[{"name":"Cong Fei","slug":"cong-fei","given":"Cong","family":"Fei","affiliation":"Huawei Noah's Ark Lab; Tsinghua University"},{"name":"Bin Wang","slug":"bin-wang","given":"Bin","family":"Wang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Yuzheng Zhuang","slug":"yuzheng-zhuang","given":"Yuzheng","family":"Zhuang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"Nanjing University"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"Huawei Noah's Ark Lab"},{"name":"Hongbo Zhang","slug":"hongbo-zhang","given":"Hongbo","family":"Zhang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Xuewu Ji","slug":"xuewu-ji","given":"Xuewu","family":"Ji","affiliation":"Tsinghua University"},{"name":"Wulong Liu","slug":"wulong-liu","given":"Wulong","family":"Liu","affiliation":"Huawei Noah's Ark Lab"}],"abstract":"Generative adversarial imitation learning (GAIL) has shown promising results by taking advantage of generative adversarial nets, especially in the field of robot learning. However, the requirement of isolated single modal demonstrations limits the scalability of the approach to real world scenarios such as autonomous vehicles' demand for a proper understanding of human drivers' behavior. In this paper, we propose a novel multi-modal GAIL framework, named Triple-GAIL, that is able to learn skill selection and imitation jointly from both expert demonstrations and continuously generated experiences with data augmentation purpose by introducing an auxiliary selector. We provide theoretical guarantees on the convergence to optima for both of the generator and the selector respectively. Experiments on real driver trajectories and real-time strategy game datasets demonstrate that Triple-GAIL can better fit multi-modal behaviors close to the demonstrators and outperforms state-of-the-art methods.","doi":"10.24963/ijcai.2020/405","url":"https://ijcai.org/proceedings/2020/405","pdf":"https://ijcai.org/proceedings/2020/0405.pdf","markdown":"https://ijcai.org/proceedings/2020/405.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/405","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":405,"pages":{"first":2929,"last":2935},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning","Machine Learning: Adversarial Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24186/file_24186.m3u8","length_sec":276,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the twenty-first international con- ference on Machine learning, pages 1-8, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chongxuan et al., 2017] LI Chongxuan, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in neural information processing systems, pages 4088-4098, 2017.","title":"Triple generative adversarial nets","authors":"Taufik Li Chongxuan, Jun Xu, Bo Zhu, Zhang","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fang et al., 2019] Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, pages 1-8, 2019.","title":"Survey of imitation learning for robotic manipulation","authors":"Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, Fuchun Sun","year":"2019","venue":"International Journal of Intelligent Robotics and Applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Halkias and Colyar, 2006] John Halkias and James Colyar. Next generation simulation fact sheet. US Department of Transportation: Federal Highway Administration, 2006.","title":"Next generation simulation fact sheet","authors":"Colyar Halkias, John Halkias, James Colyar","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Henaff et al., 2019] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with un- certainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.02705, 2019.","title":"Model-predictive policy learning with uncertainty regularization for driving in dense traffic","authors":"Mikael Henaff, Alfredo Canziani, Yann Lecun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kuefler and Kochenderfer, 2018] Alex Kuefler and Mykel J Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 1071-1078, 2018.","title":"Burn-in demonstrations for multi-modal imitation learning","authors":"Alex Kuefler, J Mykel, Kochenderfer","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Yunzhu Li, Jiaming Song, and Stefano Er- mon. Infogail: Interpretable imitation learning from vi- sual demonstrations. In Advances in Neural Information Processing Systems, pages 3812-3822, 2017.","title":"Ermon. Infogail: Interpretable imitation learning from visual demonstrations","authors":"Li","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin and Zhang, 2018] Jiahao Lin and Zongzhang Zhang. Acgail: Imitation learning about multiple intentions with auxiliary classifier gans. In Pacific Rim International Conference on Artificial Intelligence, pages 321-334. Springer, 2018.","title":"Acgail: Imitation learning about multiple intentions with auxiliary classifier gans","authors":"Lin, Zhang, Jiahao Lin, Zongzhang Zhang","year":"2018","venue":"Pacific Rim International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Merel et al., 2017] Josh Merel, Yuval Tassa, Sriram Srini- vasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201, 2017.","title":"Learning human behaviors from motion capture by adversarial imitation","authors":"Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, Nicolas Heess","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pflueger et al., 2019] Max Pflueger, Ali Agha, and Gaurav S Sukhatme. Rover-irl: Inverse reinforcement learning with soft value iteration networks for planetary rover path plan- ning. IEEE Robotics and Automation Letters, 4(2):1387- 1394, 2019.","title":"Rover-irl: Inverse reinforcement learning with soft value iteration networks for planetary rover path planning","authors":"Pflueger","year":"2019","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Reddy et al., 2019] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: imitation learning via regularized be- havioral cloning. arXiv preprint arXiv:1905.11108, 2019.","title":"Sqil: imitation learning via regularized behavioral cloning","authors":"Siddharth Reddy, Anca D Dragan, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tian et al., 2017] Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C Lawrence Zitnick. Elf: An ex- tensive, lightweight and flexible research platform for real- time strategy games. In Advances in Neural Information Processing Systems, pages 2659-2669, 2017.","title":"Elf: An extensive, lightweight and flexible research platform for realtime strategy games","authors":"Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, C Lawrence, Zitnick","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2017] Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Ro- bust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pages 5320-5329, 2017.","title":"Robust imitation of diverse behaviors","authors":"Ziyu Wang, Josh S Merel, Scott E Reed, Nando De Freitas, Gregory Wayne, Nicolas Heess","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wulfmeier et al., 2015] Markus Wulfmeier, Peter On- druska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015.","title":"Maximum entropy deep inverse reinforcement learning","authors":"Markus Wulfmeier, Peter Ondruska, Ingmar Posner","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}