{"type":"ScholarlyArticle","title":"Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving","authors":[{"name":"Akifumi Wachi","slug":"akifumi-wachi","given":"Akifumi","family":"Wachi","affiliation":"IBM Research AI"}],"abstract":"We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multi-agent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator.","doi":"10.24963/ijcai.2019/832","url":"https://ijcai.org/proceedings/2019/832","pdf":"https://ijcai.org/proceedings/2019/0832.pdf","markdown":"https://ijcai.org/proceedings/2019/832.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/832","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":832,"pages":{"first":6006,"last":6012},"keywords":["Special Track on AI for Improving Human-Well Being: AI safety (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: AI applications for Improving Human-Well Being (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Devlin et al., 2014] Sam Devlin, Logan Yliniemi, Daniel Kudenko, et al. Potential-based difference rewards for multiagent reinforcement learning. In International Con- ference on Autonomous agents and Multi-Agent Systems (AAMAS), 2014.","title":"Potential-based difference rewards for multiagent reinforcement learning","authors":"Devlin","year":"2014","venue":"International Conference on Autonomous agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, et al. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelli- gence (AAAI), 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, et al. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza","year":"2014","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Horgan et al., 2018] Dan Horgan, John Quan, David Bud- den, et al. Distributed prioritized experience replay. In International Conference on Learning Representation (ICLR), 2018.","title":"Distributed prioritized experience replay","authors":"Dan Horgan, John Quan, David Budden","year":"2018","venue":"International Conference on Learning Representation (ICLR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kim and Shim, 2003] H Jin Kim and David H Shim. A flight control system for aerial robots: algorithms and ex- periments. Control engineering practice, 11(12):1389- 1400, 2003.","title":"A flight control system for aerial robots: algorithms and experiments","authors":"Jin Kim, David H Shim","year":"2003","venue":"Control engineering practice","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kurakin et al., 2016] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian Goodfellow, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lauer and Riedmiller, 2000] Martin Lauer and Martin Ried- miller. An algorithm for distributed reinforcement learn- ing in cooperative multi-agent systems. In International Conference on Machine Learning (ICML), 2000.","title":"An algorithm for distributed reinforcement learning in cooperative multi-agent systems","authors":"Riedmiller Lauer, Martin Lauer, Martin Riedmiller","year":"2000","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, et al. End-to-end training of deep visuomotor poli- cies. The Journal of Machine Learning Research (JMLR), 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2019] Shihui Li, Yi Wu, Xinyue Cui, et al. Ro- bust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In AAAI Conference on Ar- tificial Intelligence (AAAI), 2019.","title":"Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient","authors":"Shihui Li, Yi Wu, Xinyue Cui","year":"2019","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, et al. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mordatch and Abbeel, 2017] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional lan- guage in multi-agent populations. arXiv preprint arXiv:1703.04908, 2017.","title":"Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations","authors":"Abbeel Mordatch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nguyen et al., 2018] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multiagent RL with global rewards. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2018.","title":"Credit assignment for collective multiagent RL with global rewards","authors":"Akshat Duc Thien Nguyen, Hoong Chuin Kumar, Lau","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Panait and Luke, 2005] Liviu Panait and Sean Luke. Co- operative multi-agent learning: The state of the art. Au- tonomous Agents and Multi-Agent Systems, 11(3):387- 434, 2005.","title":"Cooperative multi-agent learning: The state of the art","authors":"Liviu Panait, Sean Luke","year":"2005","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pei et al., 2017] Kexin Pei, Yinzhi Cao, Junfeng Yang, et al. Deepxplore: Automated whitebox testing of deep learning systems. In Symposium on Operating Systems Principles (SOSP), 2017.","title":"Deepxplore: Automated whitebox testing of deep learning systems","authors":"Kexin Pei, Yinzhi Cao, Junfeng Yang","year":"2017","venue":"Symposium on Operating Systems Principles (SOSP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pinto et al., 2017] Lerrel Pinto, James Davidson, Rahul Sukthankar, et al. Robust adversarial reinforcement learn- ing. In International Conference on Machine Learning (ICML), 2017.","title":"Robust adversarial reinforcement learning","authors":"Pinto","year":"2017","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, et al. Prioritized experience replay. In Inter- national Conference on Learning Representation (ICLR), 2016.","title":"Prioritized experience replay","authors":"Schaul","year":"2016","venue":"International Conference on Learning Representation (ICLR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shah et al., 2017] Shital Shah, Debadeepta Dey, Chris Lovett, et al. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics (FSR), 2017.","title":"Airsim: High-fidelity visual and physical simulation for autonomous vehicles","authors":"Shital Shah, Debadeepta Dey, Chris Lovett","year":"2017","venue":"Field and Service Robotics (FSR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Mad- dison, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Stanton and Young, 1998] N. A. Stanton and M. S. Young. Vehicle automation and driving performance. Ergonomics, 41(7):1014-1028, 1998.","title":"Vehicle automation and driving performance","authors":"N A Stanton, M S Young","year":"1998","venue":"Ergonomics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, et al. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tian et al., 2018] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: Automated testing of deep- neural-network-driven autonomous cars. In International Conference on Software Engineering (ICSE), 2018.","title":"Deeptest: Automated testing of deepneural-network-driven autonomous cars","authors":"Yuchi Tian, Kexin Pei, Suman Jana, Baishakhi Ray","year":"2018","venue":"International Conference on Software Engineering (ICSE)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Urmson et al., 2008] Chris Urmson, Joshua Anhalt, Drew Bagnell, et al. Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics, 25(8):425-466, 2008.","title":"Autonomous driving in urban environments: Boss and the urban challenge","authors":"Chris Urmson, Joshua Anhalt, Drew Bagnell","year":"2008","venue":"Journal of Field Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vogel, 2003] Katja Vogel. A comparison of headway and time to collision as safety indicators. Accident analysis & prevention, 35(3):427-433, 2003.","title":"A comparison of headway and time to collision as safety indicators","authors":"Katja Vogel","year":"2003","venue":"Accident analysis & prevention","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wolpert and Tumer, 2002] David H Wolpert and Kagan Tumer. Optimal payoff functions for members of collec- tives. In Modeling complexity in economic and social sys- tems, pages 355-369. World Scientific, 2002.","title":"Optimal payoff functions for members of collectives","authors":"H David, Kagan Wolpert, Tumer","year":"2002","venue":"Modeling complexity in economic and social systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}