{"type":"ScholarlyArticle","title":"Toward Policy Explanations for Multi-Agent Reinforcement Learning","authors":[{"name":"Kayla Boggess","slug":"kayla-boggess","given":"Kayla","family":"Boggess","affiliation":"University of Virginia"},{"name":"Sarit Kraus","slug":"sarit-kraus","given":"Sarit","family":"Kraus","affiliation":"Bar-Ilan University"},{"name":"Lu Feng","slug":"lu-feng","given":"Lu","family":"Feng","affiliation":"University of Virginia"}],"abstract":"Advances in multi-agent reinforcement learning (MARL) enable sequential decision making for a range of exciting multi-agent applications such as cooperative AI and autonomous driving. Explaining agent decisions is crucial for improving system transparency, increasing user satisfaction, and facilitating human-agent collaboration. However, existing works on explainable reinforcement learning mostly focus on the single-agent setting and are not suitable for addressing challenges posed by multi-agent environments. We present novel methods to generate two types of policy explanations for MARL: (i) policy summarization about the agent cooperation and task sequence, and (ii) language explanations to answer queries about agent behavior. Experimental results on three MARL domains demonstrate the scalability of our methods. A user study shows that the generated explanations significantly improve user performance and increase subjective ratings on metrics such as user satisfaction.","doi":"10.24963/ijcai.2022/16","url":"https://ijcai.org/proceedings/2022/16","pdf":"https://ijcai.org/proceedings/2022/0016.pdf","markdown":"https://ijcai.org/proceedings/2022/16.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/16","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":16,"pages":{"first":109,"last":115},"keywords":["Agent-based and Multi-agent Systems: Human-Agent Interaction"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987079","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985222","length_sec":657,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amir and Amir, 2018] Dan Amir and Ofra Amir. High- lights: Summarizing agent behavior to people. In Proceed- ings of the 17th International Conference on Autonomous Agents and Multi Agent Systems, pages 1168-1176, 2018.","title":"Highlights: Summarizing agent behavior to people","authors":"Amir Amir, Dan Amir, Ofra Amir","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and Multi Agent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boggess et al., 2022] Kayla Boggess, Sarit Kraus, and Lu Feng. Toward policy explanations for multi-agent re- inforcement learning. arXiv preprint arXiv:2204.12568, 2022.","title":"Toward policy explanations for multi-agent reinforcement learning","authors":"Kayla Boggess, Sarit Kraus, Lu Feng","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chakraborti et al., 2020] Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The emerging landscape of explainable automated planning & decision making. In IJCAI, pages 4803-4811, 2020.","title":"The emerging landscape of explainable automated planning & decision making","authors":"Tathagata Chakraborti, Sarath Sreedharan, Subbarao Kambhampati","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/669"},{"position":4,"raw":"[Christianos et al., 2020] Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In Thirty-fourth Conference on Neural Information Process- ing Systems, pages 10707-10717.","title":"Shared experience actor-critic for multi-agent reinforcement learning","authors":"Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":null,"venue":"Thirty-fourth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dafoe et al., 2020] Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in coop- erative ai. arXiv preprint arXiv:2012.08630, 2020.","title":"Open problems in cooperative ai","authors":"Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R Mckee, Joel Z Leibo, Kate Larson, Thore Graepel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dijkstra et al., 1959] Edsger W Dijkstra, others, et al. A note on two problems in connexion with graphs. Nu- merische mathematik, 1(1):269-271, 1959.","title":"A note on two problems in connexion with graphs","authors":"Edsger W Dijkstra","year":"1959","venue":"Numerische mathematik","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hayes and Shah, 2017] Bradley Hayes and Julie A Shah. Improving robot controller transparency through au- tonomous policy explanation. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pages 303-312. IEEE, 2017.","title":"Improving robot controller transparency through autonomous policy explanation","authors":"Bradley Hayes, Julie A Shah","year":"2017","venue":"12th ACM/IEEE International Conference on Human-Robot Interaction (HRI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heuillet et al., 2021] Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. Explain- ability in deep reinforcement learning. Knowledge-Based Systems, 214:106685, 2021.","title":"Explainability in deep reinforcement learning","authors":"Alexandre Heuillet, Fabien Couthouis, Natalia Díaz-Rodríguez","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoffman et al., 2018] Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explain- able ai: Challenges and prospects. arXiv preprint arXiv:1812.04608, 2018.","title":"Metrics for explainable ai: Challenges and prospects","authors":"Shane T Robert R Hoffman, Gary Mueller, Jordan Klein, Litman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kazhdan et al., 2020] Dmitry Kazhdan, Zohreh Shams, and Pietro Liò. Marleme: A multi-agent reinforcement learn- ing model extraction library. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.","title":"Marleme: A multi-agent reinforcement learning model extraction library","authors":"Dmitry Kazhdan, Zohreh Shams, Pietro Liò","year":"2020","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiran et al., 2021] B Ravi Kiran, Ibrahim Sobh, Victor Tal- paert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yo- gamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.","title":"Deep reinforcement learning for autonomous driving: A survey","authors":"Ibrahim Ravi Kiran, Victor Sobh, Patrick Talpaert, Ahmad A Al Mannion, Senthil Sallab, Patrick Yogamani, Pérez","year":"2021","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kraus et al., 2020] Sarit Kraus, Amos Azaria, Jelena Fiosina, Maike Greve, Noam Hazon, Lutz Kolbe, Tim- Benjamin Lembcke, Jorg P Muller, Soren Schleibaum, and Mark Vollrath. Ai for explaining decisions in multi-agent environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13534-13538, 2020.","title":"Ai for explaining decisions in multi-agent environments","authors":"Sarit Kraus, Amos Azaria, Jelena Fiosina, Maike Greve, Noam Hazon, Lutz Kolbe, Tim-Benjamin Lembcke, P Jorg, Soren Muller, Mark Schleibaum, Vollrath","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.","title":"Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks","authors":"Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":"2021","venue":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Puiutta and Veith, 2020] Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In Inter- national Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 77-95. Springer, 2020.","title":"Explainable reinforcement learning: A survey","authors":"Veith Puiutta, Erika Puiutta, Eric Msp Veith","year":"2020","venue":"International Cross-Domain Conference for Machine Learning and Knowledge Extraction","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Quine, 1952] Willard V Quine. The problem of simplify- ing truth functions. The American mathematical monthly, 59(8):521-531, 1952.","title":"The problem of simplifying truth functions","authors":"Willard V Quine","year":"1952","venue":"The American mathematical monthly","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Topin and Veloso, 2019] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-2521, 2019.","title":"Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning","authors":"Veloso Topin","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wells and Bednarz, 2021] Lindsay Wells and Tomasz Bed- narz. Explainable ai and reinforcement learning-a sys- tematic review of current approaches and trends. Frontiers in artificial intelligence, 4:48, 2021.","title":"Lindsay Wells and Tomasz Bednarz. Explainable ai and reinforcement learning-a systematic review of current approaches and trends","authors":"Bednarz Wells","year":"2021","venue":"Frontiers in artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}