{"type":"ScholarlyArticle","title":"Explainable Multi-Agent Reinforcement Learning for Temporal Queries","authors":[{"name":"Kayla Boggess","slug":"kayla-boggess","given":"Kayla","family":"Boggess","affiliation":"University of Virginia"},{"name":"Sarit Kraus","slug":"sarit-kraus","given":"Sarit","family":"Kraus","affiliation":"Bar-Ilan University"},{"name":"Lu Feng","slug":"lu-feng","given":"Lu","family":"Feng","affiliation":"University of Virginia"}],"abstract":"As multi-agent reinforcement learning (MARL) systems are increasingly deployed throughout society, it is imperative yet challenging for users to understand the emergent behaviors of MARL agents in complex environments. This work presents an approach for generating policy-level contrastive explanations for MARL to answer a temporal user query, which specifies a sequence of tasks completed by agents with possible cooperation. The proposed approach encodes the temporal query as a PCTL* logic formula and checks if the query is feasible under a given MARL policy via probabilistic model checking. Such explanations can help reconcile discrepancies between the actual and anticipated multi-agent behaviors. The proposed approach also generates correct and complete explanations to pinpoint reasons that make a user query infeasible. We have successfully applied the proposed approach to four benchmark MARL domains (up to 9 agents in one domain). Moreover, the results of a user study show that the generated explanations significantly improve user performance and satisfaction.","doi":"10.24963/ijcai.2023/7","url":"https://ijcai.org/proceedings/2023/7","pdf":"https://ijcai.org/proceedings/2023/0007.pdf","markdown":"https://ijcai.org/proceedings/2023/7.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/7","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":7,"pages":{"first":55,"last":63},"keywords":["Agent-based and Multi-agent Systems: MAS: Human-agent interaction"],"videos":[],"references":[{"position":1,"raw":"[Amir and Amir, 2018] Dan Amir and Ofra Amir. High- lights: Summarizing agent behavior to people. In Proceed- ings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1168-1176, 2018.","title":"Highlights: Summarizing agent behavior to people","authors":"Amir Amir, Dan Amir, Ofra Amir","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atrey et al., 2019] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counter- factual analysis of saliency maps for deep reinforcement learning. In International Conference on Learning Repre- sentations, 2019.","title":"Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning","authors":"Akanksha Atrey, Kaleigh Clary, David Jensen","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aziz et al., 1995] Adnan Aziz, Vigyan Singhal, Felice Balarin, Robert K Brayton, and Alberto L Sangiovanni- Vincentelli. It usually works: The temporal logic of stochastic systems. In International Conference on Com- puter Aided Verification, pages 155-165. Springer, 1995.","title":"It usually works: The temporal logic of stochastic systems","authors":"Adnan Aziz, Vigyan Singhal, Felice Balarin, Robert K Brayton, Alberto L Sangiovanni-Vincentelli","year":"1995","venue":"International Conference on Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Katoen. Principles of model checking. MIT press, 2008.","title":"Principles of model checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boggess et al., 2022] Kayla Boggess, Sarit Kraus, and Lu Feng. Toward policy explanations for multi-agent rein- forcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.","title":"Toward policy explanations for multi-agent reinforcement learning","authors":"Kayla Boggess, Sarit Kraus, Lu Feng","year":"2022","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/16"},{"position":6,"raw":"[Chandra and Markowsky, 1978] Ashok K Chandra and George Markowsky. On the number of prime implicants. Discrete Mathematics, 24(1):7-11, 1978.","title":"On the number of prime implicants","authors":"K Ashok, George Chandra, Markowsky","year":"1978","venue":"Discrete Mathematics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Christianos et al., 2020] Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In Thirty-fourth Conference on Neural Information Process- ing Systems, pages 10707-10717.","title":"Shared experience actor-critic for multi-agent reinforcement learning","authors":"Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":null,"venue":"Thirty-fourth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Danesh et al., 2021] Mohamad H Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. Re-understanding finite- state representations of recurrent policy networks. In Inter- national Conference on Machine Learning, pages 2388- 2397. PMLR, 2021.","title":"Re-understanding finitestate representations of recurrent policy networks","authors":"Mohamad H Danesh, Anurag Koul, Alan Fern, Saeed Khorram","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Finkelstein et al., 2022] Mira Finkelstein, Lucy Liu, Yoav Kolumbus, David C Parkes, Jeffrey Rosenschein, Sarah Keren, et al. Explainable reinforcement learning via model transforms. In Advances in Neural Information Processing Systems, 2022.","title":"Explainable reinforcement learning via model transforms","authors":"Mira Finkelstein, Lucy Liu, Yoav Kolumbus, David C Parkes, Jeffrey Rosenschein, Sarah Keren","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hayes and Shah, 2017] Bradley Hayes and Julie A Shah. Improving robot controller transparency through au- tonomous policy explanation. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 303-312. IEEE, 2017.","title":"Improving robot controller transparency through autonomous policy explanation","authors":"Bradley Hayes, Julie A Shah","year":"2017","venue":"12th ACM/IEEE International Conference on Human-Robot Interaction (HRI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Heuillet et al., 2021] Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. Explain- ability in deep reinforcement learning. Knowledge-Based Systems, 214:106685, 2021.","title":"Explainability in deep reinforcement learning","authors":"Alexandre Heuillet, Fabien Couthouis, Natalia Díaz-Rodríguez","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heuillet et al., 2022] Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. Collective explainable ai: Explaining cooperative strategies and agent contribution in multiagent reinforcement learning with shapley values. IEEE Computational Intelligence Magazine, 17(1):59-71, 2022.","title":"Collective explainable ai: Explaining cooperative strategies and agent contribution in multiagent reinforcement learning with shapley values","authors":"Alexandre Heuillet, Fabien Couthouis, Natalia Díaz-Rodríguez","year":"2022","venue":"IEEE Computational Intelligence Magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoffman et al., 2018] Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explain- able ai: Challenges and prospects. arXiv preprint arXiv:1812.04608, 2018.","title":"Metrics for explainable ai: Challenges and prospects","authors":"Shane T Robert R Hoffman, Gary Mueller, Jordan Klein, Litman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Juozapaitis et al., 2019] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Ex- plainable reinforcement learning via reward decomposi- tion. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.","title":"Explainable reinforcement learning via reward decomposition","authors":"Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, Finale Doshi-Velez","year":"2019","venue":"IJCAI/ECAI Workshop on explainable artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kwiatkowska et al., 2011] Marta Kwiatkowska, Gethin Norman, and David Parker. Prism 4.0: Verification of probabilistic real-time systems. In Computer Aided Verification: 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings 23, pages 585-591. Springer, 2011.","title":"Prism 4.0: Verification of probabilistic real-time systems","authors":"Marta Kwiatkowska, Gethin Norman, David Parker","year":"2011","venue":"Computer Aided Verification: 23rd International Conference, CAV 2011","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kwiatkowska et al., 2018] Marta Kwiatkowska, Gethin Norman, and David Parker. Probabilistic model checking: Advances and applications. Formal System Verification: State-of the-Art and Future Trends, pages 73-121, 2018.","title":"Probabilistic model checking: Advances and applications","authors":"Marta Kwiatkowska, Gethin Norman, David Parker","year":"2018","venue":"Formal System Verification: State-of the-Art and Future Trends","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Landajuela et al., 2021] Mikel Landajuela, Brenden K Pe- tersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Dis- covering symbolic policies with deep reinforcement learn- ing. In International Conference on Machine Learning, pages 5979-5989. PMLR, 2021.","title":"Discovering symbolic policies with deep reinforcement learning","authors":"Mikel Landajuela, Sookyung Brenden K Petersen, Claudio P Kim, Ruben Santiago, Nathan Glatt, Jacob F Mundhenk, Daniel Pettit, Faissol","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2021] Zhengxian Lin, Kin-Ho Lam, and Alan Fern. Contrastive explanations for reinforcement learning via embedded self predictions. In International Confer- ence on Learning Representations, 2021.","title":"Contrastive explanations for reinforcement learning via embedded self predictions","authors":"Lin","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Madumal et al., 2020] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2493-2500, 2020.","title":"Explainable reinforcement learning through a causal lens","authors":"Prashan Madumal, Tim Miller, Liz Sonenberg, Frank Vetere","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[McInroe and Christianos, 2022] Trevor McInroe and Filip- pos Christianos. Pressureplate. https://github.com/uoe- agents/pressureplate, 2022. Accessed: 2022-11-22.","title":"Trevor McInroe and Filippos Christianos. Pressureplate","authors":"Christianos Mcinroe","year":"2022","venue":null,"doi":null,"url":"https://github.com/uoe-agents/pressureplate,2022","internal_url":null},{"position":21,"raw":"[Miller, 2019] Tim Miller. Explanation in artificial intelli- gence: Insights from the social sciences. Artificial intelli- gence, 267:1-38, 2019.","title":"Explanation in artificial intelligence: Insights from the social sciences","authors":"Tim Miller, Miller","year":"2019","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Olson et al., 2021] Matthew L Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong. Coun- terfactual state explanations for reinforcement learning agents via generative deep learning. Artificial Intelligence, 295:103455, 2021.","title":"Counterfactual state explanations for reinforcement learning agents via generative deep learning","authors":"Roli Matthew L Olson, Lawrence Khanna, Fuxin Neal, Weng-Keen Li, Wong","year":"2021","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.","title":"Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks","authors":"Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":"2021","venue":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puiutta and Veith, 2020] Erika Puiutta and Eric Veith. Ex- plainable reinforcement learning: A survey. In Interna- tional cross-domain conference for machine learning and knowledge extraction, pages 77-95. Springer, 2020.","title":"International cross-domain conference for machine learning and knowledge extraction","authors":"Veith Puiutta, Erika Puiutta, Eric Veith","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Quine, 1952] Willard V Quine. The problem of simplify- ing truth functions. The American mathematical monthly, 59(8):521-531, 1952.","title":"The problem of simplifying truth functions","authors":"Willard V Quine","year":"1952","venue":"The American mathematical monthly","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sreedharan et al., 2022] Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, and Subbarao Kamb- hampati. Bridging the gap: Providing post-hoc sym- bolic explanations for sequential decision-making prob- lems with inscrutable representations. In International Conference on Learning Representations, 2022.","title":"Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations","authors":"Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, Subbarao Kambhampati","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Topin and Veloso, 2019] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-2521, 2019.","title":"Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning","authors":"Veloso Topin","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Topin et al., 2021] Nicholay Topin, Stephanie Milani, Fei Fang, and Manuela Veloso. Iterative bounding mdps: Learning interpretable policies via non-interpretable meth- ods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9923-9931, 2021.","title":"Iterative bounding mdps: Learning interpretable policies via non-interpretable methods","authors":"Topin","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wells and Bednarz, 2021] Lindsay Wells and Tomasz Bed- narz. Explainable ai and reinforcement learning-a sys- tematic review of current approaches and trends. Frontiers in artificial intelligence, 4:48, 2021.","title":"Lindsay Wells and Tomasz Bednarz. Explainable ai and reinforcement learning-a systematic review of current approaches and trends","authors":"Bednarz Wells","year":"2021","venue":"Frontiers in artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}