{"type":"ScholarlyArticle","title":"EMOTE: An Explainable Architecture for Modelling the Other through Empathy","authors":[{"name":"Manisha Senadeera","slug":"manisha-senadeera","given":"Manisha","family":"Senadeera","affiliation":"Applied Artificial Intelligence Institute , Deakin University"},{"name":"Thommen Karimpanal George","slug":"thommen-karimpanal-george","given":"Thommen","family":"Karimpanal George","affiliation":"Applied Artificial Intelligence Institute , Deakin University; School of Information Technology , Deakin University"},{"name":"Stephan Jacobs","slug":"stephan-jacobs","given":"Stephan","family":"Jacobs","affiliation":"University of Queensland {msenadeera ,"},{"name":"Sunil Gupta","slug":"sunil-gupta","given":"Sunil","family":"Gupta","affiliation":"Applied Artificial Intelligence Institute , Deakin University"},{"name":"Santu Rana","slug":"santu-rana","given":"Santu","family":"Rana","affiliation":"Applied Artificial Intelligence Institute , Deakin University"}],"abstract":"Empathy allows us to assume others are like us and have goals analogous to our own. This can also at times be applied to multi-agent games - e.g. Agent 1's attraction to green balls is analogous to Agent 2's attraction to red balls. \r\nDrawing inspiration from empathy, we propose EMOTE, a simple and explainable inverse reinforcement learning (IRL) approach designed to model another agent's action-value function and from it, infer a unique reward function. This is done by referencing the learning agent's own action value function, removing the need to maintain independent action-value estimates for the modelled agents whilst simultaneously addressing the ill-posed nature of IRL by inferring a unique reward function. We experiment on minigrid environments showing EMOTE: (a) produces more consistent reward estimates relative to other IRL baselines (b) is robust in scenarios with composite reward and action-value functions (c) produces human-interpretable states, helping to explain how the agent views other agents.","doi":"10.24963/ijcai.2024/539","url":"https://ijcai.org/proceedings/2024/539","pdf":"https://ijcai.org/proceedings/2024/0539.pdf","markdown":"https://ijcai.org/proceedings/2024/539.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/539","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":539,"pages":{"first":4876,"last":4884},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Reinforcement learning","AI Ethics, Trust, Fairness: ETF: Explainability and interpretability","Machine Learning: ML: Multiagent Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Alamdari et al., 2021] Parand Alizadeh Alamdari, Toryn Q Klassen, Rodrigo Toro Icarte, and Sheila A McIlraith. Be considerate: Objectives, side effects, and deciding how to act. arXiv preprint arXiv:2106.02617, 2021.","title":"Be considerate: Objectives, side effects, and deciding how to act","authors":"Parand Alizadeh Alamdari, Rodrigo Toryn Q Klassen, Sheila A Toro Icarte, Mcilraith","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boularias et al., 2011] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 182-189. JMLR Workshop and Conference Proceedings, 2011.","title":"Relative entropy inverse reinforcement learning","authors":"Abdeslam Boularias, Jens Kober, Jan Peters","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bussmann et al., 2019] Bart Bussmann, Jacqueline Hein- erman, and Joel Lehman. Towards empathic deep q- learning. In Huáscar Espinoza, Han Yu, Xiaowei Huang, Freddy Lecue, Cynthia Chen, José Hernández-Orallo, Seán Ó h Éigeartaigh, and Richard Mallah, editors, Arti- ficial Intelligence Safety 2019, CEUR Workshop Proceed- ings, pages 1-7. CEUR-WS.org, 8 2019.","title":"Towards empathic deep qlearning","authors":"Bart Bussmann, Jacqueline Heinerman, Joel Lehman","year":"2019","venue":"Artificial Intelligence Safety 2019, CEUR Workshop Proceedings","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium, 2018.","title":"Minimalistic gridworld environment for gymnasium","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster et al., 2018] Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In AAMAS 2018: Proceedings of the Sev- enteenth International Joint Conference on Autonomous Agents and Multi-Agent Systems, July 2018.","title":"Learning with opponent-learning awareness","authors":"Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch","year":"2018","venue":"AAMAS 2018: Proceedings of the Seventeenth International Joint Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé, III. Opponent modeling in deep rein- forcement learning. In Maria Florina Balcan and Kil- ian Q. Weinberger, editors, Proceedings of The 33rd Inter- national Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1804- 1813, New York, New York, USA, 20-22 Jun 2016. PMLR.","title":"Opponent modeling in deep reinforcement learning","authors":"He","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hoffman, 1996] Martin L Hoffman. Empathy and moral de- velopment. The annual report of educational psychology in Japan, 35:157-162, 1996.","title":"Empathy and moral development","authors":"Martin L Hoffman","year":"1996","venue":"The annual report of educational psychology in Japan","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2020] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. \"other-play\" for zero-shot coordination. In International Conference on Machine Learning, pages 4399-4410. PMLR, 2020.","title":"other-play\" for zero-shot coordination","authors":"Hengyuan Hu, Adam Lerer, Alex Peysakhovich, Jakob Foerster","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Klein et al., 2012] Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. In NIPS, 2012.","title":"Inverse reinforcement learning through structured classification","authors":"Edouard Klein, Matthieu Geist, Bilal Piot, Olivier Pietquin","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Klein et al., 2013] Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pietquin. A cascaded supervised learn- ing approach to inverse reinforcement learning. In Hen- drik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Železný, editors, Machine Learning and Knowledge Discovery in Databases, pages 1-16, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.","title":"A cascaded supervised learning approach to inverse reinforcement learning","authors":"Edouard Klein, Bilal Piot, Matthieu Geist, Olivier Pietquin","year":"2013","venue":"Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2017] Xiaomin Lin, Peter A Beling, and Randy Cogill. Multiagent inverse reinforcement learning for two- person zero-sum games. IEEE Transactions on Games, 10(1):56-68, 2017.","title":"Multiagent inverse reinforcement learning for twoperson zero-sum games","authors":"Xiaomin Lin, Peter A Beling, Randy Cogill","year":"2017","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moreno et al., 2021] Pol Moreno, Edward Hughes, Kevin R McKee, Bernardo Avila Pires, and Théophane Weber. Neural recursive belief states in multi-agent reinforcement learning. arXiv preprint arXiv:2102.02274, 2021.","title":"Neural recursive belief states in multi-agent reinforcement learning","authors":"Pol Moreno, Edward Hughes, Kevin R Mckee, Bernardo Avila Pires, Théophane Weber","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Natarajan et al., 2010] Sriraam Natarajan, Gautam Kuna- puli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude Shavlik. Multi-agent inverse reinforcement learning. In 2010 ninth international conference on ma- chine learning and applications, pages 395-400. IEEE, 2010.","title":"Multi-agent inverse reinforcement learning","authors":"Natarajan","year":"2010","venue":"2010 ninth international conference on machine learning and applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ndousse, 2020] Kamal Ndousse. Marlgrid. https://github. com/kandouss/marlgrid, 2020.","title":"Marlgrid","authors":"Kamal Ndousse","year":"2020","venue":null,"doi":null,"url":"https://github.com/kandouss/marlgrid","internal_url":null},{"position":16,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceed- ings of the Sixteenth International Conference on Machine Learning, ICML '99, page 278-287, San Francisco, CA, USA, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart J Russell","year":"1999","venue":"Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Al- gorithms for inverse reinforcement learning. In Icml, vol- ume 1, page 2, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Noothigattu et al., 2019] Ritesh Noothigattu, Djallel Boun- effouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush R Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi. Teaching ai agents ethical values us- ing reinforcement learning and policy orchestration. IBM Journal of Research and Development, 63(4/5):2-1, 2019.","title":"Teaching ai agents ethical values using reinforcement learning and policy orchestration","authors":"Ritesh Noothigattu, Djallel Bouneffouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, R Kush, Murray Varshney, Moninder Campbell, Francesca Singh, Rossi","year":"2019","venue":"IBM Journal of Research and Development","doi":null,"url":null,"internal_url":"/proceedings/2019/891"},{"position":19,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19222, 2021.","title":"Agent modelling under partial observability for deep reinforcement learning","authors":"Georgios Papoudakis, Filippos Christianos, Stefano Albrecht","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rabinowitz et al., 2018] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Jen- nifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Research, pages 4218-4227. PMLR, 10-15 Jul 2018.","title":"Machine theory of mind","authors":"Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S M Ali Eslami, Matthew Botvinick","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raileanu et al., 2018] Roberta Raileanu, Emily L. Denton, Arthur D. Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In ICML, 2018.","title":"Modeling others using oneself in multi-agent reinforcement learning","authors":"Roberta Raileanu, Emily L Denton, Arthur D Szlam, Rob Fergus","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reddy et al., 2012] Tummalapalli Sudhamsh Reddy, Vam- sikrishna Gopikrishna, Gergely Zaruba, and Manfred Hu- ber. Inverse reinforcement learning for decentralized non- cooperative multiagent systems. In 2012 IEEE Inter- national Conference on Systems, Man, and Cybernetics (SMC), pages 1930-1935. IEEE, 2012.","title":"Tummalapalli Sudhamsh Reddy, Vamsikrishna Gopikrishna, Gergely Zaruba, and Manfred Huber. Inverse reinforcement learning for decentralized non-cooperative multiagent systems","authors":"Reddy","year":"2012","venue":"2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Senadeera et al., 2022] Manisha Senadeera, Thom- men George Karimpanal, Sunil Gupta, and Santu Rana. Sympathy-based reinforcement learning agents. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, page 1164-1172, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems.","title":"Sympathy-based reinforcement learning agents","authors":"Manisha Senadeera, Thommen George Karimpanal, Sunil Gupta, Santu Rana","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shu and Tian, 2018] Tianmin Shu and Yuandong Tian. M 3 rl: Mind-aware multi-agent management reinforcement learning. arXiv preprint arXiv:1810.00147, 2018.","title":"Tianmin Shu and Yuandong Tian. M 3 rl: Mind-aware multi-agent management reinforcement learning","authors":"Tian Shu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2019] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207, 2019.","title":"Probabilistic recursive reasoning for multi-agent reinforcement learning","authors":"Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, Wei Pan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2023] Haochen Wu, Pedro Sequeira, and David V. Pynadath. Multiagent inverse reinforcement learning via theory of mind reasoning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, page 708-716, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems.","title":"Multiagent inverse reinforcement learning via theory of mind reasoning","authors":"Haochen Wu, Pedro Sequeira, David V Pynadath","year":"2023","venue":"Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2022] Jian Zhao, Mingyu Yang, Youpeng Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Mcmarl: Parameterizing value function via mixture of categorical distributions for multi-agent rein- forcement learning, 2022.","title":"Mcmarl: Parameterizing value function via mixture of categorical distributions for multi-agent reinforcement learning","authors":"Jian Zhao, Mingyu Yang, Youpeng Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, Houqiang Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"Aaai","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}