{"type":"ScholarlyArticle","title":"Multi-Agent Intention Progression with Reward Machines","authors":[{"name":"Michael Dann","slug":"michael-dann","given":"Michael","family":"Dann","affiliation":"RMIT University"},{"name":"Yuan Yao","slug":"yuan-yao","given":"Yuan","family":"Yao","affiliation":"University of Nottingham , Ningbo China"},{"name":"Natasha Alechina","slug":"natasha-alechina","given":"Natasha","family":"Alechina","affiliation":"Utrecht University"},{"name":"Brian Logan","slug":"brian-logan","given":"Brian","family":"Logan","affiliation":"Utrecht University; University of Aberdeen"},{"name":"John Thangarajah","slug":"john-thangarajah","given":"John","family":"Thangarajah","affiliation":"RMIT University"}],"abstract":"Recent work in multi-agent intention scheduling has shown that enabling agents to predict the actions of other agents when choosing their own actions can be beneficial. However existing approaches to 'intention-aware' scheduling assume that the programs of other agents are known, or are \"similar\" to that of the agent making the prediction. While this assumption is reasonable in some circumstances, it is less plausible when the agents are not co-designed. In this paper, we present a new approach to multi-agent intention scheduling in which agents predict the actions of other agents based on a high-level specification of the tasks performed by an agent in the form of a reward machine (RM) rather than on its (assumed) program. We show how a reward machine can be used to generate tree and rollout policies for an MCTS-based scheduler. We evaluate our approach in a range of multi-agent environments, and show that RM-based scheduling out-performs previous intention-aware scheduling approaches in settings where agents are not co-designed","doi":"10.24963/ijcai.2022/31","url":"https://ijcai.org/proceedings/2022/31","pdf":"https://ijcai.org/proceedings/2022/0031.pdf","markdown":"https://ijcai.org/proceedings/2022/31.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/31","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":31,"pages":{"first":215,"last":222},"keywords":["Agent-based and Multi-agent Systems: Agent Theories and Models","Agent-based and Multi-agent Systems: Engineering Methods, Platforms, Languages and Tools"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987049","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985192","length_sec":671,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Andreas et al., 2017] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th Inter- national Conference on Machine Learning (ICML 2017), pages 166-175. PMLR, 2017.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML 2017)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Broekens et al., 2010] Joost Broekens, Maaike Harbers, Koen V. Hindriks, Karel van den Bosch, Catholijn M. Jonker, and John-Jules Ch. Meyer. Do you get it? user- evaluated explainable BDI agents. In Proceedings of the 8th German Conference on Multiagent System Technolo- gies (MATES 2010), volume 6251 of LNCS, pages 28-39.","title":"Do you get it? userevaluated explainable BDI agents","authors":"Joost Broekens, Maaike Harbers, Koen V Hindriks, Karel Van Den, Catholijn M Bosch, John-Jules Jonker, Ch, Meyer","year":null,"venue":"Proceedings of the 8th German Conference on Multiagent System Technologies (MATES 2010)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Camacho et al., 2019] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. LTL and beyond: Formal lan- guages for reward function specification in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI 2019), pages 6065-6073. IJCAI, 2019.","title":"LTL and beyond: Formal languages for reward function specification in reinforcement learning","authors":"Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, Sheila A Mcilraith","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI 2019)","doi":null,"url":null,"internal_url":"/proceedings/2019/840"},{"position":4,"raw":"[Dann et al., 2020] Michael Dann, John Thangarajah, Yuan Yao, and Brian Logan. Intention-aware multiagent scheduling. In Proceedings of the 19th International Con- ference on Autonomous Agents and Multiagent Systems (AAMAS 2020), pages 285-293.","title":"Intention-aware multiagent scheduling","authors":"Michael Dann, John Thangarajah, Yuan Yao, Brian Logan","year":null,"venue":"Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"IFAAMAS, 2020.","title":null,"authors":"Ifaamas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dann et al., 2021] Michael Dann, Yuan Yao, Brian Logan, and John Thangarajah. Multi-agent intention progression with black box agents. In Proceedings of the 30th Inter- national Joint Conference on Artificial Intelligence (IJCAI 2021). IJCAI, 2021.","title":"Multi-agent intention progression with black box agents","authors":"Michael Dann, Yuan Yao, Brian Logan, John Thangarajah","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021). IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/19"},{"position":7,"raw":"[de Silva et al., 2020] Lavindra de Silva, Felipe Meneguzzi, and Brian Logan. BDI agent architectures: A survey. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020). IJCAI, 2020.","title":"Lavindra de Silva, Felipe Meneguzzi, and Brian Logan. BDI agent architectures: A survey","authors":"Silva De","year":"2020","venue":"Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020). IJCAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gelly and Silver, 2008] Sylvain Gelly and David Silver. Achieving master level play in 9 x 9 computer go. In Proceedings of the Twenty-Third AAAI Conference on Ar- tificial Intelligence (AAAI-08), pages 1537-1540. AAAI Press, 2008.","title":"Achieving master level play in 9 x 9 computer go","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2008","venue":"Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (AAAI-08)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Illanes et al., 2020] León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila A McIlraith. Symbolic plans as high- level instructions for reinforcement learning. In Pro- ceedings of the Thirtieth International Conference on Au- tomated Planning and Scheduling (ICAPS 2020), pages 540-550. AAAI Press, 2020.","title":"Symbolic plans as highlevel instructions for reinforcement learning","authors":"León Illanes, Xi Yan, Rodrigo Toro Icarte, Sheila A Mcilraith","year":"2020","venue":"Proceedings of the Thirtieth International Conference on Automated Planning and Scheduling (ICAPS 2020)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Logan et al., 2017] Brian Logan, John Thangarajah, and Neil Yorke-Smith. Progressing intention progresson: A call for a goal-plan tree contest. In Proceedings of the 16th International Conference on Autonomous Agents and Mul- tiagent Systems (AAMAS 2017), pages 768-772.","title":"Progressing intention progresson: A call for a goal-plan tree contest","authors":"Brian Logan, John Thangarajah, Neil Yorke-Smith","year":null,"venue":"Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2017)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Norling and Ritter, 2004] Emma Norling and Frank E. Rit- ter. Towards supporting psychologically plausible variabil- ity in agent-based human modelling. In Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2004), pages 758-765. IEEE Computer Society, 2004.","title":"Towards supporting psychologically plausible variability in agent-based human modelling","authors":"Ritter Norling, Emma Norling, Frank E Ritter","year":"2004","venue":"Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2004)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rao and Georgeff, 1992] Anand S. Rao and Michael P. Georgeff. An abstract architecture for rational agents. In Proceedings of the 3rd International Conference on Prin- ciples of Knowledge Representation and Reasoning (KR 1992), pages 439-449.","title":"An abstract architecture for rational agents","authors":"Anand S Rao, Michael P Georgeff","year":"1992","venue":"Proceedings of the 3rd International Conference on Principles of Knowledge Representation and Reasoning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Morgan Kaufmann, 1992.","title":null,"authors":"Morgan Kaufmann","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-Che ; Marc, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Thangarajah and Padgham, 2011] John Thangarajah and Lin Padgham. Computationally effective reasoning about goal interactions. Journal of Automated Reasoning, 47(1):17-56, 2011.","title":"Computationally effective reasoning about goal interactions","authors":"Padgham Thangarajah, John Thangarajah, Lin Padgham","year":"2011","venue":"Journal of Automated Reasoning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Thangarajah et al., 2003] John Thangarajah, Lin Padgham, and Michael Winikoff. Detecting & avoiding interference between goals in intelligent agents. In Proceedings of the 18th International Joint Conference on Artificial In- telligence (IJCAI-03), pages 721-726.","title":"Detecting & avoiding interference between goals in intelligent agents","authors":"John Thangarajah, Lin Padgham, Michael Winikoff","year":null,"venue":"Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI-03)","doi":null,"url":null,"internal_url":"/proceedings/2003/105"},{"position":17,"raw":"Morgan Kaufmann, 2003.","title":null,"authors":"Morgan Kaufmann","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Toro Icarte et al., 2018] Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines for high-level task specification and de- composition in reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), pages 2107-2116. PMLR, 2018.","title":"Using reward machines for high-level task specification and decomposition in reinforcement learning","authors":"Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, Sheila Mcilraith","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML 2018)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Waters et al., 2014] Max Waters, Lin Padgham, and Sebas- tian Sardiña. Evaluating coverage based intention selec- tion. In Proceedings of the 13th International Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2014), pages 957-964. IFAAMAS, 2014.","title":"Evaluating coverage based intention selection","authors":"Waters","year":"2014","venue":"Proceedings of the 13th International Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2014)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Waters et al., 2015] Max Waters, Lin Padgham, and Sebas- tian Sardiña. Improving domain-independent intention se- lection in BDI systems. Autonomous Agents and Multi- Agent Systems, 29(4):683-717, 2015.","title":"Improving domain-independent intention selection in BDI systems","authors":"Waters","year":"2015","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yao and Logan, 2016] Yuan Yao and Brian Logan. Action- level intention selection for BDI agents. In Proceed- ings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016), pages 1227-1236. IFAAMAS, 2016.","title":"Actionlevel intention selection for BDI agents","authors":"Yuan Yao, Brian Logan","year":"2016","venue":"Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yao et al., 2014] Yuan Yao, Brian Logan, and John Thangarajah. SP-MCTS-based intention scheduling for BDI agents. In Proceedings of the 21st European Confer- ence on Artificial Intelligence (ECAI-2014), pages 1133- 1134. ECCAI, IOS Press, 2014.","title":"SP-MCTS-based intention scheduling for BDI agents","authors":"Yuan Yao, Brian Logan, John Thangarajah","year":"2014","venue":"Proceedings of the 21st European Conference on Artificial Intelligence (ECAI-2014)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2016a] Yuan Yao, Lavindra de Silva, and Brian Logan. Reasoning about the executability of goal-plan trees. In Proceedings of the 4th International Workshop on Engineering Multi-Agent Systems (EMAS 2016), vol- ume 10093 of LNCS, pages 176-191. Springer, 2016.","title":"Reasoning about the executability of goal-plan trees","authors":"Yao","year":"2016","venue":"Proceedings of the 4th International Workshop on Engineering Multi-Agent Systems (EMAS 2016)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yao et al., 2016b] Yuan Yao, Brian Logan, and John Thangarajah. Robust execution of BDI agent programs by exploiting synergies between intentions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), pages 2558-2565. AAAI Press, 2016.","title":"Robust execution of BDI agent programs by exploiting synergies between intentions","authors":"Yao","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}