{"type":"ScholarlyArticle","title":"Recursive Small-Step Multi-Agent A* for Dec-POMDPs","authors":[{"name":"Wietze Koops","slug":"wietze-koops","given":"Wietze","family":"Koops","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Sebastian Junges","slug":"sebastian-junges","given":"Sebastian","family":"Junges","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Thiago D. Simão","slug":"thiago-d-simao","given":"Thiago D.","family":"Simão","affiliation":"Radboud University , Nijmegen , The Netherlands"}],"abstract":"We present recursive small-step multi-agent A* (RS-MAA*), an exact algorithm that optimizes the expected reward in decentralized partially observable Markov decision processes (Dec-POMDPs). RS-MAA* builds on multi-agent A* (MAA*), an algorithm that finds policies by exploring a search tree, but tackles two major scalability concerns. First, we employ a modified, small-step variant of the search tree that avoids the double exponential outdegree of the classical formulation. Second, we use a tight and recursive heuristic that we compute on-the-fly, thereby avoiding an expensive precomputation. The resulting algorithm is conceptually simple, yet it shows superior performance on a rich set of standard benchmarks.","doi":"10.24963/ijcai.2023/600","url":"https://ijcai.org/proceedings/2023/600","pdf":"https://ijcai.org/proceedings/2023/0600.pdf","markdown":"https://ijcai.org/proceedings/2023/600.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/600","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":600,"pages":{"first":5402,"last":5410},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent planning","Planning and Scheduling: PS: Planning under uncertainty","Planning and Scheduling: PS: Distributed and multi-agent planning"],"videos":[],"references":[{"position":1,"raw":"[Amato and Zilberstein, 2009] Christopher Amato and Shlomo Zilberstein. Achieving goals in decentralized POMDPs. In AAMAS, pages 593-600, 2009.","title":"Achieving goals in decentralized POMDPs","authors":"Christopher Amato, Shlomo Zilberstein","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amato et al., 2006] Christopher Amato, Daniel S. Bern- stein, and Shlomo Zilberstein. Optimal fixed-size con- trollers for decentralized POMDPs. In AAMAS MSDM workshop, 2006.","title":"Optimal fixed-size controllers for decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2006","venue":"AAMAS MSDM workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amato et al., 2007] Christopher Amato, Daniel S. Bern- stein, and Shlomo Zilberstein. Optimizing memory- bounded controllers for decentralized POMDPs. In UAI, pages 1-8, 2007.","title":"Optimizing memorybounded controllers for decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2007","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amato et al., 2009] Christopher Amato, Jilles Steeve Dibangoye, and Shlomo Zilberstein. Incremental policy generation for finite-horizon DEC-POMDPs. In ICAPS, 2009.","title":"Incremental policy generation for finite-horizon DEC-POMDPs","authors":"Christopher Amato, Jilles Steeve Dibangoye, Shlomo Zilberstein","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Amato et al., 2019] Christopher Amato, George Dimitri Konidaris, Leslie Pack Kaelbling, and Jonathan P. How. Modeling and planning with macro-actions in decentral- ized POMDPs. JAIR, 64:817-859, 2019.","title":"How. Modeling and planning with macro-actions in decentralized POMDPs","authors":"Christopher Amato, George, Dimitri Konidaris, Leslie Pack Kaelbling, Jonathan P","year":"2019","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barto and Mahadevan, 2003] Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforce- ment learning. Discret. Event Dyn. Syst., 13(4):341-379, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"Andrew G Barto, Sridhar Mahadevan","year":"2003","venue":"Discret. Event Dyn. Syst","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bernstein et al., 2002] Daniel S. Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The com- plexity of decentralized control of Markov decision pro- cesses. Mathematics of operations research, 27(4):819- 840, 2002.","title":"The complexity of decentralized control of Markov decision processes","authors":"S Daniel, Robert Bernstein, Neil Givan, Shlomo Immerman, Zilberstein","year":"2002","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cazenave, 2010] Tristan Cazenave. Partial move A*. In 22nd IEEE International Conference on Tools with Arti- ficial Intelligence, volume 2, pages 25-31. IEEE, 2010.","title":"Partial move A*","authors":"Tristan Cazenave","year":"2010","venue":"22nd IEEE International Conference on Tools with Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dibangoye et al., 2016] Jilles Steeve Dibangoye, Christo- pher Amato, Olivier Buffet, and Franc ¸ois Charpillet. Op- timally solving Dec-POMDPs as continuous-state MDPs. JAIR, 55:443-497, 2016.","title":"Optimally solving Dec-POMDPs as continuous-state MDPs","authors":"Jilles Steeve Dibangoye, Christopher Amato, Olivier Buffet, Franc ¸ois, Charpillet","year":"2016","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen et al., 2004] Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. Dynamic programming for par- tially observable stochastic games. In AAAI, pages 709- 715, 2004.","title":"Dynamic programming for partially observable stochastic games","authors":"Eric A Hansen, Daniel S Bernstein, Shlomo Zilberstein","year":"2004","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artif. Intell., 101(1-2):99-134, 1998.","title":"Planning and Acting in Partially Observable Stochastic Domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lauri et al., 2019] Mikko Lauri, Joni Pajarinen, and Jan Pe- ters. Information gathering in decentralized POMDPs by policy graph improvement. In AAMAS, pages 1143-1151.","title":"Information gathering in decentralized POMDPs by policy graph improvement","authors":"Lauri","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"IFAAMAS, 2019.","title":null,"authors":"Ifaamas","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lauri et al., 2020] Mikko Lauri, Joni Pajarinen, and Jan Pe- ters. Multi-agent active information gathering in dis- crete and continuous-state decentralized POMDPs by pol- icy graph improvement. Auton. Agents Multi Agent Syst., 34(2):42, 2020.","title":"Mikko Lauri, Joni Pajarinen, and Jan Peters. Multi-agent active information gathering in discrete and continuous-state decentralized POMDPs by policy graph improvement","authors":"Lauri","year":"2020","venue":"Auton. Agents Multi Agent Syst","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nair et al., 2003] Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming de- centralized POMDPs: Towards efficient policy computa- tion for multiagent settings. In IJCAI, pages 705-711, 2003.","title":"Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings","authors":"Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, Stacy Marsella","year":"2003","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2003/103"},{"position":16,"raw":"[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentralized POMDPs. Briefs in Intelligent Systems. Springer, 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"Frans A Oliehoek, Christopher Amato","year":"2016","venue":"Briefs in Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oliehoek and Vlassis, 2007] Frans A. Oliehoek and Nikos Vlassis. Q-value heuristics for approximate solutions of Dec-POMDPs. In AAAI Spring Symposium: Game Theo- retic and Decision Theoretic Agents, pages 31-37, 2007.","title":"Q-value heuristics for approximate solutions of Dec-POMDPs","authors":"Frans A Oliehoek, Nikos Vlassis","year":"2007","venue":"AAAI Spring Symposium: Game Theoretic and Decision Theoretic Agents","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oliehoek et al., 2008] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate Q- value functions for decentralized POMDPs. JAIR, 32:289- 353, 2008.","title":"Optimal and approximate Qvalue functions for decentralized POMDPs","authors":"Frans A Oliehoek, T J Matthijs, Nikos Spaan, Vlassis","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oliehoek et al., 2009] Frans A. Oliehoek, Shimon White- son, and Matthijs T. J. Spaan. Lossless clustering of his- tories in decentralized POMDPs. In AAMAS, pages 577- 584, 2009.","title":"Lossless clustering of histories in decentralized POMDPs","authors":"Frans A Oliehoek, Shimon Whiteson, T J Matthijs, Spaan","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oliehoek et al., 2013] Frans A. Oliehoek, Matthijs T. J. Spaan, Christopher Amato, and Shimon Whiteson. Incre- mental clustering and expansion for faster optimal plan- ning in Dec-POMDPs. JAIR, 46:449-509, 2013.","title":"Incremental clustering and expansion for faster optimal planning in Dec-POMDPs","authors":"Frans A Oliehoek, T J Matthijs, Christopher Spaan, Shimon Amato, Whiteson","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oliehoek et al., 2017] Frans A. Oliehoek, Matthijs T. J. Spaan, Bas Terwijn, Philipp Robbel, and João V. Mes- sias. The MADP toolbox: An open source library for plan- ning and learning in (multi-)agent systems. JMLR, 18:1-5, 2017.","title":"The MADP toolbox: An open source library for planning and learning in (multi-)agent systems","authors":"Frans A Oliehoek, T J Matthijs, Bas Spaan, Philipp Terwijn, João V Robbel, Messias","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pateria et al., 2022] Shubham Pateria, Budhitama Sub- agdja, Ah-Hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey. ACM Com- put. Surv., 54(5):109:1-109:35, 2022.","title":"Hierarchical reinforcement learning: A comprehensive survey","authors":"Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, Chai Quek","year":"2022","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Russell and Norvig, 2020] Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach (4th Edition).","title":"Artificial Intelligence: A Modern Approach","authors":"Stuart Russell, Peter Norvig","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Seuken and Zilberstein, 2007] Sven Seuken and Shlomo Zilberstein. Memory-bounded dynamic programming for DEC-POMDPs. In IJCAI, pages 2009-2015, 2007.","title":"Memory-bounded dynamic programming for DEC-POMDPs","authors":"Zilberstein Seuken, Sven Seuken, Shlomo Zilberstein","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/324"},{"position":25,"raw":"[Spaan and Melo, 2008] Matthijs T. J. Spaan and Fran- cisco S. Melo. Interaction-driven Markov games for de- centralized multiagent planning under uncertainty. In AA- MAS, pages 525-532, 2008.","title":"Interaction-driven Markov games for decentralized multiagent planning under uncertainty","authors":"T J Matthijs, Francisco S Spaan, Melo","year":"2008","venue":"AA-MAS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Spaan et al., 2011] Matthijs T. J. Spaan, Frans A. Oliehoek, and Christopher Amato. Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion. In IJ- CAI, 2011.","title":"Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion","authors":"T J Matthijs, Frans A Spaan, Christopher Oliehoek, Amato","year":"2011","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2011/338"},{"position":27,"raw":"[Szer et al., 2005] Daniel Szer, Franc ¸ois Charpillet, and Shlomo Zilberstein. MAA*: A heuristic search algorithm for solving decentralized POMDPs. In UAI, 2005.","title":"MAA*: A heuristic search algorithm for solving decentralized POMDPs","authors":"Daniel Szer, Shlomo Franc ¸ois Charpillet, Zilberstein","year":"2005","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2020] Yuxuan Xie, Jilles Dibangoye, and Olivier Buffet. Optimally solving two-agent decentralized POMDPs under one-sided information sharing. In ICML, volume 119 of Proceedings of Machine Learning Re- search, pages 10473-10482, 2020.","title":"Optimally solving two-agent decentralized POMDPs under one-sided information sharing","authors":"Yuxuan Xie, Jilles Dibangoye, Olivier Buffet","year":"2020","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}