{"type":"ScholarlyArticle","title":"Approximate Dec-POMDP Solving Using Multi-Agent A*","authors":[{"name":"Wietze Koops","slug":"wietze-koops","given":"Wietze","family":"Koops","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Sebastian Junges","slug":"sebastian-junges","given":"Sebastian","family":"Junges","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Ruhr-University Bochum , Germany; Radboud University , Nijmegen , The Netherlands"}],"abstract":"We present an A*-based algorithm to compute policies for finite-horizon Dec-POMDPs. Our goal is to sacrifice optimality in favor of scalability for larger horizons. The main ingredients of our approach are (1) using clustered sliding window memory, (2) pruning the A* search tree, and (3) using novel A* heuristics. Our experiments show competitive performance to the state-of-the-art. Moreover, for multiple benchmarks, we achieve superior performance. In addition, we provide an A* algorithm that finds upper bounds for the optimum, tailored towards problems with long horizons. The main ingredient is a new heuristic that periodically reveals the state, thereby limiting the number of reachable beliefs. Our experiments demonstrate the efficacy and scalability of the approach.","doi":"10.24963/ijcai.2024/745","url":"https://ijcai.org/proceedings/2024/745","pdf":"https://ijcai.org/proceedings/2024/0745.pdf","markdown":"https://ijcai.org/proceedings/2024/745.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/745","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":745,"pages":{"first":6743,"last":6751},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent planning","Planning and Scheduling: PS: Planning under uncertainty","Planning and Scheduling: PS: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Abdoo et al., 2022] Eliran Abdoo, Ronen I. Brafman, Guy Shani, and Nitsan Soffair. Team-Imitate-Synchronize for solving Dec-POMDPs. In ECML/PKDD, pages 216-232. Springer, 2022.","title":"Team-Imitate-Synchronize for solving Dec-POMDPs","authors":"Eliran Abdoo, Ronen I Brafman, Guy Shani, Nitsan Soffair","year":"2022","venue":"ECML/PKDD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amato and Zilberstein, 2009] Christopher Amato and Shlomo Zilberstein. Achieving goals in decentralized POMDPs. In AAMAS, pages 593-600, 2009.","title":"Achieving goals in decentralized POMDPs","authors":"Christopher Amato, Shlomo Zilberstein","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amato et al., 2006] Christopher Amato, Daniel S. Bern- stein, and Shlomo Zilberstein. Optimal fixed-size con- trollers for decentralized POMDPs. In AAMAS MSDM workshop, 2006.","title":"Optimal fixed-size controllers for decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2006","venue":"AAMAS MSDM workshop","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amato et al., 2007] Christopher Amato, Daniel S. Bern- stein, and Shlomo Zilberstein. Optimizing memory- bounded controllers for decentralized POMDPs. In UAI, pages 1-8, 2007.","title":"Optimizing memorybounded controllers for decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2007","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Amato et al., 2009] Christopher Amato, Jilles Steeve Dibangoye, and Shlomo Zilberstein. Incremental policy generation for finite-horizon DEC-POMDPs. In ICAPS, 2009.","title":"Incremental policy generation for finite-horizon DEC-POMDPs","authors":"Christopher Amato, Jilles Steeve Dibangoye, Shlomo Zilberstein","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bernstein et al., 2002] Daniel S. Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The com- plexity of decentralized control of Markov decision pro- cesses. Mathematics of operations research, 27(4):819- 840, 2002.","title":"The complexity of decentralized control of Markov decision processes","authors":"S Daniel, Robert Bernstein, Neil Givan, Shlomo Immerman, Zilberstein","year":"2002","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bhustali and Andriotis, 2023] Prateek Bhustali and Char- alampos P. Andriotis. Assessing the optimality of decentralized inspection and maintenance policies for stochastically degrading engineering systems. In BNAIC/BeNeLearn 2023: Joint International Scientific Conferences on AI and Machine Learning, 2023.","title":"Assessing the optimality of decentralized inspection and maintenance policies for stochastically degrading engineering systems","authors":"Prateek Bhustali, Charalampos P Andriotis","year":"2023","venue":"BNAIC/BeNeLearn 2023: Joint International Scientific Conferences on AI and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bono et al., 2018] Guillaume Bono, Jilles Steeve Diban- goye, Laëtitia Matignon, Florian Pereyron, and Olivier Simonin. Cooperative multi-agent policy gradient. In ECML/PKDD, volume 11051 of Lecture Notes in Com- puter Science, pages 459-476. Springer, 2018.","title":"Cooperative multi-agent policy gradient","authors":"Guillaume Bono, Jilles Steeve Dibangoye, Laëtitia Matignon, Florian Pereyron, Olivier Simonin","year":"2018","venue":"ECML/PKDD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Carlin and Zilberstein, 2008] Alan Carlin and Shlomo Zil- berstein. Value-based observation compression for DEC- POMDPs. In AAMAS, pages 501-508, 2008.","title":"Alan Carlin and Shlomo Zilberstein. Value-based observation compression for DEC-POMDPs","authors":"Zilberstein Carlin","year":"2008","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cazenave, 2010] Tristan Cazenave. Partial move A*. In 22nd IEEE International Conference on Tools with Arti- ficial Intelligence, volume 2, pages 25-31. IEEE, 2010.","title":"Partial move A*","authors":"Tristan Cazenave","year":"2010","venue":"22nd IEEE International Conference on Tools with Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dibangoye and Buffet, 2018] Jilles Steeve Dibangoye and Olivier Buffet. Learning to act in continuous Dec- POMDPs. In JFPDA. HAL, 2018.","title":"Learning to act in continuous Dec-POMDPs","authors":"Buffet ; Dibangoye, Steeve Jilles, Olivier Dibangoye, Buffet","year":"2018","venue":"JFPDA. HAL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dibangoye et al., 2009] Jilles Steeve Dibangoye, Abdel- Illah Mouaddib, and Brahim Chaib-draa. Point-based incremental pruning heuristic for solving finite-horizon DEC-POMDPs. In AAMAS, pages 569-576, 2009.","title":"Point-based incremental pruning heuristic for solving finite-horizon DEC-POMDPs","authors":"Dibangoye","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dibangoye et al., 2014] Jilles Steeve Dibangoye, Christo- pher Amato, Olivier Buffet, and Franc ¸ois Charpillet. Op- timally solving Dec-POMDPs as continuous-state MDPs: Theory and algorithms. Technical report, Tech. Rep. RR- 8517, Inria, 2014.","title":"Optimally solving Dec-POMDPs as continuous-state MDPs: Theory and algorithms","authors":"Jilles Steeve Dibangoye, Christopher Amato, Olivier Buffet, Franc ¸ois, Charpillet","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dibangoye et al., 2016] Jilles Steeve Dibangoye, Christo- pher Amato, Olivier Buffet, and Franc ¸ois Charpillet. Op- timally solving Dec-POMDPs as continuous-state MDPs. JAIR, 55:443-497, 2016.","title":"Optimally solving Dec-POMDPs as continuous-state MDPs","authors":"Jilles Steeve Dibangoye, Christopher Amato, Olivier Buffet, Franc ¸ois, Charpillet","year":"2016","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Eker and Akin, 2013] Baris Eker and H. Levent Akin. Solv- ing decentralized POMDP problems using genetic algo- rithms. AAMAS, 27(1):161-196, 2013.","title":"Solving decentralized POMDP problems using genetic algorithms","authors":"Baris Eker, H Levent Akin","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Emery-Montemerlo et al., 2004] Rosemary Emery- Montemerlo, Geoffrey J. Gordon, Jeff G. Schneider, and Sebastian Thrun. Approximate solutions for partially observable stochastic games with common payoffs. In AAMAS, pages 136-143, 2004.","title":"Approximate solutions for partially observable stochastic games with common payoffs","authors":"Rosemary Emery-Montemerlo, Geoffrey J Gordon, Jeff G Schneider, Sebastian Thrun","year":"2004","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hansen et al., 2004] Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. Dynamic programming for par- tially observable stochastic games. In AAAI, pages 709- 715, 2004.","title":"Dynamic programming for partially observable stochastic games","authors":"Eric A Hansen, Daniel S Bernstein, Shlomo Zilberstein","year":"2004","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hemmati et al., 2015] Mahdi Hemmati, Abdulsalam Yas- sine, and Shervin Shirmohammadi. A Dec-POMDP model for congestion avoidance and fair allocation of network bandwidth in rate-adaptive video streaming. In SSCI, pages 1182-1189. IEEE, 2015.","title":"Mahdi Hemmati, Abdulsalam Yassine, and Shervin Shirmohammadi. A Dec-POMDP model for congestion avoidance and fair allocation of network bandwidth in rate-adaptive video streaming","authors":"Hemmati","year":"2015","venue":"SSCI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Koops et al., 2023] Wietze Koops, Nils Jansen, Sebastian Junges, and Thiago D. Simão. Recursive small-step multi- agent A* for Dec-POMDPs. In IJCAI, pages 5402-5410, 2023.","title":"Recursive small-step multiagent A* for Dec-POMDPs","authors":"Wietze Koops, Nils Jansen, Sebastian Junges, Thiago D Simão","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kraemer and Banerjee, 2016] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomput- ing, 190:82-94, 2016.","title":"Multi-agent reinforcement learning as a rehearsal for decentralized planning","authors":"Landon Kraemer, Bikramjit Banerjee","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kumar and Zilberstein, 2009] Akshat Kumar and Shlomo Zilberstein. Dynamic programming approximations for partially observable stochastic games. In FLAIRS. AAAI Press, 2009.","title":"Dynamic programming approximations for partially observable stochastic games","authors":"Akshat Kumar, Shlomo Zilberstein","year":"2009","venue":"FLAIRS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Littman et al., 1995] Michael L. Littman, Anthony R. Cas- sandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In ICML, pages 362-370, 1995.","title":"Learning policies for partially observable environments: Scaling up","authors":"L Michael, Anthony R Littman, Leslie Cassandra, Kaelbling Pack","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mao et al., 2020] Weichao Mao, Kaiqing Zhang, Erik Miehling, and Tamer Basar. Information state embedding in partially observable cooperative multi-agent reinforce- ment learning. In CDC, pages 6124-6131. IEEE, 2020.","title":"Information state embedding in partially observable cooperative multi-agent reinforcement learning","authors":"Mao","year":"2020","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nair et al., 2003] Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming de- centralized POMDPs: Towards efficient policy computa- tion for multiagent settings. In IJCAI, pages 705-711, 2003.","title":"Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings","authors":"Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, Stacy Marsella","year":"2003","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2003/103"},{"position":25,"raw":"[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentralized POMDPs. Briefs in Intelligent Systems. Springer, 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"Frans A Oliehoek, Christopher Amato","year":"2016","venue":"Briefs in Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Oliehoek et al., 2008a] Frans A. Oliehoek, Julian F. P. Kooij, and Nikos Vlassis. The cross-entropy method for policy search in decentralized POMDPs. Informatica (Slovenia), 32(4):341-357, 2008.","title":"The cross-entropy method for policy search in decentralized POMDPs","authors":"Oliehoek","year":"2008","venue":"Informatica (Slovenia)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Oliehoek et al., 2008b] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate Q- value functions for decentralized POMDPs. JAIR, 32:289- 353, 2008.","title":"Optimal and approximate Qvalue functions for decentralized POMDPs","authors":"Oliehoek","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Oliehoek et al., 2009] Frans A. Oliehoek, Shimon White- son, and Matthijs T. J. Spaan. Lossless clustering of his- tories in decentralized POMDPs. In AAMAS, pages 577- 584, 2009.","title":"Lossless clustering of histories in decentralized POMDPs","authors":"Frans A Oliehoek, Shimon Whiteson, T J Matthijs, Spaan","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Oliehoek et al., 2013] Frans A. Oliehoek, Matthijs T. J. Spaan, Christopher Amato, and Shimon Whiteson. Incre- mental clustering and expansion for faster optimal plan- ning in Dec-POMDPs. JAIR, 46:449-509, 2013.","title":"Incremental clustering and expansion for faster optimal planning in Dec-POMDPs","authors":"Frans A Oliehoek, T J Matthijs, Christopher Spaan, Shimon Amato, Whiteson","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rabinovich et al., 2003] Zinovi Rabinovich, Claudia V. Goldman, and Jeffrey S. Rosenschein. The complexity of multiagent systems: the price of silence. In AAMAS, pages 1102-1103, 2003.","title":"The complexity of multiagent systems: the price of silence","authors":"Rabinovich","year":"2003","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Roth et al., 2005] Maayan Roth, Reid G. Simmons, and Manuela M. Veloso. Reasoning about joint beliefs for execution-time communication decisions. In AAMAS, pages 786-793, 2005.","title":"Reasoning about joint beliefs for execution-time communication decisions","authors":"Maayan Roth, Reid G Simmons, Manuela M Veloso","year":"2005","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Russell and Norvig, 2020] Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach (4th Edition).","title":"Artificial Intelligence: A Modern Approach","authors":"Stuart Russell, Peter Norvig","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Russell, 1992] Stuart Russell. Efficient memory-bounded search methods. In ECAI, pages 1-5, 1992.","title":"Efficient memory-bounded search methods","authors":"Stuart Russell","year":"1992","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Seuken and Zilberstein, 2007] Sven Seuken and Shlomo Zilberstein. Improved memory-bounded dynamic pro- gramming for decentralized POMDPs. In UAI, pages 344- 351. AUAI Press, 2007.","title":"Improved memory-bounded dynamic programming for decentralized POMDPs","authors":"Zilberstein Seuken, Sven Seuken, Shlomo Zilberstein","year":"2007","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Spaan and Melo, 2008] Matthijs T. J. Spaan and Fran- cisco S. Melo. Interaction-driven Markov games for de- centralized multiagent planning under uncertainty. In AA- MAS, pages 525-532, 2008.","title":"Interaction-driven Markov games for decentralized multiagent planning under uncertainty","authors":"T J Matthijs, Francisco S Spaan, Melo","year":"2008","venue":"AA-MAS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Szer et al., 2005] Daniel Szer, Franc ¸ois Charpillet, and Shlomo Zilberstein. MAA*: A heuristic search algorithm for solving decentralized POMDPs. In UAI, pages 576- 590, 2005.","title":"MAA*: A heuristic search algorithm for solving decentralized POMDPs","authors":"Daniel Szer, Shlomo Franc ¸ois Charpillet, Zilberstein","year":"2005","venue":"UAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}