{"type":"ScholarlyArticle","title":"Scalable Anytime Planning for Multi-Agent MDPs (Extended Abstract)","authors":[{"name":"Shushman Choudhury","slug":"shushman-choudhury","given":"Shushman","family":"Choudhury","affiliation":"Stanford University"},{"name":"Jayesh K. Gupta","slug":"jayesh-k-gupta","given":"Jayesh K.","family":"Gupta","affiliation":"Microsoft"},{"name":"Mykel J. Kochenderfer","slug":"mykel-j-kochenderfer","given":"Mykel J.","family":"Kochenderfer","affiliation":"Stanford University"}],"abstract":"We present a scalable planning algorithm for multi-agent sequential decision problems that require dynamic collaboration.\r\nTeams of agents need to coordinate decisions in many domains,\r\nbut naive approaches fail due to the exponential growth of the joint action space with the number of agents.\r\nWe circumvent this complexity through an anytime approach that allows us to trade computation for\r\napproximation quality and also dynamically coordinate actions. \r\nOur algorithm comprises three elements: online planning with Monte Carlo\r\nTree Search (MCTS), factorizing local agent interactions with coordination graphs, and\r\nselecting optimal joint actions with the Max-Plus method.\r\nOn the benchmark SysAdmin domain with static coordination graphs, our approach achieves comparable performance with much lower computation cost than the MCTS baselines.\r\nWe also introduce a multi-drone delivery domain with dynamic, i.e., state-dependent coordination graphs, and demonstrate how our approach scales to large problems on this domain that are intractable for other MCTS methods.","doi":"10.24963/ijcai.2022/735","url":"https://ijcai.org/proceedings/2022/735","pdf":"https://ijcai.org/proceedings/2022/0735.pdf","markdown":"https://ijcai.org/proceedings/2022/735.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/735","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":735,"pages":{"first":5279,"last":5283},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987258","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985401","length_sec":714,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amato and Oliehoek, 2014] Christopher Amato and Frans A Oliehoek. Scalable Planning and Learning for Mul- tiagent POMDPs: Extended version. arXiv preprint arXiv:1404.1140, 2014.","title":"Scalable Planning and Learning for Multiagent POMDPs: Extended version","authors":"Christopher Amato, Frans A Oliehoek","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 2005] Dimitri P Bertsekas. Dynamic Program- ming and Optimal Control, volume 1. Athena Scientific Belmont, MA, 2005.","title":"Dynamic Programming and Optimal Control","authors":"Dimitri P Bertsekas","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bezanson et al., 2017] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A Fresh Approach to Numerical Computing. SIAM Review, 59(1):65-98, 2017.","title":"Julia: A Fresh Approach to Numerical Computing","authors":"Jeff Bezanson, Alan Edelman, Stefan Karpinski, B Viral, Shah","year":"2017","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier, 1996] Craig Boutilier. Planning, Learning and Co- ordination in Multi-Agent Decision Processes. In Pro- ceedings of the Sixth Conference on Theoretical Aspects of Rationality and Knowledge, pages 195-210.","title":"Planning, Learning and Coordination in Multi-Agent Decision Processes","authors":"Craig Boutilier","year":null,"venue":"Proceedings of the Sixth Conference on Theoretical Aspects of Rationality and Knowledge","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Browne et al., 2012] Cameron Browne, Edward Jack Pow- ley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. A Survey of Monte Carlo Tree Search Methods. IEEE Trans- actions on Computational Intelligence and AI in games, 4(1):1-43, 2012.","title":"A Survey of Monte Carlo Tree Search Methods","authors":"Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, Simon Colton","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaslot et al., 2008] Guillaume M J-B Chaslot, Mark HM Winands, H Jaap van den Herik, Jos WHM Uiterwijk, and Bruno Bouzy. Progressive Strategies for Monte-Carlo Tree Search. New Mathematics and Natural Computation, 4(3):343-357, 2008.","title":"Progressive Strategies for Monte-Carlo Tree Search","authors":"M J-B Guillaume, Chaslot, H M Mark, H Winands, Jaap Van Den, Jos Whm Herik, Bruno Uiterwijk, Bouzy","year":"2008","venue":"New Mathematics and Natural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choudhury et al., 2020] Shushman Choudhury, Kiril Solovey, Mykel J. Kochenderfer, and Marco Pavone. Efficient Large-Scale Multi-Drone Delivery using Transit Networks. In IEEE International Conference on Robotics and Automation (ICRA), 2020.","title":"Efficient Large-Scale Multi-Drone Delivery using Transit Networks","authors":"Choudhury","year":"2020","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Choudhury et al., 2021] Shushman Choudhury, Jayesh K Gupta, Peter Morales, and Mykel J Kochenderfer. Scalable Anytime Planning for Multi-Agent MDPs. arXiv preprint arXiv:2101.04788, 2021.","title":"Scalable Anytime Planning for Multi-Agent MDPs","authors":"Shushman Choudhury, K Jayesh, Peter Gupta, Morales, Mykel, Kochenderfer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guestrin et al., 2002] Carlos Guestrin, Daphne Koller, and Ronald Parr. Multiagent Planning with Factored MDPs. In T G Dietterich, S Becker, and Z Ghahramani, editors, Advances in Neural Information Processing Systems, pages 1523-1530. MIT Press, 2002.","title":"Multiagent Planning with Factored MDPs","authors":"Carlos Guestrin, Daphne Koller, Ronald Parr","year":"2002","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guestrin et al., 2003] C. Guestrin, D. Koller, R. Parr, and S. Venkataraman. Efficient Solution Algorithms for Fac- tored MDPs. Journal of Artificial Intelligence Research, 19:399-468, October 2003.","title":"Efficient Solution Algorithms for Factored MDPs","authors":"C Guestrin, D Koller, R Parr, S Venkataraman","year":"2003","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gupta et al., 2017] Jayesh K Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative Multi-Agent Con- trol using Deep Reinforcement Learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 66-83. Springer, 2017.","title":"Cooperative Multi-Agent Control using Deep Reinforcement Learning","authors":"Maxim Jayesh K Gupta, Mykel J Egorov, Kochenderfer","year":"2017","venue":"International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kochenderfer, 2015] Mykel J. Kochenderfer. Decision Mak- ing under Uncertainty: Theory and Application. MIT Press, 2015.","title":"Decision Making under Uncertainty: Theory and Application","authors":"J Mykel, Kochenderfer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Eu- ropean Conference on Machine Learning (ECML), volume 4212, pages 282-293. Springer, 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"Eu-ropean Conference on Machine Learning (ECML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kok and Vlassis, 2004] Jelle R Kok and Nikos Vlassis. Sparse Cooperative Q-Learning. In International Confer- ence on Machine Learning (ICML), page 61. ACM, 2004.","title":"Sparse Cooperative Q-Learning","authors":"R Jelle, Nikos Kok, Vlassis","year":"2004","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kok and Vlassis, 2005] Jelle R. Kok and Nikos A. Vlassis. Using the Max-Plus Algorithm for Multi-Agent Decision Making in Coordination Graphs. In Proceedings of the Seventeenth Belgium-Netherlands Conference on Artificial Intelligence (BNAIC), pages 359-360, 2005.","title":"Using the Max-Plus Algorithm for Multi-Agent Decision Making in Coordination Graphs","authors":"R Jelle, Nikos A Kok, Vlassis","year":"2005","venue":"Proceedings of the Seventeenth Belgium-Netherlands Conference on Artificial Intelligence (BNAIC)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Matignon et al., 2012] Laetitia Matignon, Guillaume J. Lau- rent, and Nadine Le Fort-Piat. Independent Reinforcement Learners in Cooperative Markov Games: a Survey Regard- ing Coordination Problems. The Knowledge Engineering Review, 27(1):1-31, 2012.","title":"Independent Reinforcement Learners in Cooperative Markov Games: a Survey Regarding Coordination Problems","authors":"Laetitia Matignon, Guillaume J Laurent, Nadine Le Fort-Piat","year":"2012","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nijssen and Winands, 2011] J. A. M. Nijssen and Mark H. M. Winands. Enhancements for Multi-Player Monte- Carlo Tree Search. In Computers and Games, 2011.","title":"Enhancements for Multi-Player Monte-Carlo Tree Search","authors":"J A M Nijssen, H M Mark, Winands","year":"2011","venue":"Computers and Games","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oh et al., 2015] Kwang-Kyo Oh, Myoung-Chul Park, and Hyo-Sung Ahn. A Survey of Multi-Agent Formation Con- trol. Automatica, 53:424-440, 2015.","title":"A Survey of Multi-Agent Formation Control","authors":"Kwang-Kyo Oh, Myoung-Chul Park, Hyo-Sung Ahn","year":"2015","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oliehoek et al., 2008] Frans A. Oliehoek, Matthijs T. J. Spaan, et al. Exploiting Locality of Interaction in Fac- tored Dec-POMDPs. In International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS), pages 517-524. International Foundation for Autonomous Agents and Multiagent Systems, 2008.","title":"Exploiting Locality of Interaction in Factored Dec-POMDPs","authors":"Frans A Oliehoek, T J Matthijs, Spaan","year":"2008","venue":"International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pearl, 1989] Judea Pearl. Probabilistic Reasoning in Intel- ligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1989.","title":"Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference","authors":"Pearl","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Fac- torisation for Deep Multi-Agent Reinforcement Learning. In International Conference on Machine Learning (ICML), pages 4295-4304, 2018.","title":"QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo Planning in Large POMDPs. In Advances in Neural Information Processing Systems (NeurIPS), pages 2164-2172.","title":"Monte-Carlo Planning in Large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, et al. Value- Decomposition Networks for Cooperative Multi-Agent Learning based on Team Reward. In International Con- ference on Autonomous Agents and Multiagent Systems (AAMAS), pages 2085-2087, 2018.","title":"Value-Decomposition Networks for Cooperative Multi-Agent Learning based on Team Reward","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki","year":"2018","venue":"International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vlassis et al., 2004] N Vlassis, R Elhorst, and J R Kok. Any- time Algorithms for Multiagent Decision Making using Coordination Graphs. In IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), volume 1, pages 953-957 vol.1, October 2004.","title":"Anytime Algorithms for Multiagent Decision Making using Coordination Graphs","authors":"Vlassis, J R Elhorst, Kok","year":"2004","venue":"IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zerbel and Yliniemi, 2019] Nicholas Zerbel and Logan Yliniemi. Multiagent Monte Carlo Tree Search. In Interna- tional Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 2309-2311, 2019.","title":"Nicholas Zerbel and Logan Yliniemi. Multiagent Monte Carlo Tree Search","authors":"Yliniemi Zerbel","year":"2019","venue":"International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}