{"type":"ScholarlyArticle","title":"Decentralized MCTS via Learned Teammate Models","authors":[{"name":"Aleksander Czechowski","slug":"aleksander-czechowski","given":"Aleksander","family":"Czechowski","affiliation":"Delft University of Technology"},{"name":"Frans A. Oliehoek","slug":"frans-a-oliehoek","given":"Frans A.","family":"Oliehoek","affiliation":"Delft University of Technology"}],"abstract":"Decentralized online planning can be an attractive paradigm for cooperative \r\nmulti-agent systems, due to improved scalability and robustness.\r\nA key difficulty of such approach lies in making accurate\r\npredictions about the decisions of other agents.  \r\nIn this paper, we present a trainable online decentralized planning algorithm\r\nbased on decentralized Monte Carlo Tree Search, combined with\r\nmodels of teammates learned from previous episodic runs. \r\nBy only allowing one agent to adapt its models at a time, \r\nunder the assumption of ideal policy approximation,\r\nsuccessive iterations of our method are guaranteed to improve joint\r\npolicies, and eventually lead to convergence to a Nash equilibrium.  \r\nWe test the efficiency of the algorithm by performing experiments \r\nin several scenarios of the spatial task\r\nallocation environment introduced in [Claes et al., 2015].  We show that\r\ndeep learning and convolutional neural networks can be employed\r\nto produce accurate policy approximators which exploit the spatial features of the\r\nproblem, and that the proposed algorithm improves over the baseline\r\nplanning performance for particularly challenging domain configurations.","doi":"10.24963/ijcai.2020/12","url":"https://ijcai.org/proceedings/2020/12","pdf":"https://ijcai.org/proceedings/2020/0012.pdf","markdown":"https://ijcai.org/proceedings/2020/12.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/12","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":12,"pages":{"first":81,"last":88},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Multi-agent Planning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24388/file_24388.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Albrecht and Stone, 2018] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial In- telligence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"V Stefano, Peter Albrecht, Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amato and Oliehoek, 2015] Christopher Amato and Frans A Oliehoek. Scalable planning and learning for multiagent pomdps. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015.","title":"Scalable planning and learning for multiagent pomdps","authors":"Christopher Amato, Frans A Oliehoek","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[As ¸ık and Akın, 2012] Okan As ¸ık and H Levent Akın. Solv- ing multi-agent decision problems modeled as Dec- POMPD: A robot soccer case study. In Robot Soccer World Cup, pages 130-140, 2012.","title":"Solving multi-agent decision problems modeled as Dec-POMPD: A robot soccer case study","authors":"As ¸ık, ; ] Akın, As Okan, H ¸ık, Levent Akın","year":"2012","venue":"Robot Soccer World Cup","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Best et al., 2016] Graeme Best, O Cliff, Timothy Patten, Ramgopal Mettu, and Robert Fitch. Decentralised Monte Carlo tree search for active perception. In Workshop on the Algorithmic Foundations of Robotics, 2016.","title":"Decentralised Monte Carlo tree search for active perception","authors":"Graeme Best, Timothy Cliff, Ramgopal Patten, Robert Mettu, Fitch","year":"2016","venue":"Workshop on the Algorithmic Foundations of Robotics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bjarnason et al., 2009] Ronald Bjarnason, Alan Fern, and Prasad Tadepalli. Lower bounding Klondike solitaire with Monte-Carlo planning. In Nineteenth International Con- ference on Automated Planning and Scheduling, 2009.","title":"Lower bounding Klondike solitaire with Monte-Carlo planning","authors":"Ronald Bjarnason, Alan Fern, Prasad Tadepalli","year":"2009","venue":"Nineteenth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Borrajo and Fernández, 2019] Daniel Borrajo and Susana Fernández. Efficient approaches for multi-agent plan- ning. Knowledge and Information Systems, 58(2):425- 479, 2019.","title":"Efficient approaches for multi-agent planning","authors":"Fernández Borrajo, Daniel Borrajo, Susana Fernández","year":"2019","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chang et al., 2005] Hyeong Soo Chang, Michael C Fu, Ji- aqiao Hu, and Steven I Marcus. An adaptive sampling algorithm for solving Markov Decision Processes. Opera- tions Research, 53(1):126-139, 2005.","title":"An adaptive sampling algorithm for solving Markov Decision Processes","authors":"Hyeong Soo, Chang, Michael C Fu, Jiaqiao Hu, Steven I Marcus","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Claes et al., 2015] Daniel Claes, Philipp Robbel, Frans A. Oliehoek, Karl Tuyls, Daniel Hennes, and Wiebe Van der Hoek. Effective approximations for multi-robot coordina- tion in spatially distributed tasks. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems, pages 881-890, 2015.","title":"Effective approximations for multi-robot coordination in spatially distributed tasks","authors":"Daniel Claes, Philipp Robbel, Frans A Oliehoek, Karl Tuyls, Daniel Hennes, Wiebe Van Der Hoek","year":"2015","venue":"Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Claes et al., 2017] Daniel Claes, Frans Oliehoek, Hendrik Baier, and Karl Tuyls. Decentralised online planning for multi-robot warehouse commissioning. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems, pages 492-500, 2017.","title":"Decentralised online planning for multi-robot warehouse commissioning","authors":"Daniel Claes, Frans Oliehoek, Hendrik Baier, Karl Tuyls","year":"2017","venue":"Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Claus and Boutilier, 1998] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. Proceedings of the 15th National/10th Conference on Artificial intelli- gence/Innovative Applications of Artificial Intelligence, 1998(746-752):2, 1998.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"Claus, Boutilier, Caroline Claus, Craig Boutilier","year":"1998","venue":"Proceedings of the 15th National/10th Conference on Artificial intelligence/Innovative Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Doshi and Gmytrasiewicz, 2009] Prashant Doshi and Pi- otr J Gmytrasiewicz. Monte Carlo sampling methods for approximating interactive POMDPs. Journal of Artificial Intelligence Research, 34:297-337, 2009.","title":"Monte Carlo sampling methods for approximating interactive POMDPs","authors":"Prashant Doshi, Piotr J Gmytrasiewicz","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Durfee and Zilberstein, 2013] Edmund H Durfee and Shlomo Zilberstein. Multiagent planning, control, and execution. In Multiagent Systems, volume 11, pages 485-545. 2013.","title":"Multiagent planning, control, and execution","authors":"Zilberstein Durfee, Edmund H Durfee, Shlomo Zilberstein","year":"2013","venue":"Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Eck et al., 2019] Adam Eck, Maulik Shah, Prashant Doshi, and Leen-Kiat Soh. Scalable decision-theoretic planning in open and typed multiagent systems. arXiv preprint arXiv:1911.08642, 2019. to appear in Proceedings of the 34th AAAI Conference on Artificial Intelligence.","title":"Scalable decision-theoretic planning in open and typed multiagent systems","authors":"Adam Eck, Maulik Shah, Prashant Doshi, Leen-Kiat Soh","year":"2019","venue":"Proceedings of the 34th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gmytrasiewicz and Doshi, 2005] Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:49-79, 2005.","title":"A framework for sequential planning in multi-agent settings","authors":"J Piotr, Prashant Gmytrasiewicz, Doshi","year":"2005","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gmytrasiewicz and Durfee, 1995] Piotr J. Gmytrasiewicz and Edmund H. Durfee. A rigorous, operational formaliza- tion of recursive modeling. In Proceedings of the 1st Inter- national Conference on Multiagent Systems, pages 125- 132, 1995.","title":"A rigorous, operational formalization of recursive modeling","authors":"Piotr J Gmytrasiewicz, Edmund H Durfee","year":"1995","venue":"Proceedings of the 1st International Conference on Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Golpayegani et al., 2015] Fatemeh Golpayegani, Ivana Dusparic, and Siobhan Clarke. Collaborative, parallel Monte Carlo tree search for autonomous electricity demand management. In 2015 Sustainable Internet and ICT for Sustainability, pages 1-8, 2015.","title":"Collaborative, parallel Monte Carlo tree search for autonomous electricity demand management","authors":"Fatemeh Golpayegani, Ivana Dusparic, Siobhan Clarke","year":"2015","venue":"2015 Sustainable Internet and ICT for Sustainability","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hernandez-Leal et al., 2017] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A Survey of Learning in Multiagent Environ- ments: Dealing with Non-Stationarity. arXiv preprint arXiv:1707.09183, 2017.","title":"A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity","authors":"Hernandez-Leal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Agent modeling as aux- iliary task for deep reinforcement learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 31-37, 2019.","title":"Agent modeling as auxiliary task for deep reinforcement learning","authors":"Hernandez-Leal","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jaderberg et al., 2019] Max Jaderberg, Wojciech M Czar- necki, Iain Dunning, Luke Marris, Guy Lever, Anto- nio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):859- 865, 2019.","title":"Human-level performance in 3d multiplayer games with populationbased reinforcement learning","authors":"Jaderberg","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Eu- ropean conference on machine learning, pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kurzer et al., 2018] Karl Kurzer, Chenyang Zhou, and J Marius Zöllner. Decentralized cooperative planning for automated vehicles with hierarchical Monte Carlo tree search. In IEEE Intelligent Vehicles Symposium, pages 529-536, 2018.","title":"Decentralized cooperative planning for automated vehicles with hierarchical Monte Carlo tree search","authors":"Karl Kurzer, Chenyang Zhou, Marius Zöllner","year":"2018","venue":"IEEE Intelligent Vehicles Symposium","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2019] Minglong Li, Wenjing Yang, Zhongxuan Cai, Shaowu Yang, and Ji Wang. Integrating decision shar- ing with prediction in decentralized planning for multi- agent coordination under uncertainty. In Proceedings of the 28th International Joint Conference on Artificial Intel- ligence, pages 450-456, 2019.","title":"Integrating decision sharing with prediction in decentralized planning for multiagent coordination under uncertainty","authors":"Li","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lin, 1965] Shen Lin. Computer solutions of the travel- ing salesman problem. Bell System Technical Journal, 44(10):2245-2269, 1965.","title":"Computer solutions of the traveling salesman problem","authors":"Lin ;, Shen Lin","year":"1965","venue":"Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nair et al., 2003] Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming de- centralized POMDPs: Towards efficient policy computa- tion for multiagent settings. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, volume 3, pages 705-711, 2003.","title":"Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings","authors":"Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, Stacy Marsella","year":"2003","venue":"Proceedings of the 18th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2003/103"},{"position":26,"raw":"[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentralized POMDPs. 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"Frans A Oliehoek, Christopher Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peshkin et al., 2000] Leonid Peshkin, Kee-Eung Kim, Nicolas Meuleau, and Leslie Pack Kaelbling. Learning to cooperate via policy search. In Proceedings of the Six- teenth conference on Uncertainty in artificial intelligence, pages 489-496, 2000.","title":"Learning to cooperate via policy search","authors":"Leonid Peshkin, Kee-Eung Kim, Nicolas Meuleau, Leslie Pack, Kaelbling","year":"2000","venue":"Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Premack and Woodruff, 1978] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.","title":"Does the chimpanzee have a theory of mind?","authors":"Woodruff ; ] David Premack, Guy Premack, Woodruff","year":"1978","venue":"Behavioral and brain sciences","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rabinowitz et al., 2018] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Proceed- ings of the International Conference on Machine Learn- ing, pages 4218-4227, July 2018.","title":"Machine theory of mind","authors":"Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S M Ali Eslami, Matthew Botvinick","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Vu et al., 2018] Huan Vu, Samir Aknine, and Sarvapali D Ramchurn. A decentralised approach to intersection traf- fic management. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 527- 533, 2018.","title":"A decentralised approach to intersection traffic management","authors":"Huan Vu, Samir Aknine, Sarvapali D Ramchurn","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/73"},{"position":33,"raw":"[Wu et al., 2009] Feng Wu, Shlomo Zilberstein, and Xiaop- ing Chen. Multi-agent online planning with communi- cation. In Nineteenth International Conference on Auto- mated Planning and Scheduling, 2009.","title":"Multi-agent online planning with communication","authors":"Feng Wu, Shlomo Zilberstein, Xiaoping Chen","year":"2009","venue":"Nineteenth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}