{"type":"ScholarlyArticle","title":"Rethinking Formal Models of Partially Observable Multiagent Decision Making (Extended Abstract)","authors":[{"name":"Vojtěch Kovařík","slug":"vojtech-kovarik","given":"Vojtěch","family":"Kovařík","affiliation":"AI Center , FEE , Czech; Foundations of Cooperative AI Lab , Carnegie Mellon University , Pittsburgh"},{"name":"Martin Schmid","slug":"martin-schmid","given":"Martin","family":"Schmid","affiliation":"University of Alberta , Computing Science , Faculty of Science 4 DeepMind , Alberta , Edmonton"},{"name":"Neil Burch","slug":"neil-burch","given":"Neil","family":"Burch","affiliation":"University of Alberta , Computing Science , Faculty of Science 4 DeepMind , Alberta , Edmonton"},{"name":"Michael Bowling","slug":"michael-bowling","given":"Michael","family":"Bowling","affiliation":"University of Alberta , Computing Science , Faculty of Science 4 DeepMind , Alberta , Edmonton"},{"name":"Viliam Lisý","slug":"viliam-lisy","given":"Viliam","family":"Lisý","affiliation":"AI Center , FEE , Czech; Technical University in Prague"}],"abstract":"Multiagent decision-making in partially observable environments is usually modelled as either an extensive-form game (EFG) in game theory or a partially observable stochastic game (POSG) in multiagent reinforcement learning (MARL). One issue with the current situation is that while most practical problems can be modelled in both formalisms, the relationship of the two models is unclear, which hinders the transfer of ideas between the two communities. A second issue is that while EFGs have recently seen significant algorithmic progress, their classical formalization is unsuitable for efficient presentation of the underlying ideas, such as those around decomposition.\r\nTo solve the first issue, we introduce factored-observation stochastic games (FOSGs), a minor modification of the POSG formalism which distinguishes between private and public observation and thereby greatly simplifies decomposition. To remedy the second issue, we show that FOSGs and POSGs are naturally connected to EFGs: by \"unrolling\" a FOSG into its tree form, we obtain an EFG. Conversely, any perfect-recall timeable EFG corresponds to some underlying FOSG in this manner. Moreover, this relationship justifies several minor modifications to the classical EFG formalization that recently appeared as an implicit response to the model's issues with decomposition. Finally, we illustrate the transfer of ideas between EFGs and MARL by presenting three key EFG techniques -- counterfactual regret minimization, sequence form, and decomposition -- in the FOSG framework.","doi":"10.24963/ijcai.2023/783","url":"https://ijcai.org/proceedings/2023/783","pdf":"https://ijcai.org/proceedings/2023/0783.pdf","markdown":"https://ijcai.org/proceedings/2023/783.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/783","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":783,"pages":{"first":6920,"last":6924},"keywords":["Game Theory and Economic Paradigms: GTEP: Noncooperative games","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Agent theories and models"],"videos":[],"references":[{"position":1,"raw":"[Badia et al., 2020] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outper- forming the atari human benchmark. arXiv preprint arXiv:2003.13350, 2020.","title":"Agent57: Outperforming the atari human benchmark","authors":"Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Charles Blundell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown and Sandholm, 2017a] Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfect- information games. In Advances in Neural Information Processing Systems, pages 689-699, 2017.","title":"Safe and nested subgame solving for imperfectinformation games","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown and Sandholm, 2017b] Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, page eaao1733, 2017.","title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown and Sandholm, 2019] Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Sci- ence, 365(6456):885-890, 2019.","title":"Superhuman AI for multiplayer poker","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2018] Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfect- information games. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Sys- tems, pages 7674-7685, 2018.","title":"Depth-limited solving for imperfectinformation games","authors":"Brown","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown et al., 2020] Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforce- ment learning and search for imperfect-information games. arXiv preprint arXiv:2007.13544, 2020.","title":"Combining deep reinforcement learning and search for imperfect-information games","authors":"Brown","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Burch et al., 2014] Neil Burch, Michael Johanson, and Michael Bowling. Solving imperfect information games using decomposition. In AAAI, pages 602-608, 2014.","title":"Solving imperfect information games using decomposition","authors":"Neil Burch, Michael Johanson, Michael Bowling","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Davis et al., 2019] Trevor Davis, Martin Schmid, and Michael Bowling. Low-variance and zero-variance baselines for extensive-form games. arXiv preprint arXiv:1907.09633, 2019.","title":"Low-variance and zero-variance baselines for extensive-form games","authors":"Trevor Davis, Martin Schmid, Michael Bowling","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Witt et al., 2019] Christian Schroeder de Witt, Jakob Fo- erster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. In Advances in Neural Information Processing Systems, pages 9927-9939, 2019.","title":"Multi-agent common knowledge reinforcement learning","authors":"De Witt","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fagin et al., 2003] Ronald Fagin, Yoram Moses, Joseph Y Halpern, and Moshe Y Vardi. Reasoning about knowledge. MIT press, 2003.","title":"Reasoning about knowledge","authors":"Ronald Fagin, Yoram Moses, Joseph Y Halpern, Moshe Y Vardi","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Foerster et al., 2019] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian ac- tion decoder for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 1942-1951. PMLR, 2019.","title":"Bayesian action decoder for deep multi-agent reinforcement learning","authors":"Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, Michael Bowling","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Horák and Bošanský, 2019] Karel Horák and Branislav Bošanský. Solving partially observable stochastic games with public observations. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 2029- 2036, 2019.","title":"Solving partially observable stochastic games with public observations","authors":"Bošanský Horák, Karel Horák, Branislav Bošanský","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johanson et al., 2011] Michael Johanson, Kevin Waugh, Michael Bowling, and Martin Zinkevich. Accelerating best response calculation in large extensive games. In IJCAI, volume 11, pages 258-265, 2011.","title":"Accelerating best response calculation in large extensive games","authors":"Michael Johanson, Kevin Waugh, Michael Bowling, Martin Zinkevich","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/54"},{"position":14,"raw":"[Kovařík et al., 2022] Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, and Viliam Lisỳ. Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence, 303:103645, 2022.","title":"Rethinking formal models of partially observable multiagent decision making","authors":"Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, Viliam Lisỳ","year":"2022","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lerer et al., 2020] Adam Lerer, Hengyuan Hu, Jakob N Fo- erster, and Noam Brown. Improving policies via search in cooperative partially observable games. In AAAI, pages 7187-7194, 2020.","title":"Improving policies via search in cooperative partially observable games","authors":"Adam Lerer, Hengyuan Hu, Jakob N Foerster, Noam Brown","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Hui Li, Kailiang Hu, Shaohua Zhang, Lin Wang, Jun Zhou, Yuan Qi, and Le Song. Regret minimiza- tion via novel vectorized sampling policies and exploration. preprint, 2020. Accessed: August 5th, 2020.","title":"Regret minimization via novel vectorized sampling policies and exploration","authors":"Hui Li, Kailiang Hu, Shaohua Zhang, Lin Wang, Jun Zhou, Yuan Qi, Le Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moravčík et al., 2017] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelli- gence in heads-up no-limit poker. Science, 356(6337):508- 513, 2017.","title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schrittwieser et al., 2019] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, Go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019.","title":"Mastering atari, Go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Seitz et al., 2019] Dominik Seitz, Vojtěch Kovařík, Viliam Lisý, Jan Rudolf, Shuo Sun, and Karel Ha. Value functions for depth-limited solving in imperfect-information games beyond poker. arXiv preprint arXiv:1906.06412, 2019.","title":"Value functions for depth-limited solving in imperfect-information games beyond poker","authors":"Dominik Seitz, Vojtěch Kovařík, Viliam Lisý, Jan Rudolf, Shuo Sun, Karel Ha","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of Go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Spaan et al., 2008] Matthijs TJ Spaan, Frans A Oliehoek, and Nikos Vlassis. Multiagent planning under uncertainty with stochastic communication delays. In 338 Proceedings of the Eighteenth International Conference on Automated Planning and Scheduling (ICAPS 2008), pages 338-345, 2008.","title":"Multiagent planning under uncertainty with stochastic communication delays","authors":"Frans A Matthijs Tj Spaan, Nikos Oliehoek, Vlassis","year":"2008","venue":"338 Proceedings of the Eighteenth International Conference on Automated Planning and Scheduling (ICAPS 2008)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Šustr et al., 2019] Michal Šustr, Vojtěch Kovařík, and Viliam Lisý. Monte Carlo continual resolving for online strategy computation in imperfect information games. In Proceed- ings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 224-232. Interna- tional Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"Monte Carlo continual resolving for online strategy computation in imperfect information games","authors":"Šustr","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[von Neumann and Morgenstern, 1953] John von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton university press, 1953.","title":"John von Neumann and Oskar Morgenstern","authors":"Von Neumann, Morgenstern","year":"1953","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}