{"type":"ScholarlyArticle","title":"Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning","authors":[{"name":"Ying Wen","slug":"ying-wen","given":"Ying","family":"Wen","affiliation":"University College London"},{"name":"Yaodong Yang","slug":"yaodong-yang","given":"Yaodong","family":"Yang","affiliation":"University College London; Huawei Research & Development U.K."},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"University College London"}],"abstract":"Though limited in real-world decision making, most multi-agent reinforcement learning (MARL) models assume perfectly rational agents -- a property hardly met  due to individual's cognitive limitation and/or the tractability of the decision  problem. In this paper, we introduce generalized recursive reasoning (GR2) as a novel framework to model agents with different \\emph{hierarchical} levels of rationality; our framework enables agents to exhibit varying levels of ``thinking'' ability thereby allowing higher-level agents  to best respond to various less sophisticated learners.  We contribute both theoretically and empirically. On the theory side, we devise the hierarchical framework of GR2 through probabilistic graphical models and prove  the existence of a perfect Bayesian equilibrium. Within the GR2, we propose a practical actor-critic solver, and demonstrate its  convergent property to a stationary point in two-player games through Lyapunov analysis. On the empirical side, we validate our findings on a variety of MARL benchmarks. Precisely, we first illustrate the hierarchical thinking process  on the Keynes Beauty Contest, and then demonstrate significant improvements compared to state-of-the-art opponent modeling baselines on the  normal-form games and  the cooperative navigation benchmark.","doi":"10.24963/ijcai.2020/58","url":"https://ijcai.org/proceedings/2020/58","pdf":"https://ijcai.org/proceedings/2020/0058.pdf","markdown":"https://ijcai.org/proceedings/2020/58.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/58","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":58,"pages":{"first":414,"last":421},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agent Theories and Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24817/file_24817.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Albrecht and Stone, 2018] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelli- gence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"V Stefano, Peter Albrecht, Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blei et al., 2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, pages 859- 877, 2017.","title":"Variational inference: A review for statisticians","authors":"Alp David M Blei, Jon D Kucukelbir, Mcauliffe","year":"2017","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Camerer et al., 2004] Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A cognitive hierarchy model of games. The Quarterly Journal of Economics, 2004.","title":"A cognitive hierarchy model of games","authors":"Colin F Camerer, Teck-Hua Ho, Juin-Kuan Chong","year":"2004","venue":"The Quarterly Journal of Economics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Claus and Boutilier, 1998] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI, 1998:746-752, 1998.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"Claus, Boutilier, Caroline Claus, Craig Boutilier","year":"1998","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coricelli and Nagel, 2009] Giorgio Coricelli and Rosemarie Nagel. Neural correlates of depth of strategic reasoning in medial prefrontal cortex. Proceedings of the National Academy of Sciences, 106(23):9163-9168, 2009.","title":"Giorgio Coricelli and Rosemarie Nagel. Neural correlates of depth of strategic reasoning in medial prefrontal cortex","authors":"Nagel Coricelli","year":"2009","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devetag and Warglien, 2003] Giovanna Devetag and Mas- simo Warglien. Games and phone numbers: Do short-term memory bounds affect strategic behavior? Journal of Eco- nomic Psychology, 24(2):189-202, 2003.","title":"Giovanna Devetag and Massimo Warglien. Games and phone numbers: Do short-term memory bounds affect strategic behavior?","authors":"Warglien Devetag","year":"2003","venue":"Journal of Economic Psychology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Genewein et al., 2015] Tim Genewein, Felix Leibfried, Jordi Grau-Moya, and Daniel Alexander Braun. Bounded ratio- nality, abstraction, and hierarchical decision-making: An information-theoretic optimality principle. Frontiers in Robotics and AI, 2:27, 2015.","title":"Bounded rationality, abstraction, and hierarchical decision-making: An information-theoretic optimality principle","authors":"Tim Genewein, Felix Leibfried, Jordi Grau-Moya, Daniel Alexander Braun","year":"2015","venue":"Frontiers in Robotics and AI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gmytrasiewicz and Doshi, 2005] Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. JAIR, 24:49-79, 2005.","title":"A framework for sequential planning in multi-agent settings","authors":"J Piotr, Prashant Gmytrasiewicz, Doshi","year":"2005","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldman and others, 2012] Alvin I Goldman et al. Theory of mind. The Oxford handbook of philosophy of cognitive science, pages 402-424, 2012.","title":"Theory of mind. The Oxford handbook of philosophy of cognitive science","authors":"Alvin I Goldman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grau-Moya et al., 2018] Jordi Grau-Moya, Felix Leibfried, and Haitham Bou-Ammar. Balancing two-player stochastic games with soft q-learning. IJCAI, 2018.","title":"Balancing two-player stochastic games with soft q-learning","authors":"Grau-Moya","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/37"},{"position":11,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. NIPS, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.","title":"Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforce- ment learning. In ICML, 2016.","title":"Opponent modeling in deep reinforcement learning","authors":"He","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hunicke, 2005] Robin Hunicke. The case for dynamic dif- ficulty adjustment in games. In Proceedings of the 2005 ACM SIGCHI ACE. ACM, 2005.","title":"The case for dynamic difficulty adjustment in games","authors":"Robin Hunicke, Hunicke","year":"2005","venue":"Proceedings of the 2005 ACM SIGCHI ACE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Keynes, 1936] J. M. Keynes. The General Theory of Employ- ment, Interest and Money. Macmillan, 1936. 14th edition, 1973.","title":"The General Theory of Employment, Interest and Money","authors":"J M Keynes","year":"1936","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levin and Zhang, 2019] Dan Levin and Luyao Zhang. Bridging level-k to nash equilibrium. Available at SSRN 2934696, 2019.","title":"Bridging level-k to nash equilibrium","authors":"Levin, Zhang, Dan Levin, Luyao Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levine, 2018] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.","title":"Reinforcement learning and control as probabilistic inference: Tutorial and review","authors":"Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu and Wang, 2016] Qiang Liu and Dilin Wang. Stein vari- ational gradient descent: A general purpose bayesian infer- ence algorithm. In NIPS, pages 2378-2386, 2016.","title":"Stein variational gradient descent: A general purpose bayesian inference algorithm","authors":"Qiang Liu, Dilin Wang","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In NIPS, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Marquez, 2003] Horacio J Marquez. Nonlinear control sys- tems: analysis and design, volume 1. Wiley, 2003.","title":"Nonlinear control systems: analysis and design","authors":"Horacio J Marquez","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McKelvey and Palfrey, 1995] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for nor- mal form games. Games and economic behavior, 1995.","title":"Quantal response equilibria for normal form games","authors":"D Richard, Thomas R Mckelvey, Palfrey","year":"1995","venue":"Games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nash and others, 1950] John F Nash et al. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48-49, 1950.","title":"Equilibrium points in n-person games","authors":"F John, Nash","year":"1950","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rabinowitz et al., 2018] Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. ICML, 2018.","title":"Machine theory of mind","authors":"Frank Neil C Rabinowitz, Francis Perbet, Chiyuan Song, S M Zhang, Matthew Eslami, Botvinick","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Pro- ceedings of the national academy of sciences, 39(10):1095- 1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shoham et al., 2003] Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning: a critical survey. In Technical report, 2003.","title":"Multi-agent reinforcement learning: a critical survey","authors":"Yoav Shoham, Rob Powers, Trond Grenager","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simon, 1972] Herbert A Simon. Theories of bounded ratio- nality. Decision and organization, 1(1):161-176, 1972.","title":"Theories of bounded rationality","authors":"A Herbert, Simon","year":"1972","venue":"Decision and organization","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tesauro, 1995] Gerald Tesauro. Temporal difference learn- ing and td-gammon. Communications of the ACM, 38(3):58-68, 1995.","title":"Temporal difference learning and td-gammon","authors":"Gerald Tesauro","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tian et al., 2019] Zheng Tian, Ying Wen, Zhicheng Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. IJCAI, 2019.","title":"A regularized opponent model with maximum entropy objective","authors":"Zheng Tian, Ying Wen, Zhicheng Gong, Faiz Punakkath, Shihao Zou, Jun Wang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/85"},{"position":31,"raw":"[Wei et al., 2018] Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. AAAI, 2018.","title":"Multiagent soft q-learning","authors":"Ermo Wei, Drew Wicke, David Freelan, Sean Luke","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wen et al., 2019] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In ICLR, 2019.","title":"Probabilistic recursive reasoning for multi-agent reinforcement learning","authors":"Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, Wei Pan","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2018a] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In ICML, 2018.","title":"Mean field multi-agent reinforcement learning","authors":"Yang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2018b] Yaodong Yang, Lantao Yu, Yiwei Bai, Ying Wen, Weinan Zhang, and Jun Wang. A study of ai population dynamics with million-agent reinforcement learning. In 17th AAMAS, pages 2133-2135, 2018.","title":"A study of ai population dynamics with million-agent reinforcement learning","authors":"Yang","year":"2018","venue":"17th AAMAS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2019] Ming Zhou, Yong Chen, Ying Wen, Yaodong Yang, Yufeng Su, Weinan Zhang, Dell Zhang, and Jun Wang. Factorized q-learning for large-scale multi- agent systems. In Proceedings of the First International Conference on Distributed Artificial Intelligence, pages 1-7, 2019.","title":"Factorized q-learning for large-scale multiagent systems","authors":"Ming Zhou, Yong Chen, Ying Wen, Yaodong Yang, Yufeng Su, Weinan Zhang, Dell Zhang, Jun Wang","year":"2019","venue":"Proceedings of the First International Conference on Distributed Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}