{"type":"ScholarlyArticle","title":"BRExIt: On Opponent Modelling in Expert Iteration","authors":[{"name":"Daniel Hernandez","slug":"daniel-hernandez","given":"Daniel","family":"Hernandez","affiliation":"University of York , UK"},{"name":"Hendrik Baier","slug":"hendrik-baier","given":"Hendrik","family":"Baier","affiliation":"Eindhoven University of Technology , The Netherlands"},{"name":"Michael Kaisers","slug":"michael-kaisers","given":"Michael","family":"Kaisers","affiliation":"Centrum Wiskunde & Informatica , The Netherlands"}],"abstract":"Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best-response oracles to improve play against candidate opponents (typically previously learnt policies). We propose Best Response Expert Iteration (BRExIt), which accelerates learning in games by incorporating opponent models into the state-of-the-art learning algorithm Expert Iteration (ExIt). BRExIt aims to (1) improve feature shaping in the apprentice, with a policy head predicting opponent policies as an auxiliary task, and (2) bias opponent moves in planning towards the given or learnt opponent model, to generate apprentice targets that better approximate a best response. In an empirical ablation on BRExIt's algorithmic variants against a set of fixed test agents, we provide statistical evidence that BRExIt learns better performing policies than ExIt. Code available at: https://github.com/Danielhp95/on-opponent-modelling-in-expert-iteration-code. Supplementary material available\r\nat https://arxiv.org/abs/2206.00113.","doi":"10.24963/ijcai.2023/422","url":"https://ijcai.org/proceedings/2023/422","pdf":"https://ijcai.org/proceedings/2023/0422.pdf","markdown":"https://ijcai.org/proceedings/2023/422.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/422","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":422,"pages":{"first":3795,"last":3802},"keywords":["Machine Learning: ML: Reinforcement learning","Search: S: Game playing","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2021] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, et al. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.","title":"Deep reinforcement learning at the edge of the statistical precipice","authors":"Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Albrecht and Stone, 2018] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial In- telligence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"V Stefano, Peter Albrecht, Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anthony et al., 2017] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In NIPS, 2017.","title":"Thinking fast and slow with deep learning and tree search","authors":"Thomas Anthony, Zheng Tian, David Barber","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balduzzi et al., 2019] David Balduzzi, Marta Garnelo, Yoram Bachrach, et al. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443. PMLR, 2019.","title":"Open-ended learning in symmetric zero-sum games","authors":"David Balduzzi, Marta Garnelo, Yoram Bachrach","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, Brooke Chan, et al. Dota 2 with large scale deep reinforce- ment learning. arXiv preprint arXiv:1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman, Brooke Chan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown, 1951] George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allo- cation, 13(1):374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"Activity analysis of production and allocation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Browne et al., 2012] Cameron B Browne, Edward Powley, Daniel Whitehouse, et al. A survey of Monte Carlo Tree Search methods. IEEE Transactions on Computational In- telligence and AI in Games, 4(1):1-43, 2012.","title":"A survey of Monte Carlo Tree Search methods","authors":"Edward Cameron B Browne, Daniel Powley, Whitehouse","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, et al. On the utility of learning about humans for human-ai coordination. In Advances in Neural Informa- tion Processing Systems, pages 5175-5186, 2019.","title":"On the utility of learning about humans for human-ai coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Czarnecki et al., 2020] Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, et al. Real world games look like spinning tops. Advances in Neural Information Processing Systems, 33:17443-17454, 2020.","title":"Real world games look like spinning tops","authors":"Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Espeholt et al., 2018] Lasse Espeholt, Hubert Soyer, Remi Munos, et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. CoRR, abs/1802.01561, 2018.","title":"IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures","authors":"Espeholt","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodman and Lucas, 2020] James Goodman and Simon Lucas. Does it matter how well I know what you're think- ing? opponent modelling in an RTS game. In IEEE Congress on Evolutionary Computation, CEC 2020, pages 1-8. IEEE, 2020.","title":"Does it matter how well I know what you're thinking? opponent modelling in an RTS game","authors":"Lucas ; Goodman, James Goodman, Simon Lucas","year":"2020","venue":"IEEE Congress on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforce- ment learning. In International Conference on Machine Learning, pages 1804-1813, 2016.","title":"Opponent modeling in deep reinforcement learning","authors":"He","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hernandez et al., 2019] Daniel Hernandez, Kevin Denam- ganaï, Yuan Gao, et al. A generalized framework for self- play training. In 2019 IEEE Conference on Games (CoG), pages 1-8. IEEE, 2019.","title":"A generalized framework for selfplay training","authors":"Daniel Hernandez, Kevin Denamganaï, Yuan Gao","year":"2019","venue":"IEEE Conference on Games (CoG)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hernandez-Leal et al., 2017] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arxiv 2017. arXiv preprint arXiv:1707.09183, 2017.","title":"A survey of learning in multiagent environments: Dealing with non-stationarity","authors":"Hernandez-Leal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Agent modeling as auxil- iary task for deep reinforcement learning. In Gillian Smith and Levi Lelis, editors, Proceedings of the Fifteenth AAAI Conference on Artificial Intelligence and Interactive Digi- tal Entertainment, AIIDE 2019, pages 31-37. AAAI Press, 2019.","title":"Agent modeling as auxiliary task for deep reinforcement learning","authors":"Hernandez-Leal","year":"2019","venue":"Proceedings of the Fifteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2019","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hong et al., 2018] Zhang-Wei Hong, Shih-Yang Su, Tzu- Yun Shann, et al. A deep policy inference q-network for multi-agent systems. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 1388-1396. International Foundation for Autonomous Agents and Multiagent Systems, 2018.","title":"A deep policy inference q-network for multi-agent systems","authors":"Zhang-Wei Hong, Shih-Yang Su, Tzu-Yun Shann","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, et al. A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems, 30:4190-4203, 2017.","title":"A unified game-theoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Xiangyu Liu, Hangtian Jia, Ying Wen, et al. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34, 2021.","title":"Towards unifying behavioral and response diversity for open-ended learning in zero-sum games","authors":"Xiangyu Liu, Hangtian Jia, Ying Wen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Playing Atari with Deep Reinforce- ment Learning. CoRR, abs/1312.5602, 2013.","title":"Playing Atari with Deep Reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nashed and Zilberstein, 2022] Samer Nashed and Shlomo Zilberstein. A survey of opponent modeling in adversar- ial domains. Journal of Artificial Intelligence Research, 73:277-327, 2022.","title":"A survey of opponent modeling in adversarial domains","authors":"Zilberstein Nashed, Samer Nashed, Shlomo Zilberstein","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oliehoek and Amato, 2014] Frans A Oliehoek and Christo- pher Amato. Best response bayesian reinforcement learn- ing for multiagent systems with state uncertainty. In Proceedings of the Ninth AAMAS Workshop on Multi- Agent Sequential Decision Making in Uncertain Domains (MSDM), 2014.","title":"Best response bayesian reinforcement learning for multiagent systems with state uncertainty","authors":"Amato ; Oliehoek, A Frans, Christopher Oliehoek, Amato","year":"2014","venue":"Proceedings of the Ninth AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains (MSDM)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Re- play. CoRR, abs/1511.05952, 2015.","title":"Prioritized Experience Replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, et al. Proximal policy optimization algo- rithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Mad- dison, et al. Mastering the game of Go with deep neu- ral networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, et al. Mastering Chess and Shogi by Self- Play with a General Reinforcement Learning Algorithm. Science, 362:1140-1144, 2018.","title":"Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Soemers et al., 2020] Dennis JNJ Soemers, Éric Piette, Matthew Stephenson, and Cameron Browne. Manipulat- ing the distributions of experience used for self-play learn- ing in expert iteration. arXiv preprint arXiv:2006.00283, 2020.","title":"Manipulating the distributions of experience used for self-play learning in expert iteration","authors":"Jnj Dennis, Éric Soemers, Matthew Piette, Cameron Stephenson, Browne","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Timbers et al., 2022] Finbarr Timbers, Nolan Bard, Edward Lockhart, et al. Approximate exploitability: Learning a best response. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 3487- 3493, 2022.","title":"Approximate exploitability: Learning a best response","authors":"Finbarr Timbers, Nolan Bard, Edward Lockhart","year":"2022","venue":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/484"},{"position":28,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Wo- jciech M Czarnecki, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Willemsen et al., 2021] Daniel Willemsen, Hendrik Baier, and Michael Kaisers. Value targets in off-policy alphazero: a new greedy backup. Neural Computing and Applica- tions, pages 1-14, 2021.","title":"Value targets in off-policy alphazero: a new greedy backup","authors":"Daniel Willemsen, Hendrik Baier, Michael Kaisers","year":"2021","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu, 2019] David J Wu. Accelerating self-play learning in go. arXiv preprint arXiv:1902.10565, 2019.","title":"Accelerating self-play learning in go","authors":"J David, Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}