{"type":"ScholarlyArticle","title":"Approximate Exploitability: Learning a Best Response","authors":[{"name":"Finbarr Timbers","slug":"finbarr-timbers","given":"Finbarr","family":"TImbers","affiliation":"University of Alberta"},{"name":"Nolan Bard","slug":"nolan-bard","given":"Nolan","family":"Bard","affiliation":"University of Alberta"},{"name":"Edward Lockhart","slug":"edward-lockhart","given":"Edward","family":"Lockhart","affiliation":"University of Alberta"},{"name":"Marc Lanctot","slug":"marc-lanctot","given":"Marc","family":"Lanctot","affiliation":"University of Alberta"},{"name":"Martin Schmid","slug":"martin-schmid","given":"Martin","family":"Schmid","affiliation":"University of Alberta"},{"name":"Neil Burch","slug":"neil-burch","given":"Neil","family":"Burch","affiliation":"University of Alberta"},{"name":"Julian Schrittwieser","slug":"julian-schrittwieser","given":"Julian","family":"Schrittwieser","affiliation":"University of Alberta"},{"name":"Thomas Hubert","slug":"thomas-hubert","given":"Thomas","family":"Hubert","affiliation":"University of Alberta"},{"name":"Michael Bowling","slug":"michael-bowling","given":"Michael","family":"Bowling","affiliation":"University of Alberta"}],"abstract":"Researchers have shown that neural networks are vulnerable to adversarial examples and subtle environment changes.  The resulting errors can look like blunders to humans, eroding trust in these agents. In prior games research, agent evaluation often focused on the in-practice game outcomes. Such evaluation typically fails to evaluate robustness to worst-case outcomes. Computer poker research has examined how to assess such worst-case performance. Unfortunately, exact computation is infeasible with larger domains, and existing approximations are poker-specific. We introduce ISMCTS-BR, a scalable search-based deep reinforcement learning algorithm for learning a best response to an agent,  approximating worst-case performance. We demonstrate the technique in several games against a variety of agents, including several AlphaZero-based agents. Supplementary material is available at https://arxiv.org/abs/2004.09677.","doi":"10.24963/ijcai.2022/484","url":"https://ijcai.org/proceedings/2022/484","pdf":"https://ijcai.org/proceedings/2022/0484.pdf","markdown":"https://ijcai.org/proceedings/2022/484.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/484","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":484,"pages":{"first":3487,"last":3493},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Noncooperative Games"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986636","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984779","length_sec":354,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Allis, 1994] V. L. Allis. Searching for solutions in games and artificial intelligence. PhD thesis, University of Lim- burg, 1994.","title":"Searching for solutions in games and artificial intelligence","authors":"V L Allis","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amodei et al., 2016] D. Amodei, C. Olah, J. Steinhardt, P.F. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016.","title":"Concrete problems in AI safety","authors":"D Amodei, C Olah, J Steinhardt, P F Christiano, J Schulman, D Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Athalye et al., 2018] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In Proceedings of the 35th International Conference on Ma- chine Learning (ICML), pages 284-293. PMLR, 2018.","title":"Synthesizing robust adversarial examples","authors":"A Athalye, L Engstrom, A Ilyas, K Kwok","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balduzzi et al., 2018] D. Balduzzi, K. Tuyls, J. Pérolat, and T. Graepel. Re-evaluating evaluation. CoRR, abs/1806.02643, 2018.","title":"Re-evaluating evaluation","authors":"D Balduzzi, K Tuyls, J Pérolat, T Graepel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bard, 2016] N. DC Bard. Online Agent Modelling in Human-Scale Problems. PhD thesis, U. Alberta, 2016.","title":"Online Agent Modelling in Human-Scale Problems","authors":"N Bard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellemare et al., 2013] M. G. Bellemare, Y. Naddaf, J. Ve- ness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Ar- tificial Intelligence Research, 47:253-279, jun 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"M G Bellemare, Y Naddaf, J Veness, M Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown and Sandholm, 2017] N. Brown and T. Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 360(6385), 2017.","title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","authors":"N Brown, T Sandholm","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown et al., 2018] N. Brown, T. Sandholm, and B. Amos. Depth-limited solving for imperfect-information games. Advances in NeurIPS, pages 7663-7674, 2018.","title":"Depth-limited solving for imperfect-information games","authors":"N Brown, T Sandholm, B Amos","year":"2018","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Campbell et al., 2002] M. Campbell, A. J. Hoane Jr, and F. Hsu. Deep blue. Artif. intell., 134(1-2):57-83, 2002.","title":"Deep blue","authors":"M Campbell, A J Hoane, F Hsu","year":"2002","venue":"Artif. intell","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cowling et al., 2010] P.I. Cowling, E.J. Powley, and D. Whitehouse. Information set Monte Carlo tree search. IEEE Transactions on Computational Intelligence and AI in Games, 4(2):120-143, 2010.","title":"Information set Monte Carlo tree search","authors":"P I Cowling, E J Powley, D Whitehouse","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gleave et al., 2020] A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell. Adversarial policies: Attacking deep reinforcement learning. In 8th Int'l Con- ference on Learning Representations, ICLR, 2020.","title":"Adversarial policies: Attacking deep reinforcement learning","authors":"A Gleave, M Dennis, C Wild, N Kant, S Levine, S Russell","year":"2020","venue":"8th Int'l Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Greenwald et al., 2017] A. Greenwald, J. Li, and E. Sodomka. Solving for best responses and equi- libria in extensive-form games with reinforcement learning methods. In Rohit Parikh on Logic, Language and Society, pages 185-226. Springer, 2017.","title":"Solving for best responses and equilibria in extensive-form games with reinforcement learning methods","authors":"A Greenwald, J Li, E Sodomka","year":"2017","venue":"Rohit Parikh on Logic","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gruslys et al., 2020] A. Gruslys, M. Lanctot, R. Munos, F. Timbers, M. Schmid, and et al. The advantage regret- matching actor-critic. ArXiv, 2020.","title":"The advantage regretmatching actor-critic","authors":"A Gruslys, M Lanctot, R Munos, F Timbers, M Schmid","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jackson, 2013] E.G. Jackson. Slumbot NL: Solving large games with counterfactual regret minimization using sam- pling and distributed processing. In Workshops of the Twenty-Seventh AAAI Conference, 2013.","title":"Slumbot NL: Solving large games with counterfactual regret minimization using sampling and distributed processing","authors":"E G Jackson","year":"2013","venue":"Workshops of the Twenty-Seventh AAAI Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Johanson et al., 2011] M. Johanson, K. Waugh, M. Bowl- ing, and Martin M. Zinkevich. Accelerating best response calculation in large extensive games. In IJCAI, 2011.","title":"Accelerating best response calculation in large extensive games","authors":"M Johanson, K Waugh, M Bowling, Martin M Zinkevich","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/54"},{"position":16,"raw":"[Johanson, 2011] M. Johanson. Accelerating best response calculation in large extensive games. http://johanson.ca/publications/poker/2011-ijcai- abr/2011-ijcai-abr.html, 2011.","title":"Accelerating best response calculation in large extensive games","authors":"M Johanson","year":"2011","venue":null,"doi":null,"url":"http://johanson.ca/publications/poker/2011-ijcai-abr/2011-ijcai-abr.html","internal_url":"/proceedings/2011/54"},{"position":17,"raw":"[Johanson, 2013] M. Johanson. Measuring the size of large no-limit poker games. arXiv abs/1302.7008, 2013.","title":"Measuring the size of large no-limit poker games","authors":"M Johanson","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lanctot et al., 2019] Marc Lanctot, Edward Lockhart, Jean- Baptiste Lespiau, Vinicius Zambaldi, and et al. OpenSpiel: A framework for reinforcement learning in games. CoRR, abs/1908.09453, 2019. http://arxiv.org/abs/1908.09453.","title":"OpenSpiel: A framework for reinforcement learning in games","authors":"Lanctot","year":"2019","venue":null,"doi":null,"url":"http://arxiv.org/abs/1908.09453","internal_url":null},{"position":19,"raw":"[Lisý and Bowling, 2016] Viliam Lisý and Michael H. Bowling. Eqilibrium approximation quality of current no- limit poker bots. CoRR, abs/1612.07547, 2016.","title":"Eqilibrium approximation quality of current nolimit poker bots","authors":"Viliam Lisý, Michael H Bowling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lisý et al., 2015] V. Lisý, M. Lanctot, and M. Bowling. On- line Monte Carlo counterfactual regret minimization for search in imperfect information games. In Proceedings of the Int'l Conf. on Autonomous Agents and Multi-Agent Systems (AAMAS), pages 27-36, 2015.","title":"Online Monte Carlo counterfactual regret minimization for search in imperfect information games","authors":"V Lisý, M Lanctot, M Bowling","year":"2015","venue":"Proceedings of the Int'l Conf. on Autonomous Agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, and et al. Human-level control through deep re- inforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Rusu","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moravčík et al., 2017] M. Moravčík, M. Schmid, N. Burch, and et al. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 358(6362), 2017.","title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"M Moravčík, M Schmid, N Burch","year":null,"venue":"Science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Omidshafiei et al., 2019] S. Omidshafiei, C. Papadimitriou, Piliouras G, K. Tuyls, and et al. α-rank: Multi-agent eval- uation by evolution. Scientific Reports, 9(1):9937, 2019.","title":"α-rank: Multi-agent evaluation by evolution","authors":"S Omidshafiei, C Papadimitriou, G Piliouras, K Tuyls","year":"2019","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ozair et al., 2021] S. Ozair, Y. Li, A. Razavi, I. Antonoglou, A. van den Oord, and O. Vinyals. Vector quantized models for planning. arXiv preprint arXiv:2106.04615, 2021.","title":"Vector quantized models for planning","authors":"S Ozair, Y Li, A Razavi, I Antonoglou, A Van Den Oord, O Vinyals","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Quionero-Candela et al., 2009] J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N.D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.","title":"Dataset Shift in Machine Learning","authors":"Quionero-Candela","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schaeffer et al., 1996] J. Schaeffer, R. Lake, P. Lu, and M. Bryant. Chinook the world man-machine checkers champion. AI Magazine, 17(1):21-21, 1996.","title":"Chinook the world man-machine checkers champion","authors":"J Schaeffer, R Lake, P Lu, M Bryant","year":"1996","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, and Ju- lian Schrittwieser et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self- play. Science, 632(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Srinivasan et al., 2018] S. Srinivasan, M. Lanctot, V. Zam- baldi, and et al. Actor-critic policy optimization in par- tially observable multiagent environments. In Advances in NeurIPS, pages 3422-3435, 2018.","title":"Actor-critic policy optimization in partially observable multiagent environments","authors":"S Srinivasan, M Lanctot, V Zambaldi","year":"2018","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tammelin et al., 2015] O. Tammelin, N. Burch, M. Johan- son, and M. Bowling. Solving heads-up limit texas hold'em. In IJCAI, 2015.","title":"Solving heads-up limit texas hold","authors":"O Tammelin, N Burch, M Johanson, M Bowling","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tuyls et al., 2018] K. Tuyls, J. Perolat, M. Lanctot, J.Z. Leibo, and T. Graepel. A generalized method for empirical game theoretic analysis. In Proceedings of the Int'l Conf. on Auto. Agents and Multiagent Systems (AAMAS), 2018.","title":"A generalized method for empirical game theoretic analysis","authors":"K Tuyls, J Perolat, M Lanctot, J Z Leibo, T Graepel","year":"2018","venue":"Proceedings of the Int'l Conf. on Auto. Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Witty et al., 2018] S. Witty, J.K. Lee, E. Tosch, A. Atrey, M.L. Littman, and D.D. Jensen. Measuring and char- acterizing generalization in deep reinforcement learning. CoRR, abs/1812.02868, 2018.","title":"Measuring and characterizing generalization in deep reinforcement learning","authors":"S Witty, J K Lee, E Tosch, A Atrey, M L Littman, D D Jensen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zinkevich et al., 2008] M. Zinkevich, M. Johanson, M. Bowling, and C. Piccione. Regret minimization in games with incomplete information. In Adv. in Neural Information Processing Systems 20, pages 905-912, 2008.","title":"Regret minimization in games with incomplete information","authors":"M Zinkevich, M Johanson, M Bowling, C Piccione","year":"2008","venue":"Adv. in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}