{"type":"ScholarlyArticle","title":"Minimizing Weighted Counterfactual Regret with Optimistic Online Mirror Descent","authors":[{"name":"Hang Xu","slug":"hang-xu","given":"Hang","family":"Xu","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Kai Li","slug":"kai-li","given":"Kai","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Bingyun Liu","slug":"bingyun-liu","given":"Bingyun","family":"Liu","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Haobo Fu","slug":"haobo-fu","given":"Haobo","family":"Fu","affiliation":"Tencent AI Lab"},{"name":"Qiang Fu","slug":"qiang-fu","given":"Qiang","family":"Fu","affiliation":"Tencent AI Lab"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"Tsinghua University"},{"name":"Jian Cheng","slug":"jian-cheng","given":"Jian","family":"Cheng","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Future Technology , University of Chinese Academy of Sciences; AiRiA"}],"abstract":"Counterfactual regret minimization (CFR) is a family of algorithms for effectively solving imperfect-information games. It decomposes the total regret into counterfactual regrets, utilizing local regret minimization algorithms, such as Regret Matching (RM) or RM+, to minimize them. Recent research establishes a connection between Online Mirror Descent (OMD) and RM+, paving the way for an optimistic variant PRM+ and its extension PCFR+. However, PCFR+ assigns uniform weights for each iteration when determining regrets, leading to substantial regrets when facing dominated actions. This work explores minimizing weighted counterfactual regret with optimistic OMD, resulting in a novel CFR variant PDCFR+. It integrates PCFR+ and Discounted CFR (DCFR) in a principled manner, swiftly mitigating negative effects of dominated actions and consistently leveraging predictions to accelerate convergence. Theoretical analyses prove that PDCFR+ converges to a Nash equilibrium, particularly under distinct weighting schemes for regrets and average strategies. Experimental results demonstrate PDCFR+'s fast convergence in common imperfect-information games. The code is available at https://github.com/rpSebastian/PDCFRPlus.","doi":"10.24963/ijcai.2024/583","url":"https://ijcai.org/proceedings/2024/583","pdf":"https://ijcai.org/proceedings/2024/0583.pdf","markdown":"https://ijcai.org/proceedings/2024/583.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/583","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":583,"pages":{"first":5272,"last":5280},"keywords":["Machine Learning: ML: Game Theory","Game Theory and Economic Paradigms: GTEP: Noncooperative games"],"videos":[],"references":[{"position":1,"raw":"[Abernethy et al., 2018] Jacob Abernethy, Kevin A Lai, Kfir Y Levy, and Jun-Kun Wang. Faster rates for convex- concave games. In Conference On Learning Theory, pages 1595-1625, 2018.","title":"Faster rates for convexconcave games","authors":"Jacob Abernethy, Kevin A Lai, Kfir Y Levy, Jun-Kun Wang","year":"2018","venue":"Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck and Teboulle, 2003] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient meth- ods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.","title":"Mirror descent and nonlinear projected subgradient methods for convex optimization","authors":"Amir Beck, Marc Teboulle","year":"2003","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bowling et al., 2015] Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science, 347(6218):145-149, 2015.","title":"Heads-up limit hold'em poker is solved","authors":"Michael Bowling, Neil Burch, Michael Johanson, Oskari Tammelin","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown and Sandholm, 2017] Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfect-information games. In Advances in Neural In- formation Processing Systems, pages 689-699, 2017.","title":"Safe and nested subgame solving for imperfect-information games","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown and Sandholm, 2018] Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418- 424, 2018.","title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown and Sandholm, 2019a] Noam Brown and Tuomas Sandholm. Solving imperfect-information games via dis- counted regret minimization. In AAAI Conference on Ar- tificial Intelligence, pages 1829-1836, 2019.","title":"Solving imperfect-information games via discounted regret minimization","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2019","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown and Sandholm, 2019b] Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Sci- ence, 365(6456):885-890, 2019.","title":"Superhuman AI for multiplayer poker","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown et al., 2018] Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfect- information games. In Advances in Neural Information Processing Systems, pages 7674-7685, 2018.","title":"Depth-limited solving for imperfectinformation games","authors":"Brown","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brown et al., 2019] Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International Conference on Machine Learn- ing, pages 793-802, 2019.","title":"Deep counterfactual regret minimization","authors":"Brown","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Burch et al., 2014] Neil Burch, Michael Johanson, and Michael Bowling. Solving imperfect information games using decomposition. In AAAI Conference on Artificial In- telligence, pages 602-608, 2014.","title":"Solving imperfect information games using decomposition","authors":"Neil Burch, Michael Johanson, Michael Bowling","year":"2014","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Farina et al., 2019a] Gabriele Farina, Christian Kroer, Noam Brown, and Tuomas Sandholm. Stable-predictive optimistic counterfactual regret minimization. In In- ternational Conference on Machine Learning, pages 1853-1862, 2019.","title":"Stable-predictive optimistic counterfactual regret minimization","authors":"Farina","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Farina et al., 2019b] Gabriele Farina, Chun Kai Ling, Fei Fang, and Tuomas Sandholm. Correlation in extensive- form games: Saddle-point formulation and benchmarks. In Advances in Neural Information Processing Systems, pages 9229-9239, 2019.","title":"Correlation in extensiveform games: Saddle-point formulation and benchmarks","authors":"Farina","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Farina et al., 2021] Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In AAAI Conference on Artificial In- telligence, pages 5363-5371, 2021.","title":"Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent","authors":"Gabriele Farina, Christian Kroer, Tuomas Sandholm","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Farina et al., 2023] Gabriele Farina, Julien Grand-Clément, Christian Kroer, Chung-Wei Lee, and Haipeng Luo. Re- gret matching+:(in) stability and fast convergence in games. In Advances in Neural Information Processing Sys- tems, 2023.","title":"Regret matching+:(in) stability and fast convergence in games","authors":"Gabriele Farina, Julien Grand-Clément, Christian Kroer, Chung-Wei Lee, Haipeng Luo","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Grand-Clément and Kroer, 2023] Julien Grand-Clément and Christian Kroer. Solving optimization problems with blackwell approachability. Mathematics of Operations Research, 2023.","title":"Solving optimization problems with blackwell approachability","authors":"Julien Grand, -Clément, Christian Kroer","year":"2023","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gratch et al., 2016] Jonathan Gratch, Zahra Nazari, and Emmanuel Johnson. The misrepresentation game: How to win at negotiation while seeming like a nice guy. In In- ternational Conference on Autonomous Agents and Multi- agent Systems, pages 728-737, 2016.","title":"The misrepresentation game: How to win at negotiation while seeming like a nice guy","authors":"Jonathan Gratch, Zahra Nazari, Emmanuel Johnson","year":"2016","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hang et al., 2022] Xu Hang, Li Kai, Fu Haobo, Fu Qiang, and Xing Junliang. Autocfr: Learning to design counter- factual regret minimization algorithms. In AAAI Confer- ence on Artificial Intelligence, pages 5244-5251, 2022.","title":"Autocfr: Learning to design counterfactual regret minimization algorithms","authors":"Xu Hang, Li Kai, Fu Haobo, Fu Qiang, Xing Junliang","year":"2022","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hang et al., 2024] Xu Hang, Li Kai, Fu Haobo, Fu Qiang, Xing Junliang, and Jian Cheng. Dynamic discounted coun- terfactual regret minimization. In International Confer- ence on Learning Representations, pages 1-18, 2024.","title":"Dynamic discounted counterfactual regret minimization","authors":"Xu Hang, Li Kai, Fu Haobo, Fu Qiang, Xing Junliang, Jian Cheng","year":"2024","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hart and Mas-Colell, 2000] Sergiu Hart and Andreu Mas- Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68(5):1127-1150, 2000.","title":"A simple adaptive procedure leading to correlated equilibrium","authors":"Sergiu Hart, Andreu Mas-Colell","year":"2000","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kuhn, 1950] Harold W Kuhn. A simplified two-person poker. Contributions to the Theory of Games, 1:97-103, 1950.","title":"A simplified two-person poker","authors":"Harold W Kuhn","year":"1950","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lisỳ et al., 2015] Viliam Lisỳ, Marc Lanctot, and Michael H Bowling. Online monte carlo counterfactual regret minimization for search in imperfect information games. In International Conference on Autonomous Agents and Multiagent Systems, pages 27-36, 2015.","title":"Online monte carlo counterfactual regret minimization for search in imperfect information games","authors":"Viliam Lisỳ, Marc Lanctot, Michael H Bowling","year":"2015","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lisy et al., 2016] Viliam Lisy, Trevor Davis, and Michael Bowling. Counterfactual regret minimization in sequential security games. In AAAI Conference on Artificial Intelli- gence, pages 544-550, 2016.","title":"Counterfactual regret minimization in sequential security games","authors":"Viliam Lisy, Trevor Davis, Michael Bowling","year":"2016","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2022] Weiming Liu, Huacong Jiang, Bin Li, and Houqiang Li. Equivalence analysis between counterfac- tual regret minimization and online mirror descent. In International Conference on Machine Learning, pages 13717-13745, 2022.","title":"Equivalence analysis between counterfactual regret minimization and online mirror descent","authors":"Weiming Liu, Huacong Jiang, Bin Li, Houqiang Li","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Moravčík et al., 2017] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508-513, 2017.","title":"DeepStack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Perolat et al., 2021] Julien Perolat, Remi Munos, Jean- Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Bal- duzzi, Bart De Vylder, et al. From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Confer- ence on Machine Learning, pages 8525-8535, 2021.","title":"From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization","authors":"Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ross, 1971] Sheldon M Ross. Goofspiel-the game of pure strategy. Journal of Applied Probability, 8(3):621-625, 1971.","title":"Goofspiel-the game of pure strategy","authors":"M Sheldon, Ross","year":"1971","venue":"Journal of Applied Probability","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sandholm, 2015] Tuomas Sandholm. Steering evolution strategically: Computational game theory and opponent exploitation for treatment planning, drug design, and syn- thetic biology. In AAAI Conference on Artificial Intelli- gence, pages 4057-4061, 2015.","title":"Steering evolution strategically: Computational game theory and opponent exploitation for treatment planning, drug design, and synthetic biology","authors":"Tuomas Sandholm, Sandholm","year":"2015","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Southey et al., 2005] Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent mod- elling in poker. In Conference on Uncertainty in Artificial Intelligence, pages 550-558, 2005.","title":"Bayes' bluff: Opponent modelling in poker","authors":"Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, Chris Rayner","year":"2005","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Syrgkanis et al., 2015] Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems, pages 2989-2997, 2015.","title":"Fast convergence of regularized learning in games","authors":"Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, Robert E Schapire","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tammelin, 2014] Oskari Tammelin. Solving large imper- fect information games using CFR+. arXiv preprint arXiv:1407.5042, 2014.","title":"Oskari Tammelin. Solving large imperfect information games using CFR+","authors":"Tammelin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang and Abernethy, 2018] Jun-Kun Wang and Jacob D Abernethy. Acceleration through optimistic no-regret dy- namics. In Advances in Neural Information Processing Systems, pages 3828-3838, 2018.","title":"Acceleration through optimistic no-regret dynamics","authors":"Wang, Abernethy ; Jun-Kun Wang, Jacob D Abernethy","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zinkevich et al., 2007] Martin Zinkevich, Michael Johan- son, Michael Bowling, and Carmelo Piccione. Regret min- imization in games with incomplete information. In Ad- vances in Neural Information Processing Systems, pages 1729-1736, 2007.","title":"Regret minimization in games with incomplete information","authors":"Martin Zinkevich, Michael Johanson, Michael Bowling, Carmelo Piccione","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}