{"type":"ScholarlyArticle","title":"Unveiling Concepts Learned by a World-Class Chess-Playing Agent","authors":[{"name":"Aðalsteinn Pálsson","slug":"a-alsteinn-palsson","given":"Aðalsteinn","family":"Pálsson","affiliation":"Department of Computer Science , Reykjavik University"},{"name":"Yngvi Björnsson","slug":"yngvi-bjornsson","given":"Yngvi","family":"Björnsson","affiliation":"Department of Computer Science , Reykjavik University"}],"abstract":"In recent years, the state-of-the-art agents for playing abstract board games, like chess and others, have moved from using intricate hand-crafted models for evaluating the merits of individual game states toward using neural networks (NNs). This development has eased the encapsulation of the relevant domain-specific knowledge and resulted in much-improved playing strength. However, this has come at the cost of making the resulting models ill-interpretable and challenging to understand and use for enhancing human knowledge. Using a world-class superhuman-strength chess-playing engine as our testbed, we show how recent model probing interpretability techniques can shed light on concepts learned by the engine's NN. Furthermore, to gain additional insight, we contrast the game-state evaluations of the NN to that of its counterpart hand-crafted evaluation model and identify and explain some of the main differences.","doi":"10.24963/ijcai.2023/541","url":"https://ijcai.org/proceedings/2023/541","pdf":"https://ijcai.org/proceedings/2023/0541.pdf","markdown":"https://ijcai.org/proceedings/2023/541.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/541","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":541,"pages":{"first":4864,"last":4872},"keywords":["Machine Learning: ML: Explainable/Interpretable machine learning","Search: S: Game playing","Multidisciplinary Topics and Applications: MDA: Game playing"],"videos":[],"references":[{"position":1,"raw":"[Alain and Bengio, 2017] Guillaume Alain and Yoshua Ben- gio. Understanding intermediate layers using linear clas- sifier probes. In 5th International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Workshop Track Proceedings. OpenReview.net, 2017.","title":"Understanding intermediate layers using linear classifier probes","authors":"Alain, Bengio, Guillaume Alain, Yoshua Bengio","year":"2017","venue":"5th International Conference on Learning Representations, ICLR 2017","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bodria et al., 2021] Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Salvatore Rinzivillo. Benchmarking and survey of explanation methods for black box models. CoRR, abs/2102.13076, 2021.","title":"Benchmarking and survey of explanation methods for black box models","authors":"Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, Salvatore Rinzivillo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castro et al., 2009] Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5):1726-1730, 2009. Selected papers presented at the Tenth International Symposium on Locational Decisions (ISOLDE X). [Decodea, 2022] Decodea. Decode chess. https:// decodechess.com/, 2022. Accessed: 2022-01-30.","title":"Polynomial calculation of the shapley value based on sampling","authors":"Javier Castro, Daniel Gómez, Juan Tejada","year":"2009","venue":"Decodea. Decode chess","doi":null,"url":"https://decodechess.com/,2022","internal_url":null},{"position":4,"raw":"[Guid et al., 2013] Matej Guid, Martin Mozina, Ciril Bohak, Aleksander Sadikov, and Ivan Bratko. Building an intelli- gent tutoring system for chess endgames. In Owen Foley, Maria Teresa Restivo, James Onohuome Uhomoibhi, and Markus Helfert, editors, CSEDU 2013 -Proceedings of the 5th International Conference on Computer Supported Education, Aachen, Germany, 6-8 May, 2013, pages 263- 266. SciTePress, 2013.","title":"Building an intelligent tutoring system for chess endgames","authors":"Matej Guid, Martin Mozina, Ciril Bohak, Aleksander Sadikov, Ivan Bratko","year":"2013","venue":"CSEDU 2013 -Proceedings of the 5th International Conference on Computer Supported Education","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[HaCohen-Kerner, 1994] Yaakov HaCohen-Kerner. Case- based evaluation in computer chess. In Jean Paul Haton, Mark T. Keane, and Michel Manago, editors, Advances in Case-Based Reasoning, Second European Workshop, EWCBR-94, Chantilly, France, November 7-10, 1994, Se- lected Papers, volume 984 of Lecture Notes in Computer Science, pages 240-254. Springer, 1994.","title":"Casebased evaluation in computer chess","authors":"Hacohen-Kerner ; Hacohen-Kerner","year":"1994","venue":"Advances in Case-Based Reasoning, Second European Workshop, EWCBR-94","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[HaCohen-Kerner, 1995] Yaakov HaCohen-Kerner. Learn- ing strategies for explanation patterns: Basic game pat- terns with application to chess. In Manuela M. Veloso and Agnar Aamodt, editors, Case-Based Reasoning Research and Development, First International Conference, ICCBR- 95, Sesimbra, Portugal, October 23-26, 1995, Proceed- ings, volume 1010 of Lecture Notes in Computer Science, pages 491-500. Springer, 1995.","title":"Learning strategies for explanation patterns: Basic game patterns with application to chess","authors":"Hacohen-Kerner ;, Yaakov Hacohen-Kerner","year":"1995","venue":"Case-Based Reasoning Research and Development, First International Conference, ICCBR-95","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim et al., 2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (TCAV). In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceed- ings of Machine Learning Research, pages 2673-2682.","title":"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)","authors":"Kim","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[LeelaChessZero, 2022] LeelaChessZero. Leela chess zero. https://lczero.org/, 2022. Accessed: 2022-01-30. Lipovetsky and Conklin, 2001] Stan Lipovetsky and Michael Conklin. Analysis of regression in game theory approach. Applied Stochastic Models in Business and Industry, 17(4):319-330, 2001.","title":"Analysis of regression in game theory approach","authors":"; Leelachesszero, Conklin Lipovetsky, Stan Lipovetsky, Michael Conklin","year":"2001","venue":"LeelaChessZero. Leela chess zero","doi":null,"url":"https://lczero.org/","internal_url":null},{"position":10,"raw":"[Lovering et al., 2022] Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. Evaluation beyond task performance: Analyzing concepts in alphazero in hex. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25992-26006.","title":"Evaluation beyond task performance: Analyzing concepts in alphazero in hex","authors":"Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, Michael Littman","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2022.","title":null,"authors":"Curran Associates, Inc","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, vol- ume 2017-Decem, pages 4766-4775, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su Lundberg, Lee In","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McGrath et al., 2021] Thomas McGrath, Andrei Kapish- nikov, Nenad Tomasev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Ac- quisition of chess knowledge in alphazero. CoRR, abs/2111.09259, 2021.","title":"Acquisition of chess knowledge in alphazero","authors":"Thomas Mcgrath, Andrei Kapishnikov, Nenad Tomasev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, Vladimir Kramnik","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McGrath et al., 2022] Thomas McGrath, Andrei Kapish- nikov, Nenad Tomašev, Adam Pearce, Martin Watten- berg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alp- hazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022.","title":"Acquisition of chess knowledge in alphazero","authors":"Thomas Mcgrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, Vladimir Kramnik","year":"2022","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mi et al., 2020] Jian Xun Mi, An Di Li, and Li Fang Zhou. Review study of interpretation methods for future in- terpretable machine learning. IEEE Access, 8:191969- 191985, 2020.","title":"Review study of interpretation methods for future interpretable machine learning","authors":"Jian Xun, Mi, An Di Li, Li Fang Zhou","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nasu, 2018] Yu Nasu. Efficiently updatable neural- network-based evaluation functions for computer shogi. The 28th World Computer Shogi Championship Appeal Document, 2018.","title":"Efficiently updatable neuralnetwork-based evaluation functions for computer shogi","authors":"Yu Nasu","year":"2018","venue":"The 28th World Computer Shogi Championship Appeal Document","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pálsson and Björnsson, 2021] Aðalsteinn Pálsson and Yn- gvi Björnsson. Evaluating interpretability methods for dnns in game-playing agents. In Cameron Browne, Aki- hiro Kishimoto, and Jonathan Schaeffer, editors, Advances in Computer Games -17th International Conference, ACG 2021, Virtual Event, November 23-25, 2021, Revised Se- lected Papers, volume 13262 of Lecture Notes in Com- puter Science, pages 71-81. Springer, 2021.","title":"Aðalsteinn Pálsson and Yngvi Björnsson. Evaluating interpretability methods for dnns in game-playing agents","authors":"Björnsson Pálsson","year":"2021","venue":"Advances in Computer Games -17th International Conference, ACG 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puri et al., 2020] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krish- namurthy, and Sameer Singh. Explain your move: Under- standing agent actions using specific and relevant feature attribution. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.","title":"Explain your move: Understanding agent actions using specific and relevant feature attribution","authors":"Puri","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"Why should i trust you?\" Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, 13-17-Augu:1135-1144, 2016.","title":"Why should i trust you?\" Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sadikov et al., 2006] Aleksander Sadikov, Martin Mozina, Matej Guid, Jana Krivec, and Ivan Bratko. Automated chess tutor. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L. M. Donkers, editors, Computers and Games, volume 4630 of Lecture Notes in Computer Science, pages 13-25. Springer, 2006.","title":"Automated chess tutor","authors":"Aleksander Sadikov, Martin Mozina, Matej Guid, Jana Krivec, Ivan Bratko","year":"2006","venue":"Computers and Games","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schwalbe and Finzel, 2021] Gesina Schwalbe and Bettina Finzel. XAI method properties: A (meta-)study. CoRR, abs/2105.07190, 2021.","title":"XAI method properties: A (meta-)study","authors":"Finzel Schwalbe, Gesina Schwalbe, Bettina Finzel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stockfish, 2022a] Stockfish. Introducing NNUE eval- uation. https://blog.stockfishchess.org/post/ 625828091343896577/introducing-nnue-evaluation, 2022. Accessed: 2022-04-05.","title":"Stockfish. Introducing NNUE evaluation","authors":"Stockfish","year":"2022","venue":null,"doi":null,"url":"https://blog.stockfishchess.org/post/625828091343896577/introducing-nnue-evaluation,2022","internal_url":null},{"position":24,"raw":"[Stockfish, 2022b] Stockfish. NNUE readme. https://github. com/glinscott/nnue-pytorch/blob/master/docs/nnue.md, 2022. Accessed: 2022-04-05.","title":"Stockfish. NNUE readme","authors":"Stockfish","year":"2022","venue":null,"doi":null,"url":"https://github.com/glinscott/nnue-pytorch/blob/master/docs/nnue.md,2022","internal_url":null},{"position":25,"raw":"[Stockfish, 2022c] Stockfish. Stockfish: Strong open source chess engine. https://stockfishchess.org/, 2022. Accessed: 2022-01-30.","title":"Stockfish. Stockfish: Strong open source chess engine","authors":"Stockfish","year":"2022","venue":null,"doi":null,"url":"https://stockfishchess.org/,2022","internal_url":null},{"position":26,"raw":"[Stockfish, 2022d] Stockfish. Training datasets. https://github.com/glinscott/nnue-pytorch/wiki/ Training-datasets, 2022. Accessed: 2022-01-30.","title":"Stockfish. Training datasets","authors":"Stockfish","year":"2022","venue":null,"doi":null,"url":"https://github.com/glinscott/nnue-pytorch/wiki/Training-datasets,2022","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}