{"type":"ScholarlyArticle","title":"Inverse Reinforcement Learning for Team Sports: Valuing Actions and Players","authors":[{"name":"Yudong Luo","slug":"yudong-luo","given":"Yudong","family":"Luo","affiliation":"School of Computing Science , Simon Fraser University , Canada"},{"name":"Oliver Schulte","slug":"oliver-schulte","given":"Oliver","family":"Schulte","affiliation":"School of Computing Science , Simon Fraser University , Canada; Sportlogiq , Canada"},{"name":"Pascal Poupart","slug":"pascal-poupart","given":"Pascal","family":"Poupart","affiliation":"David R. Cheriton School of Computer Science , University of Waterloo , Canada"}],"abstract":"A major task of sports analytics is to rank players based on the impact of their actions. Recent methods have applied reinforcement learning (RL) to assess the value of actions from a learned action value or Q-function. A fundamental challenge for estimating action values is that explicit reward signals (goals) are very sparse in many team sports, such as ice hockey and soccer. This paper combines Q-function learning with inverse reinforcement learning (IRL) to provide a novel player ranking method. We treat professional play as expert demonstrations for learning an implicit reward function. Our method alternates single-agent IRL to learn a reward function for multiple agents; we provide a theoretical justification for this procedure. Knowledge transfer is used to combine learned rewards and observed rewards from goals. Empirical evaluation, based on 4.5M play-by-play events in the National Hockey League (NHL), indicates that player ranking using the learned rewards achieves high correlations with standard success measures and temporal consistency throughout a season.","doi":"10.24963/ijcai.2020/464","url":"https://ijcai.org/proceedings/2020/464","pdf":"https://ijcai.org/proceedings/2020/0464.pdf","markdown":"https://ijcai.org/proceedings/2020/464.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/464","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning Applications","track":"Main track","number":464,"pages":{"first":3356,"last":3363},"keywords":["Agent-based and Multi-agent Systems: Other","Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26572/file_26572.m3u8","length_sec":734,"provider":"hls"}],"references":[{"position":1,"raw":"[Albert et al., 2017] Jim Albert, Mark E Glickman, Tim B Swartz, and Ruud H Koning. Handbook of Statistical Methods and Analyses in Sports. 2017.","title":"Handbook of Statistical Methods and Analyses in Sports","authors":"Albert","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Decroos et al., 2018] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. arXiv preprint arXiv:1802.07127v1, 2018.","title":"Actions speak louder than goals: Valuing player actions in soccer","authors":"Decroos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Decroos et al., 2019] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. In Proceedings of the 25th International Conference on Knowledge Discovery & Data Mining (KDD-19), pages 1851-1861, 2019.","title":"Actions speak louder than goals: Valuing player actions in soccer","authors":"Decroos","year":"2019","venue":"Proceedings of the 25th International Conference on Knowledge Discovery & Data Mining (KDD-19)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gerstenberg et al., 2014] Tobias Gerstenberg, Tomer Ull- man, Max Kleiman-Weiner, David Lagnado, and Josh Tenenbaum. Wins above replacement: Responsibility at- tributions as counterfactual replacements. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 36, 2014.","title":"Wins above replacement: Responsibility attributions as counterfactual replacements","authors":"Gerstenberg","year":"2014","venue":"Proceedings of the Annual Meeting of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.","title":"A kernel two-sample test","authors":"Arthur Gretton, Karsten M Borgwardt, J Malte, Bernhard Rasch, Alexander Schölkopf, Smola","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jaynes, 1957] Edwin T Jaynes. Information theory and sta- tistical mechanics. Physical review, 106(4):620, 1957.","title":"Information theory and statistical mechanics","authors":"T Edwin, Jaynes","year":"1957","venue":"Physical review","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings of the 11st International Conference on Machine learning (ICML-94), volume 157, pages 157-163. 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"Proceedings of the 11st International Conference on Machine learning (ICML-94)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu and Schulte, 2018] Guiliang Liu and Oliver Schulte. Deep reinforcement learning in ice hockey for context- aware player evaluation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18), page 3442-3448, 2018.","title":"Deep reinforcement learning in ice hockey for contextaware player evaluation","authors":"Schulte Liu, Guiliang Liu, Oliver Schulte","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Long et al., 2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th In- ternational Conference on Machine Learning (ICML-17), volume 70, pages 2208-2217, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML-17)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Macdonald, 2011] Brian Macdonald. An improved adjusted plus-minus statistic for NHL players. In Proceedings of the 5th annual MIT Sloan Sports Analytics Conference, vol- ume 3, pages 1-8, 2011.","title":"An improved adjusted plus-minus statistic for NHL players","authors":"Brian Macdonald","year":"2011","venue":"Proceedings of the 5th annual MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Macdonald, 2012] Brian Macdonald. An expected goals model for evaluating NHL teams and players. In Proceed- ings of the 6th annual MIT Sloan Sports Analytics Confer- ence, 2012.","title":"An expected goals model for evaluating NHL teams and players","authors":"Brian Macdonald","year":"2012","venue":"Proceedings of the 6th annual MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mendez et al., 2018] Jorge Armando Mendez, Shashank Shivkumar, and Eric Eaton. Lifelong inverse reinforce- ment learning. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NIPS-18), pages 4502-4513, 2018.","title":"Lifelong inverse reinforcement learning","authors":"Jorge Armando Mendez, Shashank Shivkumar, Eric Eaton","year":"2018","venue":"Proceedings of the 32nd Conference on Neural Information Processing Systems (NIPS-18)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning (ICML-00), volume 1, page 2, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"Proceedings of the 17th International Conference on Machine Learning (ICML-00)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pettigrew, 2015] Stephen Pettigrew. Assessing the offensive productivity of nhl players using in-game win probabili- ties. In Proceedings of the 9th annual MIT sloan sports analytics conference, volume 2, page 8, 2015.","title":"Assessing the offensive productivity of nhl players using in-game win probabilities","authors":"Stephen Pettigrew","year":"2015","venue":"Proceedings of the 9th annual MIT sloan sports analytics conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Reese, 2019] Rob Reese. Fantasy defenseman top 50 rankings for 2019-20. https://www.nhl.com/news/nhl-fan tasy-hockey-top-50-defenseman-rankings-2019-20/c-2 82830728, 2019. [Online; accessed 15-October-2019].","title":"Fantasy defenseman top 50 rankings for 2019-20","authors":"Rob Reese","year":"2019","venue":null,"doi":null,"url":"https://www.nhl.com/news/nhl-fantasy-hockey-top-50-defenseman-rankings-2019-20/c-282830728,2019","internal_url":null},{"position":16,"raw":"[Routley and Schulte, 2015] Kurt Routley and Oliver Schulte. A Markov Game model for valuing player ac- tions in ice hockey. In Proceedings of the 31st Uncertainty in Artificial Intelligence (UAI-15), pages 782-791, 2015.","title":"A Markov Game model for valuing player actions in ice hockey","authors":"Schulte Routley, Kurt Routley, Oliver Schulte","year":"2015","venue":"Proceedings of the 31st Uncertainty in Artificial Intelligence (UAI-15)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schuckers and Curro, 2013] Michael Schuckers and James Curro. Total hockey rating (THoR): A comprehensive sta- tistical rating of national hockey league forwards and de- fensemen based upon all on-ice events. In Proceedings of the 7th annual MIT sloan sports analytics conference, 2013.","title":"Total hockey rating (THoR): A comprehensive statistical rating of national hockey league forwards and defensemen based upon all on-ice events","authors":"Michael Schuckers, James Curro","year":"2013","venue":"Proceedings of the 7th annual MIT sloan sports analytics conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulte et al., 2017] Oliver Schulte, Mahmoud Khademi, Sajjad Gholami, Zeyu Zhao, Mehrsan Javan, and Philippe Desaulniers. A Markov Game model for valuing actions, locations, and team performance in ice hockey. Data Min- ing and Knowledge Discovery, 31(6):1735-1757, 2017.","title":"A Markov Game model for valuing actions, locations, and team performance in ice hockey","authors":"Oliver Schulte, Mahmoud Khademi, Sajjad Gholami, Zeyu Zhao, Mehrsan Javan, Philippe Desaulniers","year":"2017","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Von Neumann and Morgenstern, 1947] John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 2nd rev. 1947.","title":"John Von Neumann and Oskar Morgenstern","authors":"Von Neumann, Morgenstern","year":"1947","venue":"Theory of games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Klabjan, 2018] Xingyu Wang and Diego Klab- jan. Competitive multi-agent inverse reinforcement learn- ing with sub-optimal demonstrations. In Proceedings of the 35th International Conference on Machine Learning (ICML-18), volume 80, pages 5130-5138, 2018.","title":"Xingyu Wang and Diego Klabjan. Competitive multi-agent inverse reinforcement learning with sub-optimal demonstrations","authors":"Klabjan Wang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML-18)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019] Baoxiang Wang, Tongfang Sun, and Xi- anjun Sam Zheng. Beyond winning and losing: Modeling human motivations and behaviors with vector-valued in- verse reinforcement learning. In Proceedings of the 15th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 195-201, 2019.","title":"Beyond winning and losing: Modeling human motivations and behaviors with vector-valued inverse reinforcement learning","authors":"Wang","year":"2019","venue":"Proceedings of the 15th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wulfmeier et al., 2016] Markus Wulfmeier, Dushyant Rao, and Ingmar Posner. Incorporating human domain knowl- edge into large scale cost function learning. arXiv preprint arXiv:1612.04318, 2016.","title":"Incorporating human domain knowledge into large scale cost function learning","authors":"Markus Wulfmeier, Dushyant Rao, Ingmar Posner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2019] Lantao Yu, Jiaming Song, and Stefano Er- mon. Multi-agent adversarial inverse reinforcement learn- ing. In Proceedings of the 36th International Confer- ence on Machine Learning (ICML-19), pages 7194-7201, 2019.","title":"Multi-agent adversarial inverse reinforcement learning","authors":"Lantao Yu, Jiaming Song, Stefano Ermon","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning (ICML-19)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew Maas, J An- drew Bagnell, and Anind K Dey. Maximum entropy in- verse reinforcement learning. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI-08), vol- ume 8, pages 1433-1438, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew Maas, Andrew Bagnell, Anind K Dey","year":"2008","venue":"Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI-08)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}