{"type":"ScholarlyArticle","title":"Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluation","authors":[{"name":"Guiliang Liu","slug":"guiliang-liu","given":"Guiliang","family":"Liu","affiliation":"University , Burnaby , Canada"},{"name":"Oliver Schulte","slug":"oliver-schulte","given":"Oliver","family":"Schulte","affiliation":"University , Burnaby , Canada"}],"abstract":"A variety of machine learning models have been proposed to assess the performance of players in professional sports. However, they have only a limited ability to model how player performance depends on the game context. This paper proposes a new approach to capturing game context: we apply Deep Reinforcement Learning (DRL) to learn an action-value Q function from 3M play-by-play events in the National Hockey League (NHL). The neural network representation integrates both continuous context signals and game history, using a possession-based LSTM. The learned Q-function is used to value players' actions under different game contexts. To assess a player's overall performance, we introduce a novel  Game Impact Metric (GIM) that aggregates the values of the player's actions. Empirical Evaluation shows GIM is consistent throughout a play season, and correlates highly with standard success measures  and future salary.","doi":"10.24963/ijcai.2018/478","url":"https://ijcai.org/proceedings/2018/478","pdf":"https://ijcai.org/proceedings/2018/0478.pdf","markdown":"https://ijcai.org/proceedings/2018/478.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/478","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":478,"pages":{"first":3442,"last":3448},"keywords":["Machine Learning: Data Mining","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Albert et al., 2017] Jim Albert, Mark E Glickman, Tim B Swartz, and Ruud H Koning. Handbook of Statistical Methods and Analyses in Sports. CRC Press, 2017.","title":"Handbook of Statistical Methods and Analyses in Sports","authors":"Albert","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Buttrey et al., 2011] Samuel Buttrey, Alan Washburn, and Wilson Price. Estimating NHL scoring rates. Journal of Quantitative Analysis in Sports, 7(3), 2011.","title":"Estimating NHL scoring rates","authors":"Buttrey","year":"2011","venue":"Journal of Quantitative Analysis in Sports","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cervone et al., 2014] Dan Cervone, Alexander D'Amour, Luke Bornn, and Kirk Goldsberry. Pointwise: Predicting points and valuing decisions in real time with NBA optical tracking data. In MIT Sloan Sports Analytics Conference, 2014.","title":"Pointwise: Predicting points and valuing decisions in real time with NBA optical tracking data","authors":"Dan Cervone, D' Alexander, Luke Amour, Kirk Bornn, Goldsberry","year":"2014","venue":"MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Decroos et al., 2017] Tom Decroos, Vladimir Dzyuba, Jan Van Haaren, and Jesse Davis. Predicting soccer high- lights from spatio-temporal match event streams. In AAAI 2017, pages 1302-1308, 2017.","title":"Predicting soccer highlights from spatio-temporal match event streams","authors":"Decroos","year":"2017","venue":"AAAI 2017","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Decroos et al., 2018] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. arXiv preprint arXiv:1802.07127, 2018.","title":"Actions speak louder than goals: Valuing player actions in soccer","authors":"Decroos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gerstenberg et al., 2014] Tobias Gerstenberg, Tomer Ull- man, Max Kleiman-Weiner, David Lagnado, and Josh Tenenbaum. Wins above replacement: Responsibility at- tributions as counterfactual replacements. In Proceedings of the Cognitive Science Society, volume 36, 2014.","title":"Wins above replacement: Responsibility attributions as counterfactual replacements","authors":"Gerstenberg","year":"2014","venue":"Proceedings of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep recurrent Q-learning for partially observ- able MDPs. CoRR, abs/1507.06527, 2015.","title":"Deep recurrent Q-learning for partially observable MDPs","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Idson and Kahane, 2000] Todd L Idson and Leo H Kahane. Team effects on compensation: an application to salary determination in the National Hockey League. Economic Inquiry, 38(2):345-357, 2000.","title":"Team effects on compensation: an application to salary determination in the National Hockey League","authors":"Kahane ; Idson, L Todd, Leo H Idson, Kahane","year":"2000","venue":"Economic Inquiry","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaplan et al., 2014] Edward H Kaplan, Kevin Mongeon, and John T Ryan. A Markov model for hockey: Man- power differential and win probability added. INFOR: In- formation Systems and Operational Research, 52(2):39- 50, 2014.","title":"A Markov model for hockey: Manpower differential and win probability added","authors":"Kevin Edward H Kaplan, John T Mongeon, Ryan","year":"2014","venue":"INFOR: Information Systems and Operational Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings International Conference on Machine Learning, volume 157, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"Proceedings International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Macdonald, 2011] Brian Macdonald. A regression-based adjusted plus-minus statistic for NHL players. Journal of Quantitative Analysis in Sports, 7(3):29, 2011.","title":"A regression-based adjusted plus-minus statistic for NHL players","authors":"Brian Macdonald","year":"2011","venue":"Journal of Quantitative Analysis in Sports","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McHale et al., 2012] Ian G McHale, Philip A Scarf, and David E Folker. On the development of a soccer player performance rating system for the English Premier League. Interfaces, 42(4):339-351, 2012.","title":"On the development of a soccer player performance rating system for the English Premier League","authors":"Ian G Mchale, Philip A Scarf, David E Folker","year":"2012","venue":"Interfaces","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Omidiran, 2011] Dapo Omidiran. A new look at adjusted plus/minus for basketball analysis. In MIT Sloan Sports Analytics Conference [online], 2011.","title":"A new look at adjusted plus/minus for basketball analysis","authors":"Dapo Omidiran","year":"2011","venue":"MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pettigrew, 2015] Stephen Pettigrew. Assessing the offensive productivity of NHL players using in-game win probabili- ties. In MIT Sloan Sports Analytics Conference, 2015.","title":"Assessing the offensive productivity of NHL players using in-game win probabilities","authors":"Stephen Pettigrew","year":"2015","venue":"MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Routley and Schulte, 2015] Kurt Routley and Oliver Schulte. A Markov game model for valuing player actions in ice hockey. In Proceedings Uncertainty in Artificial Intelligence (UAI), pages 782-791, 2015.","title":"A Markov game model for valuing player actions in ice hockey","authors":"Schulte Routley, Kurt Routley, Oliver Schulte","year":"2015","venue":"Proceedings Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schuckers and Curro, 2013] Michael Schuckers and James Curro. Total hockey rating (THoR): A comprehensive statistical rating of national hockey league forwards and defensemen based upon all on-ice events. In MIT Sloan Sports Analytics Conference, 2013.","title":"Total hockey rating (THoR): A comprehensive statistical rating of national hockey league forwards and defensemen based upon all on-ice events","authors":"Michael Schuckers, James Curro","year":"2013","venue":"MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulte et al., 2017a] Oliver Schulte, Mahmoud Khademi, Sajjad Gholami, et al. A Markov game model for valuing actions, locations, and team performance in ice hockey. Data Mining and Knowledge Discovery, pages 1-23, 2017.","title":"A Markov game model for valuing actions, locations, and team performance in ice hockey","authors":"Schulte","year":"2017","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulte et al., 2017b] Oliver Schulte, Zeyu Zhao, Mehrsan Javan, and Philippe Desaulniers. Apples-to-apples: Clus- tering and ranking NHL players using location informa- tion and scoring impact. In MIT Sloan Sports Analytics Conference, 2017.","title":"Apples-to-apples: Clustering and ranking NHL players using location information and scoring impact","authors":"Schulte","year":"2017","venue":"MIT Sloan Sports Analytics Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, volume 135. MIT Press Cambridge, 1998.","title":"Introduction to reinforcement learning","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thomas et al., 2013] A.C. Thomas, S.L. Ventura, S. Jensen, and S. Ma. Competing process hazard function models for player ratings in ice hockey. The Annals of Applied Statistics, 7(3):1497-1524, 2013.","title":"Competing process hazard function models for player ratings in ice hockey","authors":"A C Thomas, S L Ventura, S Jensen, S Ma","year":"2013","venue":"The Annals of Applied Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}