{"type":"ScholarlyArticle","title":"What Game Are We Playing? End-to-end Learning in Normal and Extensive Form Games","authors":[{"name":"Chun Kai Ling","slug":"chun-kai-ling","given":"Chun Kai","family":"Ling","affiliation":"School of Computer Science , Carnegie Mellon University"},{"name":"Fei Fang","slug":"fei-fang","given":"Fei","family":"Fang","affiliation":"School of Computer Science , Carnegie Mellon University"},{"name":"J. Zico Kolter","slug":"j-zico-kolter","given":"J. Zico","family":"Kolter","affiliation":"School of Computer Science , Carnegie Mellon University"}],"abstract":"Although recent work in AI has made great progress in solving large, zero-sum, extensive-form games, the underlying assumption in most past work is that the parameters of the game itself are known to the agents.  This paper deals with the relatively under-explored but equally important \"inverse\" setting, where the parameters of the underlying game are not known to all agents, but must be learned through observations.  We propose a differentiable, end-to-end learning framework for addressing this task.  In particular, we consider a regularized version of the game, equivalent to a particular form of quantal response equilibrium, and develop 1) a primal-dual Newton method for finding such equilibrium points in both normal and extensive form games; and 2) a backpropagation method that lets us analytically compute gradients of all relevant game parameters through the solution itself.  This ultimately lets us learn the game by training in an end-to-end fashion, effectively by integrating a \"differentiable game solver\" into the loop of larger deep network architectures. We demonstrate the effectiveness of the learning method in several settings including poker and security game tasks.","doi":"10.24963/ijcai.2018/55","url":"https://ijcai.org/proceedings/2018/55","pdf":"https://ijcai.org/proceedings/2018/0055.pdf","markdown":"https://ijcai.org/proceedings/2018/55.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/55","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":55,"pages":{"first":396,"last":402},"keywords":["Agent-based and Multi-agent Systems: Algorithmic Game Theory","Machine Learning: Deep Learning","Agent-based and Multi-agent Systems: Noncooperative Games"],"videos":[],"references":[{"position":1,"raw":"[Amin et al., 2016] Kareem Amin, Satinder Singh, and Michael P Wellman. Gradient methods for stackelberg se- curity games. In Conference on Uncertainty in Artificial Intelligence, pages 2-11, 2016.","title":"Gradient methods for stackelberg security games","authors":"Kareem Amin, Satinder Singh, Michael P Wellman","year":"2016","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amos and Kolter, 2017] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. arXiv preprint arXiv:1703.00443, 2017.","title":"Optnet: Differentiable optimization as a layer in neural networks","authors":"Amos, Kolter, Brandon Amos, J Zico, Kolter","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blum et al., 2014] Avrim Blum, Nika Haghtalab, and Ariel D Procaccia. Learning optimal commitment to over- come insecurity. In Advances in Neural Information Pro- cessing Systems, pages 1826-1834, 2014.","title":"Learning optimal commitment to overcome insecurity","authors":"Avrim Blum, Nika Haghtalab, Ariel D Procaccia","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowling and Veloso, 2000] Michael Bowling and Manuela Veloso. An analysis of stochastic game theory for multiagent reinforcement learning. Technical re- port, CARNEGIE-MELLON UNIV PITTSBURGH PA SCHOOL OF COMPUTER SCIENCE, 2000.","title":"An analysis of stochastic game theory for multiagent reinforcement learning","authors":"Michael Bowling, Manuela Veloso","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bowling et al., 2015] Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science, 347(6218):145-149, 2015.","title":"Heads-up limit hold'em poker is solved","authors":"Michael Bowling, Neil Burch, Michael Johanson, Oskari Tammelin","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge univer- sity press, 2004.","title":"Convex optimization","authors":"Vandenberghe Boyd","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown and Sandholm, 2017] Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, page eaao1733, 2017.","title":"Superhuman ai for heads-up no-limit poker: Libratus beats top professionals","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Busoniu et al., 2008] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews, 38 (2), 2008, 2008.","title":"A comprehensive survey of multiagent reinforcement learning","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter","year":"2008","venue":"IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fang et al., 2016] Fei Fang, Thanh Hong Nguyen, Rob Pickles, Wai Y Lam, Gopalasamy R Clements, Bo An, Amandeep Singh, Milind Tambe, and Andrew Lemieux. Deploying paws: Field optimization of the protection as- sistant for wildlife security. 2016.","title":"Deploying paws: Field optimization of the protection assistant for wildlife security","authors":"Fei Fang, Thanh Hong Nguyen, Rob Pickles, Wai Y Lam, Gopalasamy R Clements, Bo An, Amandeep Singh, Milind Tambe, Andrew Lemieux","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fearnley et al., 2015] John Fearnley, Martin Gairing, Paul W Goldberg, and Rahul Savani. Learning equilibria of games via payoff queries. Journal of Machine Learning Research, 16:1305-1344, 2015.","title":"Learning equilibria of games via payoff queries","authors":"John Fearnley, Martin Gairing, Paul W Goldberg, Rahul Savani","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gould et al., 2016] Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447, 2016.","title":"On differentiating parameterized argmin and argmax problems with application to bi-level optimization","authors":"Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, Edison Guo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hoda et al., 2010] Samid Hoda, Andrew Gilpin, Javier Pena, and Tuomas Sandholm. Smoothing techniques for computing nash equilibria of sequential games. Mathe- matics of Operations Research, 35(2):494-512, 2010.","title":"Smoothing techniques for computing nash equilibria of sequential games","authors":"Samid Hoda, Andrew Gilpin, Javier Pena, Tuomas Sandholm","year":"2010","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Matthew Johnson, David K Duve- naud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. In Ad- vances in neural information processing systems, pages 2946-2954, 2016.","title":"Composing graphical models with neural networks for structured representations and fast inference","authors":"Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, Sandeep R Datta","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kroer et al., 2017] Christian Kroer, Kevin Waugh, Fatma Kilinc-Karzan, and Tuomas Sandholm. Theoretical and practical advances on smoothing for extensive-form games. arXiv preprint arXiv:1702.04849, 2017.","title":"Theoretical and practical advances on smoothing for extensive-form games","authors":"Kroer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Letchford et al., 2009] Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In International Symposium on Algorithmic Game Theory, pages 250-262. Springer, 2009.","title":"Learning and approximating the optimal strategy to commit to","authors":"Joshua Letchford, Vincent Conitzer, Kamesh Munagala","year":"2009","venue":"International Symposium on Algorithmic Game Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McKelvey and Palfrey, 1995] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior, 10(1):6-38, 1995.","title":"Quantal response equilibria for normal form games","authors":"D Richard, Thomas R Mckelvey, Palfrey","year":"1995","venue":"Games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McKelvey and Palfrey, 1998] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for ex- tensive form games. Experimental economics, 1(1):9-41, 1998.","title":"Quantal response equilibria for extensive form games","authors":"D Richard, Thomas R Mckelvey, Palfrey","year":"1998","venue":"Experimental economics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mertikopoulos and Sandholm, 2016] Panayotis Mer- tikopoulos and William H Sandholm. Learning in games via reinforcement and regularization. Mathematics of Operations Research, 41(4):1297-1324, 2016.","title":"Learning in games via reinforcement and regularization","authors":"Panayotis Mertikopoulos, William H Sandholm","year":"2016","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Moravčík et al., 2017] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artifi- cial intelligence in heads-up no-limit poker. Science, 356(6337):508-513, 2017.","title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.","title":"Automatic differentiation in pytorch","authors":"Paszke","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pita et al., 2009] James Pita, Manish Jain, Fernando Ordónez, Christopher Portway, Milind Tambe, Craig Western, Praveen Paruchuri, and Sarit Kraus. Using game theory for los angeles airport security. Ai Magazine, 30(1):43, 2009.","title":"Using game theory for los angeles airport security","authors":"James Pita, Manish Jain, Fernando Ordónez, Christopher Portway, Milind Tambe, Craig Western, Praveen Paruchuri, Sarit Kraus","year":"2009","venue":"Ai Magazine","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tieleman and Hinton, 2012] T. Tieleman and G. Hinton. Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.","title":"Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude","authors":"T Tieleman, G Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Von Stengel, 1996] Bernhard Von Stengel. Efficient com- putation of behavior strategies. Games and Economic Be- havior, 14(2):220-246, 1996.","title":"Efficient computation of behavior strategies","authors":"Von Stengel, ; Bernhard, Von Stengel","year":"1996","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vorobeychik et al., 2007] Yevgeniy Vorobeychik, Michael P Wellman, and Satinder Singh. Learning payoff functions in infinite games. Mach Learn, 67:145- 168, 2007.","title":"Learning payoff functions in infinite games","authors":"Yevgeniy Vorobeychik, Satinder Michael P Wellman, Singh","year":"2007","venue":"Mach Learn","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Waugh et al., 2011] Kevin Waugh, Brian D Ziebart, and J Andrew Bagnell. Computational rationalization: the in- verse equilibrium problem. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 1169-1176. Omnipress, 2011.","title":"Computational rationalization: the inverse equilibrium problem","authors":"Kevin Waugh, Brian D Ziebart, Andrew Bagnell","year":"2011","venue":"Proceedings of the 28th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zinkevich et al., 2008] Martin Zinkevich, Michael Johan- son, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in neural information processing systems, pages 1729-1736, 2008.","title":"Regret minimization in games with incomplete information","authors":"Martin Zinkevich, Michael Johanson, Michael Bowling, Carmelo Piccione","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}