{"type":"ScholarlyArticle","title":"Robust Learning for Repeated Stochastic Games via Meta-Gaming","authors":[{"name":"Jacob W. Crandall","slug":"jacob-w-crandall","given":"Jacob W.","family":"Crandall","affiliation":"Masdar Institute of Science and Technology Abu Dhabi , UAE"}],"abstract":"In repeated stochastic games (RSGs), an agent must quickly adapt to the behavior of previously unknown associates, who may themselves be learning. This machine-learning problem is particularly challenging due, in part, to the presence of multiple (even infinite) equilibria and inherently large strategy spaces. In this paper, we introduce a method to reduce the strategy space of two-player general-sum RSGs to a handful of expert strategies. This process, called mega, effectually reduces an RSG to a bandit problem. We show that the resulting strategy space preserves several important properties of the original RSG, thus enabling a learner to produce robust strategies within a reasonably small number of interactions. To better establish strengths and weaknesses of this approach, we empirically evaluate the resulting learning system against other algorithms in three different RSGs.","doi":null,"url":"https://ijcai.org/proceedings/2015/481","pdf":"https://ijcai.org/proceedings/2015/0481.pdf","markdown":"https://ijcai.org/proceedings/2015/481.md","bibtex":"https://ijcai.org/proceedings/2015/bibtex/481","volume":{"slug":"2015","title":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2015","url":"https://ijcai.org/proceedings/2015/","year":2015,"location":"Buenos Aires, Argentina","dates":"25–31 July 2015","editors":"Qiang Yang and Michael Wooldridge","isbn":"978-1-57735-738-4","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Special Track on Machine Learning","track":"Special Track on Machine Learning","number":481,"pages":{"first":3416,"last":3421},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2012] R. Arora, O. Dekel, and A. Tewari. Online ban- dit learning against an adaptive adversary: from regret to policy regret. In ICML, pages 1503-1510, 2012.","title":"Online bandit learning against an adaptive adversary: from regret to policy regret","authors":"R Arora, O Dekel, A Tewari","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 1995] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: the adversarial multi- armed bandit problem. In Proc. of the 36th Symp. on the Foun- dations of CS, pages 322-331, 1995.","title":"Gambling in a rigged casino: the adversarial multiarmed bandit problem","authors":"P Auer, N Cesa-Bianchi, Y Freund, R E Schapire","year":"1995","venue":"Proc. of the 36th Symp. on the Foundations of CS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite- time analysis of the multi-armed bandit problem. Machine Learn- ing, 47:235-256, 2002.","title":"Finitetime analysis of the multi-armed bandit problem","authors":"P Auer, N Cesa-Bianchi, P Fischer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowling and Veloso, 2002] M. Bowling and M. Veloso. Multia- gent learning using a variable learning rate. Artificial Intelli- gence, 136(2):215-250, 2002.","title":"Multiagent learning using a variable learning rate","authors":"M Bowling, M Veloso","year":"2002","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bowling, 2004] M. Bowling. Convergence and no-regret in multi- agent learning. In NIPS, pages 209-216, 2004.","title":"Convergence and no-regret in multiagent learning","authors":"M Bowling","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cesa-Bianchi et al., 2013] N. Cesa-Bianchi, O. Dekel, and O. Shamir. Online learning with switching costs and other adaptive adversaries. In NIPS, pages 1160-1168, 2013.","title":"Online learning with switching costs and other adaptive adversaries","authors":"N Cesa-Bianchi, O Dekel, O Shamir","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Claus and Boutilier, 1998] C. Claus and C. Boutilier. The dynam- ics of reinforcement learning in cooperative multiagent systems. In AAAI, pages 746-752, 1998.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"C Claus, C Boutilier","year":"1998","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Crandall, 2012] J. W. Crandall. Just add Pepper: extending learn- ing algorithms for repeated matrix games to repeated markov games. In AAMAS, pages 399-406, 2012.","title":"Just add Pepper: extending learning algorithms for repeated matrix games to repeated markov games","authors":"J W Crandall","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Crandall, 2014] J. W. Crandall. Towards minimizing disappoint- ment in repeated games. Journal of Artificial Intelligence Re- search, 49:111-142, 2014.","title":"Towards minimizing disappointment in repeated games","authors":"J W Crandall","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[de Cote and Littman, 2008] E. de Cote and M. L. Littman. A polynomial-time Nash equilibrium algorithm for repeated stochastic games. In UAI, pages 419-426, 2008.","title":"A polynomial-time Nash equilibrium algorithm for repeated stochastic games","authors":"E De Cote, M L Littman","year":"2008","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[de Farias and Megiddo, 2004] D. de Farias and N. Megiddo. Exploration-exploitation tradeoffs for expert algorithms in reac- tive environments. In NIPS, pages 409-416, 2004.","title":"Exploration-exploitation tradeoffs for expert algorithms in reactive environments","authors":"D Farias, N Megiddo","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dietterich, 2000] T. G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, LN in CS, vol 1857, pp. 1-15. Springer Berlin Heidelberg, 2000.","title":"Ensemble methods in machine learning","authors":"T G Dietterich","year":"2000","venue":"Multiple Classifier Systems, LN in CS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Elidrisi et al., 2014] M. Elidrisi, N. Johnson, M. Gini, and J. W. Crandall. Fast adaptive learning in repeated stochastic games by game abstraction. In AAMAS, 2014.","title":"Fast adaptive learning in repeated stochastic games by game abstraction","authors":"M Elidrisi, N Johnson, M Gini, J W Crandall","year":"2014","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fudenberg and Levine, 1998] D. Fudenberg and D. Levine. The Theory of Learning in Games. The MIT Press, 1998.","title":"The Theory of Learning in Games","authors":"D Fudenberg, D Levine","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ganzfried et al., 2012] S. Ganzfried, T. Sandholm, and K. Waugh. Strategy purification and thresholding: Effective non- equilibriuam approaches for playing large games. In AAMAS, 2012.","title":"Strategy purification and thresholding: Effective nonequilibriuam approaches for playing large games","authors":"S Ganzfried, T Sandholm, K Waugh","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gilpin and Sandholm, 2006] A. Gilpin and T. Sandholm. A com- petitive Texas Hold'em poker player via automated abstraction and real-time equilibrium computation. In AAAI, 2006.","title":"A competitive Texas Hold'em poker player via automated abstraction and real-time equilibrium computation","authors":"A Gilpin, T Sandholm","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gintis, 2000] Herbert Gintis. Game Theory Evolving: A Problem- Centered Introduction to Modeling Strategic Behavior. Princeton University Press, 2000.","title":"Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Behavior","authors":"Gintis","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Greenwald and Hall, 2003] A. Greenwald and K. Hall. Correlated Q-learning. In ICML, pages 242-249, 2003.","title":"Correlated Q-learning","authors":"A Greenwald, K Hall","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hu and Wellman, 1998] J. Hu and M. P. Wellman. Multiagent re- inforcement learning: Theoretical framework and an algorithm. In ICML, pages 242-250, 1998.","title":"Multiagent reinforcement learning: Theoretical framework and an algorithm","authors":"J Hu, M P Wellman","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Johanson et al., 2012] M. Johanson, N. Bard, M. Lanctot, R. Gib- son, and M. Bowling. Evaluating state-space abstractions in extensive-form games. In AAMAS, pages 837-846, 2012.","title":"Evaluating state-space abstractions in extensive-form games","authors":"M Johanson, N Bard, M Lanctot, R Gibson, M Bowling","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Littman and Stone, 2001] M. L. Littman and P. Stone. Leading best-response strategies in repeated games. In IJCAI workshop on Economic Agents, Models, and Mechanisms, Seattle, WA, 2001.","title":"Leading best-response strategies in repeated games","authors":"M L Littman, P Stone","year":"2001","venue":"IJCAI workshop on Economic Agents, Models, and Mechanisms","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Littman et al., 1995] M. L. Littman, T. L. Dean, and L. P. Kael- bling. On the complexity of solving markov decision problems. In UAI, 1995.","title":"On the complexity of solving markov decision problems","authors":"M L Littman, T L Dean, L P Kaelbling","year":"1995","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Littman, 1994] M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"M L Littman","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Littman, 2001] M. L. Littman. Friend-or-foe: Q-learning in general-sum games. In ICML, pages 322-328, 2001.","title":"Friend-or-foe: Q-learning in general-sum games","authors":"M L Littman","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Oudah et al., 2015] M. Oudah, V. Babushkin, T. Chenlinangjia, and J. W. Crandall. Learning to interact with a human partner. In HRI, 2015.","title":"Learning to interact with a human partner","authors":"M Oudah, V Babushkin, T Chenlinangjia, J W Crandall","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Papadimitriou and Tsitsiklis, 1987] C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov chain decision processes. Mathematics of Operations Research, 12(2):441-450, 1987.","title":"The complexity of Markov chain decision processes","authors":"C H Papadimitriou, J N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Powers and Shoham, 2005] R. Powers and Y. Shoham. Learning against opponents with bounded memory. In IJCAI, pages 817- 822, 2005.","title":"Learning against opponents with bounded memory","authors":"R Powers, Y Shoham","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2005/167"},{"position":28,"raw":"[Rummery and Niranjan, 1994] G. A. Rummery and M. Niranjan. On-line Q-learning using connectionist sytems. Technical Report CUED/F-INFENG-TR 166, Cambridge University, UK, 1994.","title":"On-line Q-learning using connectionist sytems","authors":"G A Rummery, M Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sandholm and Singh, 2012] T. Sandholm and S. Singh. Lossy stochastic game abstraction with bounds. In EC, pages 880-897, 2012.","title":"Lossy stochastic game abstraction with bounds","authors":"T Sandholm, S Singh","year":"2012","venue":"EC","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Schnizlein et al., 2009] D. Schnizlein, M. Bowling, and D Szafron. Probabilistic state translation in extensive games with large action sets. In IJCAI, 2009.","title":"Probabilistic state translation in extensive games with large action sets","authors":"D Schnizlein, M Bowling, Szafron","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/55"},{"position":31,"raw":"[Taylor and Jonker, 1978] P. D. Taylor and L. Jonker. Evolution- arily stable strategies and game dynamics. Mathematical Bio- sciences, 40:145-156, 1978.","title":"Evolutionarily stable strategies and game dynamics","authors":"P D Taylor, L Jonker","year":"1978","venue":"Mathematical Biosciences","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zinkevich et al., 2007] M. Zinkevich, M. Bowling, M. Johanson, and C. Piccione. Regret minimization in games with incomplete information. In NIPS, 2007.","title":"Regret minimization in games with incomplete information","authors":"M Zinkevich, M Bowling, M Johanson, C Piccione","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}