{"type":"ScholarlyArticle","title":"Balancing Two-Player Stochastic Games with Soft Q-Learning","authors":[{"name":"Jordi Grau-Moya","slug":"jordi-grau-moya","given":"Jordi","family":"Grau-Moya","affiliation":""},{"name":"Felix Leibfried","slug":"felix-leibfried","given":"Felix","family":"Leibfried","affiliation":""},{"name":"Haitham Bou-Ammar","slug":"haitham-bou-ammar","given":"Haitham","family":"Bou-Ammar","affiliation":""}],"abstract":"Within the context of video games the notion of perfectly rational agents can be undesirable as it leads to uninteresting situations, where humans face tough adversarial decision makers. Current frameworks for stochastic games and reinforcement learning prohibit tuneable strategies as they seek optimal performance. In this paper, we enable such tuneable behaviour by generalising soft Q-learning to stochastic games, where more than one agent interact strategically. We contribute both theoretically and empirically. On the theory side, we show that games with soft Q-learning exhibit a unique value and generalise team games and zero-sum games far beyond these two extremes to cover a continuous spectrum of gaming behaviour. Experimentally, we show how tuning agents' constraints affect performance and demonstrate, through a neural network architecture, how to reliably balance games with high-dimensional representations.","doi":"10.24963/ijcai.2018/37","url":"https://ijcai.org/proceedings/2018/37","pdf":"https://ijcai.org/proceedings/2018/0037.pdf","markdown":"https://ijcai.org/proceedings/2018/37.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/37","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":37,"pages":{"first":268,"last":274},"keywords":["Agent-based and Multi-agent Systems: Agent Theories and Models","Machine Learning: Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[],"references":[{"position":1,"raw":"[Agussurja and Lau, 2012] Lucas Agussurja and Hoong Chuin Lau. Toward large-scale agent guid- ance in an urban taxi service. Uncertainty in Artificial Intelligence, 2012.","title":"Toward large-scale agent guidance in an urban taxi service","authors":"Lau Agussurja, Lucas Agussurja, Hoong Chuin Lau","year":"2012","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Braun et al., 2011] Daniel A Braun, Pedro A Ortega, Evan- gelos Theodorou, and Stefan Schaal. Path integral control and bounded rationality. In Adaptive Dynamic Program- ming And Reinforcement Learning (ADPRL), 2011 IEEE Symposium on, pages 202-209. IEEE, 2011.","title":"Path integral control and bounded rationality","authors":"Pedro A Daniel A Braun, Evangelos Ortega, Stefan Theodorou, Schaal","year":"2011","venue":"Adaptive Dynamic Programming And Reinforcement Learning (ADPRL)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Busoniu et al., 2010] Lucian Busoniu, Robert Babuska, Bart De Schutter, and Damien Ernst. Reinforcement learn- ing and dynamic programming using function approxima- tors, volume 39. CRC press, 2010.","title":"Reinforcement learning and dynamic programming using function approximators","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter, Damien Ernst","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fox et al., 2016] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft up- dates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pages 202-211. AUAI Press, 2016.","title":"Taming the noise in reinforcement learning via soft updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2016","venue":"Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodrich and Schultz, 2007] Michael A Goodrich and Alan C Schultz. Human-robot interaction: a survey. Foundations and trends in human-computer interaction, 1(3):203-275, 2007.","title":"Human-robot interaction: a survey","authors":"A Michael, Alan C Goodrich, Schultz","year":"2007","venue":"Foundations and trends in human-computer interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grau-Moya et al., 2016] Jordi Grau-Moya, Felix Leibfried, Tim Genewein, and Daniel A Braun. Planning with information-processing constraints and model uncertainty in markov decision processes. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 475-491. Springer, 2016.","title":"Planning with information-processing constraints and model uncertainty in markov decision processes","authors":"Grau-Moya","year":"2016","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Confer- ence on Machine Learning, pages 1352-1361, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hunicke, 2005] Robin Hunicke. The case for dynamic difficulty adjustment in games. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology, pages 429-433. ACM, 2005.","title":"The case for dynamic difficulty adjustment in games","authors":"Robin Hunicke, Hunicke","year":"2005","venue":"Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kappen, 2005] Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. Jour- nal of statistical mechanics: theory and experiment, 2005(11):P11011, 2005.","title":"Path integrals and symmetry breaking for optimal control theory","authors":"J Hilbert, Kappen","year":"2005","venue":"Journal of statistical mechanics: theory and experiment","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Leibfried et al., 2017] Felix Leibfried, Jordi Grau-Moya, and Haitham Bou-Ammar. An information-theoretic op- timality principle for deep reinforcement learning. arXiv preprint arXiv:1708.01867, 2017.","title":"An information-theoretic optimality principle for deep reinforcement learning","authors":"Felix Leibfried, Jordi Grau-Moya, Haitham Bou-Ammar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lieder et al., 2012] Falk Lieder, Tom Griffiths, and Noah Goodman. Burn-in, bias, and the rationality of anchor- ing. In Advances in neural information processing sys- tems, pages 2690-2798, 2012.","title":"Burn-in, bias, and the rationality of anchoring","authors":"Lieder","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Littman and Szepesvári, 1996] Michael L Littman and Csaba Szepesvári. A generalized reinforcement-learning model: Convergence and applications. In International Conference on Machine Learning, 1996.","title":"A generalized reinforcement-learning model: Convergence and applications","authors":"L Michael, Csaba Littman, Szepesvári","year":"1996","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings of the 11th International Conference on Machine Learning, 1994, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"Proceedings of the 11th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Littman, 2001] Michael L Littman. Friend or foe q-learning in general-sum games. In In Proceedings of the 18th Int. Conf. on Machine Learning. Citeseer, 2001.","title":"Friend or foe q-learning in general-sum games","authors":"Michael L Littman","year":"2001","venue":"Proceedings of the 18th Int. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mguni et al., 2018] David Mguni, Joel Jennings, and En- rique Munoz de Cote. Decentralised learning in systems with many, many strategic agents. In AAAI Conference on Artificial Intelligence, 2018.","title":"Decentralised learning in systems with many, many strategic agents","authors":"David Mguni, Joel Jennings","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ortega and Braun, 2013] Pedro A Ortega and Daniel A Braun. Thermodynamics as a theory of decision-making with information-processing costs. In Proc. R. Soc. A, vol- ume 469, page 20120683. The Royal Society, 2013.","title":"Thermodynamics as a theory of decision-making with information-processing costs","authors":"Pedro A Ortega, Daniel A Braun","year":"2013","venue":"Proc. R. Soc. A","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ortega and Stocker, 2016] Pedro A Ortega and Alan A Stocker. Human decision-making under limited time. In Advances in Neural Information Processing Systems, pages 100-108, 2016.","title":"Human decision-making under limited time","authors":"Pedro A Ortega, Alan A Stocker","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Osborne and Rubinstein, 1994] Martin J Osborne and Ariel Rubinstein. A course in game theory. 1994.","title":"A course in game theory","authors":"J Martin, Ariel Osborne, Rubinstein","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peters et al., 2010] Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Pro- ceedings of the Twenty-Fourth AAAI Conference on Artifi- cial Intelligence, pages 1607-1612. AAAI Press, 2010.","title":"Relative entropy policy search","authors":"Peters","year":"2010","venue":"Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Puterman, 1994] Martin Puterman. Markov decision pro- cesses: Discrete stochastic dynamic programming. 1994.","title":"Markov decision processes: Discrete stochastic dynamic programming","authors":"Martin Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rubin et al., 2012] Jonathan Rubin, Ohad Shamir, and Naf- tali Tishby. Trading value and information in mdps. De- cision Making with Imperfect Decision Makers, pages 57- 74, 2012.","title":"Trading value and information in mdps. Decision Making with Imperfect Decision Makers","authors":"Jonathan Rubin, Ohad Shamir, Naftali Tishby","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd In- ternational Conference on Machine Learning (ICML-15), pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] R Sutton and A Barto. Reinforce- ment learning. MIT Press, Cambridge, 1998.","title":"Reinforcement learning","authors":"R Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tishby and Polani, 2011] Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perception- action cycle, pages 601-636. Springer, 2011.","title":"Information theory of decisions and actions","authors":"Naftali Tishby, Daniel Polani","year":"2011","venue":"Perceptionaction cycle","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van den Broek et al., 2010] Bart van den Broek, Wim Wiegerinck, and Bert Kappen. Risk sensitive path integral control. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI'10, pages 615- 622, Arlington, Virginia, United States, 2010. AUAI Press.","title":"Risk sensitive path integral control","authors":"Broek Van Den","year":"2010","venue":"Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI'10","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}