{"type":"ScholarlyArticle","title":"A Regularized Opponent Model with Maximum Entropy Objective","authors":[{"name":"Zheng Tian","slug":"zheng-tian","given":"Zheng","family":"Tian","affiliation":"University College London"},{"name":"Ying Wen","slug":"ying-wen","given":"Ying","family":"Wen","affiliation":"University College London"},{"name":"Zhichen Gong","slug":"zhichen-gong","given":"Zhichen","family":"Gong","affiliation":"University College London"},{"name":"Faiz Punakkath","slug":"faiz-punakkath","given":"Faiz","family":"Punakkath","affiliation":"University College London"},{"name":"Shihao Zou","slug":"shihao-zou","given":"Shihao","family":"Zou","affiliation":"University of Alberta"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"University College London"}],"abstract":"In a single-agent setting, reinforcement learning (RL) tasks can be cast into an inference problem by introducing a binary random variable o, which stands for the \"optimality\". In this paper, we redefine the binary random variable o in multi-agent setting and formalize multi-agent reinforcement learning (MARL) as probabilistic inference. We derive a variational lower bound of the likelihood of achieving the optimality and name it as Regularized Opponent Model with Maximum Entropy Objective (ROMMEO). From ROMMEO, we present a novel perspective on opponent modeling and show how it can improve the performance of training agents theoretically and empirically in cooperative games. To optimize ROMMEO, we first introduce a tabular Q-iteration method ROMMEO-Q with proof of convergence. We extend the exact algorithm to complex environments by proposing an approximate version, ROMMEO-AC. We evaluate these two algorithms on the challenging iterated matrix game and differential game respectively and show that they can outperform strong MARL baselines.","doi":"10.24963/ijcai.2019/85","url":"https://ijcai.org/proceedings/2019/85","pdf":"https://ijcai.org/proceedings/2019/0085.pdf","markdown":"https://ijcai.org/proceedings/2019/85.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/85","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":85,"pages":{"first":602,"last":608},"keywords":["Machine Learning: Deep Learning","Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abdolmaleki et al., 2018] Abbas Abdolmaleki, Jost Tobias Sprin- genberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Mar- tin A. Riedmiller. Maximum a posteriori policy optimisation. CoRR, abs/1806.06920, 2018.","title":"Maximum a posteriori policy optimisation","authors":"Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, Martin A Riedmiller","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowling and Veloso, 2001] Michael Bowling and Manuela Veloso. Rational and convergent learning in stochastic games. In IJCAI, San Francisco, CA, USA, 2001.","title":"Rational and convergent learning in stochastic games","authors":"Michael Bowling, Manuela Veloso","year":"2001","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2001/138"},{"position":3,"raw":"[Brown, 1951] George W. Brown. Iterative solution of games by fictitious play. In AAPA. New York, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"AAPA","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chalkiadakis and Boutilier, 2003] Georgios Chalkiadakis and Craig Boutilier. Coordination in multiagent reinforcement learning: A bayesian approach. In AAMAS, pages 709-716, New York, NY, USA, 2003. ACM.","title":"Coordination in multiagent reinforcement learning: A bayesian approach","authors":"Boutilier Chalkiadakis, Georgios Chalkiadakis, Craig Boutilier","year":"2003","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Claus and Boutilier, 1998] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI '98/IAAI '98, Menlo Park, CA, USA, 1998. American Association for Artificial Intelligence.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"Claus, Boutilier, Caroline Claus, Craig Boutilier","year":"1998","venue":"AAAI '98/IAAI '98","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Florensa et al., 2017] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic Neural Networks for Hierarchical Reinforce- ment Learning. arXiv e-prints, April 2017.","title":"Stochastic Neural Networks for Hierarchical Reinforcement Learning","authors":"Carlos Florensa, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Foerster et al., 2017] Jakob Foerster, Gregory Farquhar, Tri- antafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual Multi-Agent Policy Gradients. arXiv e-prints, page arXiv:1705.08926, May 2017.","title":"Counterfactual Multi-Agent Policy Gradients","authors":"Foerster","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fox et al., 2015] Roy Fox, Ari Pakman, and Naftali Tishby. Tam- ing the Noise in Reinforcement Learning via Soft Updates. arXiv e-prints, page arXiv:1512.08562, December 2015.","title":"Taming the Noise in Reinforcement Learning via Soft Updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grau-Moya et al., 2018] Jordi Grau-Moya, Felix Leibfried, and Haitham Bou-Ammar. Balancing Two-Player Stochastic Games with Soft Q-Learning. arXiv e-prints, page arXiv:1802.03216, February 2018.","title":"Balancing Two-Player Stochastic Games with Soft Q-Learning","authors":"Grau-Moya","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/37"},{"position":10,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. CoRR, abs/1702.08165, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haarnoja et al., 2018a] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Com- posable Deep Reinforcement Learning for Robotic Manipulation. arXiv e-prints, page arXiv:1803.06773, March 2018.","title":"Composable Deep Reinforcement Learning for Robotic Manipulation","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haarnoja et al., 2018b] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Max- imum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv e-prints, page arXiv:1801.01290, January 2018.","title":"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learn- ing. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Re- search, pages 1804-1813, New York, New York, USA, 20-22 Jun 2016. PMLR. Heinrich and Silver, 2016] Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. CoRR, abs/1603.01121, 2016.","title":"Deep reinforcement learning from self-play in imperfectinformation games","authors":"He","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kalman, 1960] Rudolf Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME -Journal of basic Engineering, 82:35-45, 01 1960.","title":"A new approach to linear filtering and prediction problems","authors":"Rudolf Kalman","year":"1960","venue":"Transactions of the ASME -Journal of basic Engineering","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kapetanakis and Kudenko, 2002] Spiros Kapetanakis and Daniel Kudenko. Reinforcement learning of coordination in coopera- tive multi-agent systems. In Eighteenth National Conference on Artificial Intelligence, Menlo Park, CA, USA, 2002.","title":"Reinforcement learning of coordination in cooperative multi-agent systems","authors":"Kudenko Kapetanakis, Spiros Kapetanakis, Daniel Kudenko","year":"2002","venue":"Eighteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kappen, 2005] Hilbert J. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Me- chanics: Theory and Experiment, 2005(11), 2005.","title":"Path integrals and symmetry breaking for optimal control theory","authors":"Hilbert J Kappen","year":"2005","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levine and Koltun, 2013] Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Wein- berger, editors, NIPS '13. 2013.","title":"Variational policy search via trajectory optimization","authors":"Koltun Levine, Sergey Levine, Vladlen Koltun","year":"2013","venue":"NIPS '13","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levine, 2018] Sergey Levine. Reinforcement Learning and Con- trol as Probabilistic Inference: Tutorial and Review. arXiv e- prints, page arXiv:1805.00909, May 2018.","title":"Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review","authors":"Sergey Levine, Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. arXiv e-prints, page arXiv:1706.02275, June 2017.","title":"Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nachum et al., 2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the Gap Between Value and Policy Based Reinforcement Learning. arXiv e-prints, page arXiv:1702.08892, February 2017.","title":"Bridging the Gap Between Value and Policy Based Reinforcement Learning","authors":"Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rabinowitz et al., 2018] Neil C Rabinowitz, Frank Perbet, H Fran- cis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv:1802.07740, 2018.","title":"Machine theory of mind","authors":"Frank Neil C Rabinowitz, Francis Perbet, Chiyuan Song, S M Zhang, Matthew Eslami, Botvinick","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Raileanu et al., 2018] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi- agent reinforcement learning. CoRR, abs/1802.09640, 2018.","title":"Modeling others using oneself in multiagent reinforcement learning","authors":"Roberta Raileanu, Emily Denton, Arthur Szlam, Rob Fergus","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv e-prints, page arXiv:1803.11485, March 2018.","title":"QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rawlik et al., 2013] Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference (extended abstract). IJCAI '13, pages 3052-3056, 2013.","title":"On stochastic optimal control and reinforcement learning by approximate inference","authors":"Konrad Rawlik, Marc Toussaint, Sethu Vijayakumar","year":"2013","venue":"IJCAI '13","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2017] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence Between Policy Gradients and Soft Q- Learning. arXiv e-prints, page arXiv:1704.06440, April 2017.","title":"Equivalence Between Policy Gradients and Soft Q-Learning","authors":"John Schulman, Xi Chen, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Theodorou et al., 2010] Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to re- inforcement learning. J. Mach. Learn. Res., 11, December 2010.","title":"A generalized path integral control approach to reinforcement learning","authors":"Evangelos Theodorou, Jonas Buchli, Stefan Schaal","year":"2010","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tian et al., 2018] Zheng Tian, Shihao Zou, Tim Warr, Lisheng Wu, and Jun Wang. Learning Multi-agent Implicit Communi- cation Through Actions: A Case Study in Contract Bridge, a Collaborative Imperfect-Information Game. arXiv e-prints, page arXiv:1810.04444, October 2018.","title":"Learning Multi-agent Implicit Communication Through Actions: A Case Study in Contract Bridge, a Collaborative Imperfect-Information Game","authors":"Zheng Tian, Shihao Zou, Tim Warr, Lisheng Wu, Jun Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Todorov, 2007] Emanuel Todorov. Linearly-solvable markov de- cision problems. In B. Schölkopf, J. C. Platt, and T. Hoffman, editors, NIPS '19. 2007.","title":"Linearly-solvable markov decision problems","authors":"Emanuel Todorov","year":"2007","venue":"NIPS '19","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Todorov, 2010] Emanuel Todorov. Policy gradients in linearly- solvable mdps. In J. D. Lafferty, C. K. I. Williams, J. Shawe- Taylor, R. S. Zemel, and A. Culotta, editors, NIPS. 2010.","title":"Policy gradients in linearlysolvable mdps","authors":"Emanuel Todorov","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Toussaint and Storkey, 2006] Marc Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state markov decision processes. ICML '06, pages 945-952, New York, NY, USA, 2006. ACM.","title":"Probabilistic inference for solving discrete and continuous state markov decision processes","authors":"Storkey Toussaint, Marc Toussaint, Amos Storkey","year":"2006","venue":"ICML '06","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Toussaint, 2009] Marc Toussaint. Robot trajectory optimization using approximate inference. ICML '09, pages 1049-1056, New York, NY, USA, 2009. ACM.","title":"Robot trajectory optimization using approximate inference","authors":"Marc Toussaint","year":"2009","venue":"ICML '09","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wei et al., 2018] Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. AAAI, 2018.","title":"Multiagent soft q-learning","authors":"Ermo Wei, Drew Wicke, David Freelan, Sean Luke","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wen et al., 2019] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In ICLR, 2019.","title":"Probabilistic recursive reasoning for multi-agent reinforcement learning","authors":"Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, Wei Pan","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wilson et al., 2010] Aaron Wilson, Alan Fern, and Prasad Tade- palli. Bayesian policy search for multi-agent role discovery. AAAI'10, pages 624-629. AAAI Press, 2010.","title":"Bayesian policy search for multi-agent role discovery","authors":"Aaron Wilson, Alan Fern, Prasad Tadepalli","year":"2010","venue":"AAAI'10","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ziebart et al., 2008] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforce- ment learning. In AAAI '08, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI '08","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}