{"type":"ScholarlyArticle","title":"Towards Efficient Detection and Optimal Response against Sophisticated Opponents","authors":[{"name":"Tianpei Yang","slug":"tianpei-yang","given":"Tianpei","family":"Yang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zhaopeng Meng","slug":"zhaopeng-meng","given":"Zhaopeng","family":"Meng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Chongjie Zhang","slug":"chongjie-zhang","given":"Chongjie","family":"Zhang","affiliation":"MMW , Tsinghua University"},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Ze Zheng","slug":"ze-zheng","given":"Ze","family":"Zheng","affiliation":"Beifang Investigation, Design & Research CO.LTD"}],"abstract":"Multiagent algorithms often aim to accurately predict the behaviors of other agents and find a best response accordingly. Previous works usually assume an opponent uses a stationary strategy or randomly switches among several stationary ones. However, an opponent may exhibit more sophisticated behaviors by adopting more advanced reasoning strategies, e.g., using a Bayesian reasoning strategy. This paper proposes a novel approach called Bayes-ToMoP which can efficiently detect the strategy of opponents using either stationary or higher-level reasoning strategies. Bayes-ToMoP also supports the detection of previously unseen policies and learning a best-response policy accordingly. We provide a theoretical guarantee of the optimality on detecting the opponent's strategies. We also propose a deep version of Bayes-ToMoP by extending Bayes-ToMoP with DRL techniques. Experimental results show both Bayes-ToMoP and deep Bayes-ToMoP outperform the state-of-the-art approaches when faced with different types of opponents in two-agent competitive games.","doi":"10.24963/ijcai.2019/88","url":"https://ijcai.org/proceedings/2019/88","pdf":"https://ijcai.org/proceedings/2019/0088.pdf","markdown":"https://ijcai.org/proceedings/2019/88.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/88","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":88,"pages":{"first":623,"last":629},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agent Theories and Models"],"videos":[],"references":[{"position":1,"raw":"[Albrecht and Stone, 2018] Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial In- telligence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"Stefano V Albrecht, Peter Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2011] Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Bayesian theory of mind: Mod- eling joint belief-desire attribution. In Proceedings of An- nual Meeting of the Cognitive Science Society, 2011.","title":"Bayesian theory of mind: Modeling joint belief-desire attribution","authors":"Chris L Baker, Rebecca Saxe, Joshua B Tenenbaum","year":"2011","venue":"Proceedings of Annual Meeting of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning. Jour- nal of Machine Learning Research, 3:213-231, 2002.","title":"R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":4,"raw":"[Crandall, 2012] Jacob W. Crandall. Just add pepper: ex- tending learning algorithms for repeated matrix games to repeated markov games. In Proceedings of International Conference on Autonomous Agents and Multiagent Sys- tems, pages 399-406, 2012.","title":"Just add pepper: extending learning algorithms for repeated matrix games to repeated markov games","authors":"Jacob W Crandall","year":"2012","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[de Weerd et al., 2013] Harmen de Weerd, Rineke Ver- brugge, and Bart Verheij. How much does it help to know what she knows you know? an agent-based simulation study. Artificial Intelligence, 199:67-92, 2013.","title":"Harmen de Weerd, Rineke Verbrugge, and Bart Verheij. How much does it help to know what she knows you know? an agent-based simulation study","authors":"De Weerd","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fernández et al., 2010] Fernando Fernández, Javier García, and Manuela M. Veloso. Probabilistic policy reuse for inter-task transfer learning. Robotics and Autonomous Sys- tems, 58(7):866-871, 2010.","title":"Probabilistic policy reuse for inter-task transfer learning","authors":"Fernando Fernández, Javier García, Manuela M Veloso","year":"2010","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Foerster et al., 2018] Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of International Conference on Autonomous Agents and MultiAgent Systems, pages 122- 130, 2018.","title":"Learning with opponent-learning awareness","authors":"Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch","year":"2018","venue":"Proceedings of International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gmytrasiewicz and Doshi, 2005] Piotr J. Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:49-79, 2005.","title":"A framework for sequential planning in multi-agent settings","authors":"Piotr J Gmytrasiewicz, Prashant Doshi","year":"2005","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodrich et al., 2003] Michael A. Goodrich, Jacob W. Crandall, and Jeffrey L. Stimpson. Neglect tolerant team- ing: Issues and dilemmas. In AAAI Spring Sympo. on Hu- man Interaction with Autonomous Systems in Complex En- vironments, 2003.","title":"Neglect tolerant teaming: Issues and dilemmas","authors":"A Michael, Jacob W Goodrich, Jeffrey L Crandall, Stimpson","year":"2003","venue":"AAAI Spring Sympo. on Human Interaction with Autonomous Systems in Complex Environments","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He and Boyd-Graber, 2016] He He and Jordan L. Boyd- Graber. Opponent modeling in deep reinforcement learn- ing. In Proceedings of International Conference on Ma- chine Learning, pages 1804-1813, 2016.","title":"Jordan L. Boyd-Graber. Opponent modeling in deep reinforcement learning","authors":"Boyd-Graber ; He He, He","year":"2016","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hernandez-Leal and Kaisers, 2017] Pablo Hernandez-Leal and Michael Kaisers. Towards a fast detection of oppo- nents in repeated stochastic games. In Proceedings of In- ternational Conference on Autonomous Agents and Multi- agent Systems, pages 239-257, 2017.","title":"Towards a fast detection of opponents in repeated stochastic games","authors":"Kaisers Hernandez-Leal, Pablo Hernandez-Leal, Michael Kaisers","year":"2017","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hernandez-Leal et al., 2016] Pablo Hernandez-Leal, Matthew E. Taylor, Benjamin Rosman, Luis Enrique Sucar, and Enrique Munoz de Cote. Identifying and tracking switching, non-stationary opponents: A bayesian approach. In Multiagent Interaction without Prior Coordination, Papers from the 2016 AAAI Workshop, 2016.","title":"Identifying and tracking switching, non-stationary opponents: A bayesian approach","authors":"Hernandez-Leal","year":"2016","venue":"Multiagent Interaction without Prior Coordination","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hernandez-Leal et al., 2017a] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. CoRR, abs/1707.09183, 2017.","title":"A survey of learning in multiagent environments: Dealing with non-stationarity","authors":"Hernandez-Leal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hernandez-Leal et al., 2017b] Pablo Hernandez-Leal, Yusen Zhan, Matthew E. Taylor, Luis Enrique Sucar, and Enrique Munoz de Cote. Efficiently detecting switches against non-stationary opponents. Autonomous Agents and Multi-Agent Systems, 31(4):767-789, 2017.","title":"Efficiently detecting switches against non-stationary opponents","authors":"Hernandez-Leal","year":"2017","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Littman, 1994] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings of International Conference on Machine Learn- ing, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"L Michael, Littman","year":"1994","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Powers and Shoham, 2005] Rob Powers and Yoav Shoham. Learning against opponents with bounded memory. In Proceedings of International Joint Conference on Artifi- cial Intelligence, pages 817-822, 2005.","title":"Rob Powers and Yoav Shoham. Learning against opponents with bounded memory","authors":"Shoham Powers","year":"2005","venue":"Proceedings of International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rosman et al., 2016] Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy. Bayesian policy reuse. Machine Learning, 104(1):99-127, 2016.","title":"Bayesian policy reuse","authors":"Benjamin Rosman, Majd Hawasly, Subramanian Ramamoorthy","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shoham and Leyton-Brown, 2009] Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems -Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, 2009.","title":"Multiagent Systems -Algorithmic, Game-Theoretic, and Logical Foundations","authors":"Yoav Shoham, Kevin Leyton-Brown","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wunder et al., 2012] Michael Wunder, John Robert Yaros, Michael Kaisers, and Michael L. Littman. A framework for modeling population strategies by depth of reason- ing. In Proceedings of International Conference on Au- tonomous Agents and Multiagent Systems-Volume 2, pages 947-954, 2012.","title":"A framework for modeling population strategies by depth of reasoning","authors":"Michael Wunder, John Robert Yaros, Michael Kaisers, Michael L Littman","year":"2012","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2019] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Yan Zheng, Chongjie Zhang, and Ze Zheng. Bayes- tomop: A fast detection and best response algorithm to- wards sophisticated opponents. In Proceedings of Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 2282-2284, 2019.","title":"Bayestomop: A fast detection and best response algorithm towards sophisticated opponents","authors":"Yang","year":"2019","venue":"Proceedings of International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zheng et al., 2018] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non- stationary agents. In Advances in Neural Information Pro- cessing Systems, pages 954-964, 2018.","title":"A deep bayesian policy reuse approach against nonstationary agents","authors":"Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, Changjie Fan","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}