{"type":"ScholarlyArticle","title":"The Minds of Many: Opponent Modeling in a Stochastic Game","authors":[{"name":"Friedrich Burkhard von der Osten","slug":"friedrich-burkhard-von-der-osten","given":"Friedrich Burkhard","family":"von der Osten","affiliation":"School of Computing and Information Systems The University of Melbourne , Australia"},{"name":"Michael Kirley","slug":"michael-kirley","given":"Michael","family":"Kirley","affiliation":"School of Computing and Information Systems The University of Melbourne , Australia"},{"name":"Tim Miller","slug":"tim-miller","given":"Tim","family":"Miller","affiliation":"School of Computing and Information Systems The University of Melbourne , Australia"}],"abstract":"The Theory of Mind provides a framework for an agent to predict the actions of adversaries by building an abstract model of their strategies using recursive nested beliefs. In this paper, we extend a recently introduced technique for opponent modeling based on Theory of Mind reasoning. Our extended multi-agent Theory of Mind model explicitly considers multiple opponents simultaneously.  We introduce a stereotyping mechanism, which segments the agent population into sub-groups of agents with similar behavior.  Here, sub-group profiles guide decision making in place of individual agent profiles. We evaluate our model using a multi-player stochastic game, which presents agents with the challenge of unknown adversaries in a partially-observable environment. Simulation results demonstrate that the model performs well under uncertainty and that stereotyping allows larger groups of agents to be modeled robustly. The findings strengthen results showing that Theory of Mind modeling is useful in many artificial intelligence applications.","doi":"10.24963/ijcai.2017/537","url":"https://ijcai.org/proceedings/2017/537","pdf":"https://ijcai.org/proceedings/2017/0537.pdf","markdown":"https://ijcai.org/proceedings/2017/537.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/537","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":537,"pages":{"first":3845,"last":3851},"keywords":["Multidisciplinary Topics and Applications: Cognitive Modeling","Multidisciplinary Topics and Applications: AI and Social Sciences"],"videos":[],"references":[{"position":1,"raw":"[Albrecht et al., 2016] Stefano V Albrecht, Jacob W Cran- dall, and Subramanian Ramamoorthy. Belief and truth in hypothesised behaviours. Artificial Intelligence, 235:63- 94, 2016.","title":"Belief and truth in hypothesised behaviours","authors":"Jacob W Stefano V Albrecht, Subramanian Crandall, Ramamoorthy","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brennan and Lo, 2012] Thomas J Brennan and Andrew W Lo. An evolutionary model of bounded rationality and in- telligence. PloS One, 7(11):e50310, 2012.","title":"An evolutionary model of bounded rationality and intelligence","authors":"J Thomas, Andrew W Brennan, Lo","year":"2012","venue":"PloS One","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Camerer et al., 2015] Colin F Camerer, Teck-Hua Ho, and Juin Kuan Chong. A psychological approach to strategic thinking in games. Current Opinion in Behavioral Sci- ences, 3:157-162, 2015.","title":"A psychological approach to strategic thinking in games","authors":"Colin F Camerer, Teck-Hua Ho, Juin Kuan, Chong","year":"2015","venue":"Current Opinion in Behavioral Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cavalcanti et al., 2010] Carina Cavalcanti, Felix Schläpfer, and Bernhard Schmid. Public participation and willing- ness to cooperate in common-pool resource management. Ecological Economics, 69(3):613-622, 2010.","title":"Public participation and willingness to cooperate in common-pool resource management","authors":"Carina Cavalcanti, Felix Schläpfer, Bernhard Schmid","year":"2010","venue":"Ecological Economics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Condon, 1992] Anne Condon. The complexity of stochastic games. Inf. and Comp., 96(2):203-224, 1992.","title":"The complexity of stochastic games","authors":"Anne Condon","year":"1992","venue":"Inf. and Comp","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Weerd et al., 2013a] Harmen De Weerd, Rineke Ver- brugge, and Bart Verheij. Higher-order theory of mind in negotiations under incomplete information. In PRIMA, pages 101-116. Springer, 2013.","title":"Higher-order theory of mind in negotiations under incomplete information","authors":"De Weerd","year":"2013","venue":"PRIMA","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[De Weerd et al., 2013b] Harmen De Weerd, Rineke Ver- brugge, and Bart Verheij. How much does it help to know what she knows you know? an agent-based simulation study. Artificial Intelligence, 199:67-92, 2013.","title":"How much does it help to know what she knows you know? an agent-based simulation study","authors":"De Weerd","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[De Weerd et al., 2015] Harmen De Weerd, Rineke Ver- brugge, and Bart Verheij. Higher-order theory of mind in the tacit communication game. Biologically Inspired Cognitive Architectures, 11:10-21, 2015.","title":"Higher-order theory of mind in the tacit communication game","authors":"De Weerd","year":"2015","venue":"Biologically Inspired Cognitive Architectures","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devaine et al., 2014] Marie Devaine, Guillaume Hollard, and Jean Daunizeau. Theory of mind: did evolution fool us? PloS One, 9(2):e87619, 2014.","title":"Theory of mind: did evolution fool us?","authors":"Marie Devaine, Guillaume Hollard, Jean Daunizeau","year":"2014","venue":"PloS One","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Felli et al., 2015] Paolo Felli, Tim Miller, Christian Muise, Adrian R. Pearce, and Liz Sonenberg. Computing social behaviours using agent models. In IJCAI 2015, 2015.","title":"Computing social behaviours using agent models","authors":"Paolo Felli, Tim Miller, Christian Muise, Adrian R Pearce, Liz Sonenberg","year":"2015","venue":"IJCAI 2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fiske, 2000] Susan T Fiske. Stereotyping, prejudice, and discrimination at the seam between the centuries: Evolu- tion, culture, mind, and brain. European Journal of Social Psychology, 30(3):299-322, 2000.","title":"Stereotyping, prejudice, and discrimination at the seam between the centuries: Evolution, culture, mind, and brain","authors":"Susan T Fiske","year":"2000","venue":"European Journal of Social Psychology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ganzfried and Sandholm, 2011] Sam Ganzfried and Tuo- mas Sandholm. Game theory-based opponent modeling in large imperfect-information games. In AAMAS, volume 2, pages 533-540, 2011.","title":"Game theory-based opponent modeling in large imperfect-information games","authors":"Sandholm Ganzfried","year":"2011","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gmytrasiewicz and Doshi, 2005] Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:49-79, 2005.","title":"A framework for sequential planning in multi-agent settings","authors":"J Piotr, Prashant Gmytrasiewicz, Doshi","year":"2005","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goldman, 2012] Alvin I Goldman. Theory of mind. The Oxford handbook of philosophy of cognitive science, pages 402-424, 2012.","title":"Theory of mind. The Oxford handbook of philosophy of cognitive science","authors":"Alvin I Goldman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kashima et al., 2000] Yoshihisa Kashima, Jodie Woolcock, and Emiko S Kashima. Group impressions as dynamic configurations: The tensor product model of group im- pression formation and change. Psychological Review, 107(4):914, 2000.","title":"Group impressions as dynamic configurations: The tensor product model of group impression formation and change","authors":"Yoshihisa Kashima, Jodie Woolcock, Emiko S Kashima","year":"2000","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kimbrough et al., 2014] Erik O Kimbrough, Nikolaus Robalino, and Arthur J Robson. The evolution of 'theory of mind': Theory and experiments, 2014.","title":"The evolution of 'theory of mind","authors":"Erik O Kimbrough, Nikolaus Robalino, Arthur J Robson","year":"2014","venue":"Theory and experiments","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mertens and Neyman, 1981] J-F Mertens and Abraham Neyman. Stochastic games. International Journal of Game Theory, 10(2):53-66, 1981.","title":"Stochastic games","authors":"Neyman ; F Mertens, Abraham Mertens, Neyman","year":"1981","venue":"International Journal of Game Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moll and Tomasello, 2007] Henrike Moll and Michael Tomasello. Cooperation and human cognition: the vygot- skian intelligence hypothesis. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 362(1480):639-648, 2007.","title":"Cooperation and human cognition: the vygotskian intelligence hypothesis","authors":"Tomasello ; Moll, Henrike Moll, Michael Tomasello","year":"1480","venue":"Philosophical Transactions of the Royal Society of London B: Biological Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oakes et al., 1994] Penelope J Oakes, S Alexander Haslam, and John C Turner. Stereotyping and social reality. Black- well Publishing, 1994.","title":"Stereotyping and social reality","authors":"Penelope J Oakes, S Alexander Haslam, John C Turner","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pereira et al., 2011] Luís Moniz Pereira, Francisco C San- tos, et al. Intention recognition promotes the emergence of cooperation. Adaptive Behavior, 19(4):264-279, 2011.","title":"Intention recognition promotes the emergence of cooperation","authors":"Luís Moniz Pereira, Francisco C Santos","year":"2011","venue":"Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pereira et al., 2012] Luís Moniz Pereira, Francisco C San- tos, et al. Corpus-based intention recognition in coopera- tion dilemmas. Artificial Life, 18(4):365-383, 2012.","title":"Corpus-based intention recognition in cooperation dilemmas","authors":"Luís Moniz Pereira, Francisco C Santos","year":"2012","venue":"Artificial Life","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perner, 1999] Josef Perner. Theory of mind. Dev. psych.: Achievements and prospects, pages 205-230, 1999.","title":"Theory of mind","authors":"Perner","year":"1999","venue":"Achievements and prospects","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pynadath and Marsella, 2005] David V Pynadath and Stacy C Marsella. PsychSim: Modeling theory of mind with decision-theoretic agents. In IJCAI, volume 5, pages 1181-1186, 2005.","title":"PsychSim: Modeling theory of mind with decision-theoretic agents","authors":"V David, Stacy C Pynadath, Marsella","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Robalino and Robson, 2012] Nikolaus Robalino and Arthur Robson. The economic approach to theory of mind. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 367(1599):2224-2233, 2012.","title":"The economic approach to theory of mind","authors":"Robson Robalino, Nikolaus Robalino, Arthur Robson","year":"1599","venue":"Philosophical Transactions of the Royal Society B: Biological Sciences","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sethi and Somanathan, 1996] Rajiv Sethi and Eswaran So- manathan. The evolution of social norms in common prop- erty resource use. The American Economic Review, pages 766-788, 1996.","title":"The evolution of social norms in common property resource use","authors":"Rajiv Sethi, Eswaran Somanathan","year":"1996","venue":"The American Economic Review","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[von der Osten et al., 2017] Friedrich Burkhard von der Os- ten, Michael Kirley, and Tim Miller. Sustainability is pos- sible despite greed -exploring the nexus between prof- itability and sustainability in common pool resource sys- tems. Scientific Reports, 7(1):2307, 2017.","title":"Sustainability is possible despite greed -exploring the nexus between profitability and sustainability in common pool resource systems","authors":"Von Der Osten","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yoshida et al., 2008] Wako Yoshida, Ray J Dolan, and Karl J Friston. Game theory of mind. PLoS Comput Biol, 4(12):e1000254, 2008.","title":"Game theory of mind","authors":"Wako Yoshida, Ray J Dolan, Karl J Friston","year":"2008","venue":"PLoS Comput Biol","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}