{"type":"ScholarlyArticle","title":"Balancing Individual Preferences and Shared Objectives in Multiagent Reinforcement Learning","authors":[{"name":"Ishan Durugkar","slug":"ishan-durugkar","given":"Ishan","family":"Durugkar","affiliation":"University of Texas at Austin"},{"name":"Elad Liebman","slug":"elad-liebman","given":"Elad","family":"Liebman","affiliation":"SparkCognition Research"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"University of Texas at Austin; Sony AI"}],"abstract":"In multiagent reinforcement learning scenarios, it is often the case that independent agents must jointly learn to perform a cooperative task. This paper focuses on such a scenario in which agents have individual preferences regarding how to accomplish the shared task. We consider a framework for this setting which balances individual preferences against task rewards using a linear mixing scheme.  In our theoretical analysis we establish that agents can reach an equilibrium that leads to optimal shared task reward even when they consider individual preferences which aren't fully aligned with this task. We then empirically show, somewhat  counter-intuitively, that there exist mixing schemes that outperform a purely task-oriented baseline.  We further consider empirically how to  optimize the mixing scheme.","doi":"10.24963/ijcai.2020/347","url":"https://ijcai.org/proceedings/2020/347","pdf":"https://ijcai.org/proceedings/2020/0347.pdf","markdown":"https://ijcai.org/proceedings/2020/347.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/347","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":347,"pages":{"first":2505,"last":2511},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26894/file_26894.m3u8","length_sec":934,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the Twenty-first International Con- ference on Machine Learning, ICML '04, pages 1-, New York, NY, USA, 2004. ACM.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the Twenty-first International Conference on Machine Learning, ICML '04","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ammar et al., 2014] Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In International Conference on Machine Learning, pages 1206-1214, 2014.","title":"Online multi-task learning for policy gradient methods","authors":"Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, Matthew Taylor","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aumann, 1961] Robert J Aumann. The core of a cooper- ative game without side payments. Transactions of the American Mathematical Society, 98(3):539-552, 1961.","title":"The core of a cooperative game without side payments","authors":"Robert J Aumann","year":"1961","venue":"Transactions of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barrett et al., 2013] Samuel Barrett, Peter Stone, Sarit Kraus, and Avi Rosenfeld. Teamwork with limited knowl- edge of teammates. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI), July 2013.","title":"Teamwork with limited knowledge of teammates","authors":"Barrett","year":"2013","venue":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barto et al., 2004] Andrew G Barto, Satinder Singh, and Nuttapong Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages 112-19, 2004.","title":"Intrinsically motivated learning of hierarchical collections of skills","authors":"Satinder Andrew G Barto, Nuttapong Singh, Chentanez","year":"2004","venue":"Proceedings of the 3rd International Conference on Development and Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barto, 2013] Andrew G Barto. Intrinsic motivation and re- inforcement learning. In Intrinsically motivated learning in natural and artificial systems, pages 17-47. Springer, 2013.","title":"Intrinsic motivation and reinforcement learning","authors":"G Andrew, Barto","year":"2013","venue":"Intrinsically motivated learning in natural and artificial systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Busoniu et al., 2008] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews, 38 (2), 2008, 2008.","title":"A comprehensive survey of multiagent reinforcement learning","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter","year":"2008","venue":"IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Colman et al., 2008] Andrew M Colman, Briony D Pulford, and Jo Rose. Team reasoning and collective rational- ity: Piercing the veil of obviousness. Acta psychologica, 128(2):409-412, 2008.","title":"Team reasoning and collective rationality: Piercing the veil of obviousness","authors":"Andrew M Colman, Jo Briony D Pulford, Rose","year":"2008","venue":"Acta psychologica","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cook, 1994] Nicholas Cook. A guide to musical analysis. Oxford University Press, 1994.","title":"A guide to musical analysis","authors":"Nicholas Cook","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Foerster et al., 2017] Jakob Foerster, Nantas Nardelli, Gre- gory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising expe- rience replay for deep multi-agent reinforcement learning. arXiv preprint arXiv:1702.08887, 2017.","title":"Stabilising experience replay for deep multi-agent reinforcement learning","authors":"Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Pushmeet Philip Hs Torr, Shimon Kohli, Whiteson","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Foerster et al., 2018] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750-797, 2019.","title":"A survey and critique of multiagent deep reinforcement learning","authors":"Hernandez-Leal","year":"2019","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems 29, pages 4565-4573.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":null,"venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaderberg et al., 2019] Max Jaderberg, Wojciech M Czar- necki, Iain Dunning, Luke Marris, Guy Lever, Anto- nio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):859- 865, 2019.","title":"Human-level performance in 3d multiplayer games with populationbased reinforcement learning","authors":"Jaderberg","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kang et al., 2018] Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In Interna- tional Conference on Machine Learning, pages 2474- 2483, 2018.","title":"Policy optimization with demonstrations","authors":"Kang","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Leyton-Brown and Shoham, 2008] Kevin Leyton-Brown and Yoav Shoham. Essentials of game theory: A concise multidisciplinary introduction. Synthesis lectures on artificial intelligence and machine learning, 2(1):1-88, 2008.","title":"Essentials of game theory: A concise multidisciplinary introduction","authors":"Leyton-Brown, Shoham, Kevin Leyton-Brown, Yoav Shoham","year":"2008","venue":"Synthesis lectures on artificial intelligence and machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Narvekar et al., 2020] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning do- mains: A framework and survey, 2020.","title":"Curriculum learning for reinforcement learning domains: A framework and survey","authors":"Narvekar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Osborne and Rubinstein, 1994] Martin J Osborne and Ariel Rubinstein. A course in game theory. MIT press, 1994.","title":"A course in game theory","authors":"J Martin, Ariel Osborne, Rubinstein","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peterson, 2009] Elisha Peterson. Cooperation in subset team games: altruism and selfishness. arXiv preprint arXiv:0907.2376, 2009.","title":"Elisha Peterson. Cooperation in subset team games: altruism and selfishness","authors":"Peterson","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rȃdulescu et al., 2020] Roxana Rȃdulescu, Patrick Man- nion, Diederik M Roijers, and Ann Nowé. Multi-objective multi-agent decision making: a utility-based analysis and survey. Autonomous Agents and Multi-Agent Systems, 34(1):10, 2020.","title":"Multi-objective multi-agent decision making: a utility-based analysis and survey","authors":"Roxana Rȃdulescu, Patrick Mannion, Diederik M Roijers, Ann Nowé","year":"2020","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485, 2018.","title":"Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De, Gregory Witt, Jakob Farquhar, Shimon Foerster, Whiteson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sequeira et al., 2011] Pedro Sequeira, Francisco S Melo, Rui Prada, and Ana Paiva. Emerging social awareness: Exploring intrinsic motivation in multiagent learning. In 2011 IEEE International Conference on Development and Learning (ICDL), volume 2, pages 1-6. IEEE, 2011.","title":"Emerging social awareness: Exploring intrinsic motivation in multiagent learning","authors":"Pedro Sequeira, Rui Francisco S Melo, Ana Prada, Paiva","year":"2011","venue":"IEEE International Conference on Development and Learning (ICDL)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Snoek et al., 2012] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of ma- chine learning algorithms. In Advances in neural informa- tion processing systems, pages 2951-2959, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Snoek","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Stone et al., 2010] Peter Stone, Gal A Kaminka, Sarit Kraus, Jeffrey S Rosenschein, et al. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In AAAI, 2010.","title":"Ad hoc autonomous agent teams: Collaboration without pre-coordination","authors":"Peter Stone, A Gal, Sarit Kaminka, Kraus, Jeffrey S Rosenschein","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sugden, 2008] Robert Sugden. Nash equilibrium, team rea- soning and cognitive hierarchy theory. Acta Psychologica, 128(2):402-404, 2008.","title":"Nash equilibrium, team reasoning and cognitive hierarchy theory","authors":"Robert Sugden","year":"2008","venue":"Acta Psychologica","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tampuu et al., 2017] Ardi Tampuu, Tambet Matiisen, Do- rian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.","title":"Multiagent cooperation and competition with deep reinforcement learning","authors":"Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente","year":"2017","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wilson et al., 2007] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learn- ing: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015-1022. ACM, 2007.","title":"Multi-task reinforcement learning: a hierarchical bayesian approach","authors":"Aaron Wilson, Alan Fern, Soumya Ray, Prasad Tadepalli","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2020] Jiachen Yang, Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha. Cm3: Coop- erative multi-goal multi-stage multi-agent reinforcement learning. In International Conference on Learning Rep- resentations, 2020.","title":"Cm3: Cooperative multi-goal multi-stage multi-agent reinforcement learning","authors":"Yang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2018] Yuke Zhu, Ziyu Wang, Josh Merel, An- drei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/1802.09564, 2018.","title":"Reinforcement and imitation learning for diverse visuomotor skills","authors":"Yuke Zhu, Ziyu Wang, Josh Merel, Andrei A Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando De Freitas, Nicolas Heess","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}