{"type":"ScholarlyArticle","title":"Multiagent Decision Making and Learning in Urban Environments","authors":[{"name":"Akshat Kumar","slug":"akshat-kumar","given":"Akshat","family":"Kumar","affiliation":"School of Information Systems , Singapore Management University"}],"abstract":"Our increasingly interconnected urban environments provide several opportunities to deploy intelligent agents---from self-driving cars, ships to aerial drones---that promise to radically improve productivity and safety. Achieving coordination among agents in such urban settings presents several algorithmic challenges---ability to scale to thousands of agents, addressing uncertainty, and partial observability in the environment. In addition, accurate domain models need to be learned from data that is often noisy and available only at an aggregate level. In this paper, I will overview some of our recent contributions towards developing planning and reinforcement learning strategies to address several such challenges present in large-scale urban multiagent systems.","doi":"10.24963/ijcai.2019/895","url":"https://ijcai.org/proceedings/2019/895","pdf":"https://ijcai.org/proceedings/2019/0895.pdf","markdown":"https://ijcai.org/proceedings/2019/895.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/895","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Early Career","track":"Early Career","number":895,"pages":{"first":6398,"last":6402},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making","Agent-based and Multi-agent Systems: Multi-agent Planning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[],"references":[{"position":1,"raw":"[Agussurja et al., 2018] Lucas Agussurja, Akshat Kumar, and Hoong Chuin Lau. Resource-constrained scheduling for mar- itime traffic management. In AAAI Conference on Artificial In- telligence, pages 6086-6093, 2018.","title":"Resource-constrained scheduling for maritime traffic management","authors":"Lucas Agussurja, Akshat Kumar, Hoong Chuin Lau","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amato et al., 2010] Christopher Amato, Daniel S. Bernstein, and Shlomo Zilberstein. Optimizing fixed-size stochastic controllers for pomdps and decentralized pomdps. Autonomous Agents and Multi-Agent Systems, 21(3):293-320, 2010.","title":"Optimizing fixed-size stochastic controllers for pomdps and decentralized pomdps","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2010","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Au et al., 2016] Tsz-Chiu Au, Shun Zhang, and Peter Stone. Au- tonomous intersection management for semi-autonomous vehi- cles. In Dusan Teodorovi'c, editor, Handbook of Transportation, pages 88-104. Routledge, 2016.","title":"Autonomous intersection management for semi-autonomous vehicles","authors":"Tsz-Chiu Au, Shun Zhang, Peter Stone","year":"2016","venue":"Handbook of Transportation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Becker et al., 2004] Raphen Becker, Shlomo Zilberstein, and Vic- tor Lesser. Decentralized Markov decision processes with event- driven interactions. In International Conference on Autonomous Agents and Multiagent Systems, pages 302-309, 2004.","title":"Raphen Becker, Shlomo Zilberstein, and Victor Lesser. Decentralized Markov decision processes with eventdriven interactions","authors":"Becker","year":"2004","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bernstein et al., 2002] Daniel S. Bernstein, Rob Givan, Neil Im- merman, and Shlomo Zilberstein. The complexity of decentral- ized control of Markov decision processes. Mathematics of Op- erations Research, 27:819-840, 2002.","title":"The complexity of decentralized control of Markov decision processes","authors":"S Daniel, Rob Bernstein, Neil Givan, Shlomo Immerman, Zilberstein","year":"2002","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Daganzo, 1994] C.F. Daganzo. The cell transmission model: A dynamic representation of highway traffic consistent with the hy- drodynamic theory. Transportation Research Part B: Method- ological, 28(4):269-287, 1994.","title":"The cell transmission model: A dynamic representation of highway traffic consistent with the hydrodynamic theory","authors":"C F Daganzo","year":"1994","venue":"Transportation Research Part B: Methodological","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diaconis and Freedman, 1980] P Diaconis and Diaconis Freed- man. De Finetti's generalizations of exchangeability. Studies in Inductive Logic and Probability, 2:233-249, 1980.","title":"P Diaconis and Diaconis Freedman. De Finetti's generalizations of exchangeability","authors":"Freedman Diaconis","year":"1980","venue":"Studies in Inductive Logic and Probability","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dibangoye and Buffet, 2018] Jilles Steeve Dibangoye and Olivier Buffet. Learning to act in decentralized partially observable MDPs. In International Conference on Machine Learning, pages 1241-1250, 2018.","title":"Learning to act in decentralized partially observable MDPs","authors":"Buffet ; Dibangoye, Steeve Jilles, Olivier Dibangoye, Buffet","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Tri- antafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghosh et al., 2015] Supriyo Ghosh, Akshat Kumar, and Pradeep Varakantham. Probabilistic inference based message-passing for resource constrained DCOPs. In International Joint Conference on Artificial Intelligence, pages 411-417, 2015.","title":"Probabilistic inference based message-passing for resource constrained DCOPs","authors":"Supriyo Ghosh, Akshat Kumar, Pradeep Varakantham","year":"2015","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/64"},{"position":11,"raw":"[Iwata and Shimizu, 2019] Tomoharu Iwata and Hitoshi Shimizu. Neural collective graphical models for estimating spatio- temporal population flow from aggregated data. In AAAI Con- ference on Artificial Intelligence, 2019.","title":"Neural collective graphical models for estimating spatiotemporal population flow from aggregated data","authors":"Shimizu ; ] Tomoharu Iwata, Hitoshi Iwata, Shimizu","year":"2019","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar and Zilberstein, 2009] Akshat Kumar and Shlomo Zilber- stein. Constraint-based dynamic programming for decentralized pomdps with structured interactions. In International Joint Con- ference on Autonomous Agents and Multiagent Systems, pages 561-568, 2009.","title":"Constraint-based dynamic programming for decentralized pomdps with structured interactions","authors":"Akshat Kumar, Shlomo Zilberstein","year":"2009","venue":"International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar and Zilberstein, 2010] Akshat Kumar and Shlomo Zilber- stein. Anytime planning for decentralized POMDPs using expec- tation maximization. In Conference on Uncertainty in Artificial Intelligence, pages 294-301, 2010.","title":"Anytime planning for decentralized POMDPs using expectation maximization","authors":"Akshat Kumar, Shlomo Zilberstein","year":"2010","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2011] Akshat Kumar, Shlomo Zilberstein, and Marc Toussaint. Scalable multiagent planning using probabilistic in- ference. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, pages 2140-2146, Barcelona, Spain, 2011.","title":"Scalable multiagent planning using probabilistic inference","authors":"Akshat Kumar, Shlomo Zilberstein, Marc Toussaint","year":"2011","venue":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/357"},{"position":15,"raw":"[Kumar et al., 2013] Akshat Kumar, Daniel Sheldon, and Biplav Srivastava. Collective diffusion over networks: Models and in- ference. In Conference on Uncertainty in Artificial Intelligence, 2013.","title":"Collective diffusion over networks: Models and inference","authors":"Akshat Kumar, Daniel Sheldon, Biplav Srivastava","year":"2013","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2015] Akshat Kumar, Shlomo Zilberstein, and Marc Toussaint. Probabilistic inference techniques for scalable multia- gent decision making. Journal of Artificial Intelligence Research, 53:223-270, 2015.","title":"Probabilistic inference techniques for scalable multiagent decision making","authors":"Akshat Kumar, Shlomo Zilberstein, Marc Toussaint","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kumar et al., 2016] Akshat Kumar, Hala Mostafa, and Shlomo Zilberstein. Dual formulations for optimizing Dec-POMDP con- trollers. In International Conference on Automated Planning and Scheduling, pages 202-210, 2016.","title":"Dual formulations for optimizing Dec-POMDP controllers","authors":"Akshat Kumar, Hala Mostafa, Shlomo Zilberstein","year":"2016","venue":"International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pages 6382-6393, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nair et al., 2005] R. Nair, P. Varakantham, M. Tambe, and M. Yokoo. Networked distributed POMDPs: A synthesis of dis- tributed constraint optimization and POMDPs. In AAAI Confer- ence on Artificial Intelligence, pages 133-139, 2005.","title":"Networked distributed POMDPs: A synthesis of distributed constraint optimization and POMDPs","authors":"R Nair, P Varakantham, M Tambe, M Yokoo","year":"2005","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2016] Duc Thien Nguyen, Akshat Kumar, Hoong Chuin Lau, and Daniel Sheldon. Approximate in- ference using DC programming for collective graphical models. In International Conference on Artificial Intelligence and Statistics, pages 685-693, 2016.","title":"Approximate inference using DC programming for collective graphical models","authors":"Akshat Duc Thien Nguyen, Hoong Kumar, Chuin Lau, Daniel Sheldon","year":"2016","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen et al., 2017a] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Collective multiagent sequential decision making under uncertainty. In AAAI Conference on Artificial In- telligence, pages 3036-3043, 2017.","title":"Collective multiagent sequential decision making under uncertainty","authors":"Nguyen","year":"2017","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen et al., 2017b] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Policy gradient with value function approxi- mation for collective multiagent planning. In Neural Information Processing Systems, pages 4322-4332, 2017.","title":"Policy gradient with value function approximation for collective multiagent planning","authors":"Nguyen","year":"2017","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nguyen et al., 2018] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multiagent RL with global rewards. In Advances in Neural Information Pro- cessing Systems, pages 8113-8124, 2018.","title":"Credit assignment for collective multiagent RL with global rewards","authors":"Akshat Duc Thien Nguyen, Hoong Chuin Kumar, Lau","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Niepert and Van den Broeck, 2014] Mathias Niepert and Guy Van den Broeck. Tractability through exchangeability: A new perspective on efficient probabilistic inference. In AAAI Confer- ence on Artificial Intelligence, pages 2467-2475, July 2014.","title":"Tractability through exchangeability: A new perspective on efficient probabilistic inference","authors":"Van Niepert, Broeck Den, Mathias Niepert, Guy Van Den Broeck","year":"2014","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sheldon and Dietterich, 2011] Daniel R. Sheldon and Thomas G. Dietterich. Collective graphical models. In Neural Information Processing Systems, pages 1161-1169, 2011.","title":"Collective graphical models","authors":"Daniel R Sheldon, Thomas G Dietterich","year":"2011","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sheldon et al., 2013] Daniel Sheldon, Tao Sun, Akshat Kumar, and Thomas G. Dietterich. Approximate inference in collec- tive graphical models. In International Conference on Machine Learning, pages 1004-1012, 2013.","title":"Approximate inference in collective graphical models","authors":"Daniel Sheldon, Tao Sun, Akshat Kumar, Thomas G Dietterich","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Singh and Kumar, 2019] Arambam James Singh and Akshat Ku- mar. Graph based optimization for multiagent cooperation. In International Conference on Autonomous Agents and MultiAgent Systems, pages 1497-1505, 2019.","title":"Arambam James Singh and Akshat Kumar. Graph based optimization for multiagent cooperation","authors":"Kumar Singh","year":"2019","venue":"International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Singh et al., 2019] Arambam James Singh, Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Multiagent decision mak- ing for maritime traffic management. In AAAI Conference on Artificial Intelligence, 2019.","title":"Multiagent decision making for maritime traffic management","authors":"Singh","year":"2019","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Spaan and Melo, 2008] Matthijs T. J. Spaan and Francisco S. Melo. Interaction-driven Markov games for decentralized mul- tiagent planning under uncertainty. In International COnference on Autonomous Agents and Multi Agent Systems, pages 525-532, 2008.","title":"Interaction-driven Markov games for decentralized multiagent planning under uncertainty","authors":"T J Matthijs, Francisco S Spaan, Melo","year":"2008","venue":"International COnference on Autonomous Agents and Multi Agent Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun et al., 2015] Tao Sun, Daniel Sheldon, and Akshat Kumar. Message passing for collective graphical models. In International Conference on Machine Learning, pages 853-861, 2015.","title":"Message passing for collective graphical models","authors":"Tao Sun, Daniel Sheldon, Akshat Kumar","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Toussaint and Storkey, 2006] Marc Toussaint and Amos J. Storkey. Probabilistic inference for solving discrete and continu- ous state markov decision processes. In International Conference on Machine Learning, pages 945-952, 2006.","title":"Probabilistic inference for solving discrete and continuous state markov decision processes","authors":"Marc Toussaint, Amos J Storkey","year":"2006","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Witwicki and Durfee, 2010] Stefan J. Witwicki and Edmund H. Durfee. Influence-based policy abstraction for weakly-coupled Dec-POMDPs. In International Conference on Automated Plan- ning and Scheduling, pages 185-192, 2010.","title":"Influence-based policy abstraction for weakly-coupled Dec-POMDPs","authors":"Stefan J Witwicki, Edmund H Durfee","year":"2010","venue":"International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}