{"type":"ScholarlyArticle","title":"Online Decision-Making for Scalable Autonomous Systems","authors":[{"name":"Kyle Hollins Wray","slug":"kyle-hollins-wray","given":"Kyle Hollins","family":"Wray","affiliation":"College of Information and Computer Sciences , University of Massachusetts , Amherst , MA 01002; Nissan Research Center -Silicon Valley , Sunnyvale , CA 94089"},{"name":"Stefan J. Witwicki","slug":"stefan-j-witwicki","given":"Stefan J.","family":"Witwicki","affiliation":"Nissan Research Center -Silicon Valley , Sunnyvale , CA 94089"},{"name":"Shlomo Zilberstein","slug":"shlomo-zilberstein","given":"Shlomo","family":"Zilberstein","affiliation":"College of Information and Computer Sciences , University of Massachusetts , Amherst , MA 01002"}],"abstract":"We present a general formal model called MODIA that can tackle a central challenge for autonomous vehicles (AVs), namely the ability to interact with an unspecified, large number of world entities. In MODIA, a collection of possible decision-problems (DPs), known a priori, are instantiated online and executed as decision-components (DCs), unknown a priori. To combine their individual action recommendations of the DCs into a single action, we propose the lexicographic executor action function (LEAF) mechanism. We analyze the complexity of MODIA and establish LEAF’s relation to regret minimization. Finally, we implement MODIA and LEAF using collections of partially observable Markov decision process (POMDP) DPs, and use them for complex AV intersection decision-making. We evaluate the approach in six scenarios within an industry-standard vehicle simulator, and present its use on an AV prototype.","doi":"10.24963/ijcai.2017/664","url":"https://ijcai.org/proceedings/2017/664","pdf":"https://ijcai.org/proceedings/2017/0664.pdf","markdown":"https://ijcai.org/proceedings/2017/664.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/664","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"AI and autonomy track","track":"AI and autonomy track","number":664,"pages":{"first":4768,"last":4774},"keywords":["Technical: Techniques","Technical: Models"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2015] Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee. Intention-aware online POMDP planning for au- tonomous driving in a crowd. In IEEE Int'l. Conf. on Robotics and Automation (ICRA), pages 454-460, 2015.","title":"Intention-aware online POMDP planning for autonomous driving in a crowd","authors":"Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, Wee Sun, Lee","year":"2015","venue":"IEEE Int'l. Conf. on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bandyopadhyay et al., 2013] Tirthankar Bandyopadhyay, Chong Zhuang Jie, David Hsu, Marcelo H. Ang, Daniela Rus, and Emilio Frazzoli. Intention-aware pedestrian avoidance. In Proc. of the 13th Int'l. Symposium on Experimental Robotics, pages 963-977, 2013.","title":"Intention-aware pedestrian avoidance","authors":"Tirthankar Bandyopadhyay, Chong Zhuang Jie, David Hsu, Marcelo H Ang, Daniela Rus, Emilio Frazzoli","year":"2013","venue":"Proc. of the 13th Int'l. Symposium on Experimental Robotics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barto and Mahadevan, 2003] Andrew G. Barto and Sridhar Ma- hadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4):341-379, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"Andrew G Barto, Sridhar Mahadevan","year":"2003","venue":"Discrete Event Dynamic Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brechtel et al., 2014] Sebastian Brechtel, Tobias Gindele, and Rüdiger Dillmann. Probabilistic decision-making under uncer- tainty for autonomous driving using continuous POMDPs. In Proc. of the 17th Int'l. IEEE Conf. on Intelligent Transportation Systems (ITSC), pages 392-399, 2014.","title":"Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs","authors":"Sebastian Brechtel, Tobias Gindele, Rüdiger Dillmann","year":"2014","venue":"Proc. of the 17th Int'l. IEEE Conf. on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brooks, 1986] Rodney Brooks. A robust layered control system for a mobile robot. IEEE Journal on Robotics and Automation, 2(1):14-23, 1986.","title":"A robust layered control system for a mobile robot","authors":"Rodney Brooks","year":"1986","venue":"IEEE Journal on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2015] Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. DeepDriving: Learning affordance for direct perception in autonomous driving. In Proc. of the 15th IEEE Int'l. Conf. on Computer Vision (ICCV), pages 2722-2730, 2015.","title":"DeepDriving: Learning affordance for direct perception in autonomous driving","authors":"Chen","year":"2015","venue":"Proc. of the 15th IEEE Int'l. Conf. on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Decker, 1996] Keith Decker. TAEMS: A framework for environ- ment centered analysis & design of coordination mechanisms. Foundations of Distributed Artificial Intelligence, pages 429- 448, 1996.","title":"TAEMS: A framework for environment centered analysis & design of coordination mechanisms","authors":"Keith Decker","year":"1996","venue":"Foundations of Distributed Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dolgov et al., 2010] Dmitri Dolgov, Sebastian Thrun, Michael Montemerlo, and James Diebel. Path planning for autonomous vehicles in unknown semi-structured environments. The Int'l. Journal of Robotics Research, 29(5):485-501, 2010.","title":"Path planning for autonomous vehicles in unknown semi-structured environments","authors":"Dmitri Dolgov, Sebastian Thrun, Michael Montemerlo, James Diebel","year":"2010","venue":"The Int'l. Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hauskrecht et al., 1998] Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Hi- erarchical solution of Markov decision processes using macro- actions. In Proc. of the 14th Conf. on Uncertainty in Artificial Intelligence (UAI), pages 220-229, 1998.","title":"Hierarchical solution of Markov decision processes using macroactions","authors":"Hauskrecht","year":"1998","venue":"Proc. of the 14th Conf. on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jo et al., 2015] Kichun Jo, Junsoo Kim, Dongchul Kim, Chulhoon Jang, and Myoungho Sunwoo. Development of autonomous car- Part II: A case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Transac- tions on Industrial Electronics, 62(8):5119-5132, 2015.","title":"Development of autonomous car-Part II: A case study on the implementation of an autonomous driving system based on distributed architecture","authors":"Jo","year":"2015","venue":"IEEE Transactions on Industrial Electronics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaelbling et al., 1998] Leslie P. Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially ob- servable stochastic domains. Artificial Intelligence, 101(1):99- 134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie P Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kurniawati and Yadav, 2016] Hanna Kurniawati and Vinay Yadav. An online POMDP solver for uncertainty planning in dynamic environment. In Proc. of the 16th Int'l. Symposium Robotics Re- search (ISRR), pages 611-629, 2016.","title":"An online POMDP solver for uncertainty planning in dynamic environment","authors":"Yadav ; ] Kurniawati, Kurniawati Hanna, Vinay Yadav","year":"2016","venue":"Proc. of the 16th Int'l. Symposium Robotics Research (ISRR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Parr and Russell, 1998] Ronald Parr and Stuart Russell. Reinforce- ment learning with hierarchies of machines. In Proc. of the 10th Conf. on Advances in Neural Information Processing Systems (NIPS), pages 1043-1049, 1998.","title":"Reinforcement learning with hierarchies of machines","authors":"Russell ; Parr, Ronald Parr, Stuart Russell","year":"1998","venue":"Proc. of the 10th Conf. on Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pineau et al., 2001] Joelle Pineau, Nicholas Roy, and Sebastian Thrun. A hierarchical approach to POMDP planning and execu- tion. In Proc. of the ICML Workshop on Hierarchy and Memory in Reinforcement Learning, 2001.","title":"A hierarchical approach to POMDP planning and execution","authors":"Joelle Pineau, Nicholas Roy, Sebastian Thrun","year":"2001","venue":"Proc. of the ICML Workshop on Hierarchy and Memory in Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rosenblatt, 1997] Julio K. Rosenblatt. DAMN: A distributed ar- chitecture for mobile navigation. Journal of Experimental & The- oretical Artificial Intelligence, 9(2-3):339-360, 1997.","title":"DAMN: A distributed architecture for mobile navigation","authors":"Julio K Rosenblatt","year":"1997","venue":"Journal of Experimental & Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ross et al., 2008] Stéphane Ross, Joelle Pineau, Sébastien Pa- quet, and Brahim Chaib-Draa. Online planning algorithms for POMDPs. Journal of Artificial Intelligence Research, 32:663- 704, 2008.","title":"Online planning algorithms for POMDPs","authors":"Stéphane Ross, Joelle Pineau, Sébastien Paquet, Brahim Chaib-Draa","year":"2008","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sivaraman and Trivedi, 2013] Sayanan Sivaraman and Mohan M. Trivedi. Looking at vehicles on the road: A survey of vision- based vehicle detection, tracking, and behavior analysis. IEEE Transactions on Intelligent Transportation Systems, 14(4):1773- 1795, 2013.","title":"Looking at vehicles on the road: A survey of visionbased vehicle detection, tracking, and behavior analysis","authors":"Trivedi Sivaraman, Sayanan Sivaraman, Mohan M Trivedi","year":"2013","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Somani et al., 2013] Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP planning with regu- larization. In Proc. of the 26th Conf. on Advances in Neural In- formation Processing Systems (NIPS), pages 1772-1780, 2013.","title":"DESPOT: Online POMDP planning with regularization","authors":"Adhiraj Somani, Nan Ye, David Hsu, Wee Sun, Lee","year":"2013","venue":"Proc. of the 26th Conf. on Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sridharan et al., 2010] Mohan Sridharan, Jeremy Wyatt, and Richard Dearden. Planning to see: A hierarchical approach to planning visual actions on a robot using POMDPs. Artificial In- telligence, 174(11):704-725, 2010.","title":"Planning to see: A hierarchical approach to planning visual actions on a robot using POMDPs","authors":"Mohan Sridharan, Jeremy Wyatt, Richard Dearden","year":"2010","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tao et al., 2009] Yong Tao, Tianmiao Wang, Hongxing Wei, and Diansheng Chen. A behavior control method based on hierarchi- cal POMDP for intelligent wheelchair. In Proc. of IEEE/ASME Int'l. Conf. on Advanced Intelligent Mechatronics (AIM), pages 893-898, 2009.","title":"A behavior control method based on hierarchical POMDP for intelligent wheelchair","authors":"Yong Tao, Tianmiao Wang, Hongxing Wei, Diansheng Chen","year":"2009","venue":"Proc. of IEEE/ASME Int'l. Conf. on Advanced Intelligent Mechatronics (AIM)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thrun et al., 2006] Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny, Gabriel Hoffmann, et al. Stanley: The robot that won the DARPA Grand Challenge. Jour- nal of Field Robotics, 23(9):661-692, 2006.","title":"Stanley: The robot that won the DARPA Grand Challenge","authors":"Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny, Gabriel Hoffmann","year":"2006","venue":"Journal of Field Robotics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ulbrich and Maurer, 2013] Simon Ulbrich and Markus Maurer. Probabilistic online POMDP decision making for lane changes in fully automated driving. In Proc. of the 16th Int'l. IEEE Conf. on Intelligent Transportation Systems (ITSC), pages 2063-2067, 2013.","title":"Probabilistic online POMDP decision making for lane changes in fully automated driving","authors":"Maurer Ulbrich, Simon Ulbrich, Markus Maurer","year":"2013","venue":"Proc. of the 16th Int'l. IEEE Conf. on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wray and Zilberstein, 2015] Kyle H. Wray and Shlomo Zilber- stein. A parallel point-based POMDP algorithm leveraging GPUs. In Proc. of the 2015 AAAI Fall Symposium on Sequen- tial Decision Making for Intelligent Agents, pages 95-96, 2015.","title":"A parallel point-based POMDP algorithm leveraging GPUs","authors":"Kyle H Wray, Shlomo Zilberstein","year":"2015","venue":"Proc. of the 2015 AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wray et al., 2016a] Kyle H. Wray, Luis Pineda, and Shlomo Zil- berstein. Hierarchical approach to transfer of control in semi- autonomous systems. In Proc. of the 25th Int'l. Joint Conf. on Artificial Intelligence (IJCAI), pages 517-523, 2016.","title":"Hierarchical approach to transfer of control in semiautonomous systems","authors":"Wray","year":"2016","venue":"Proc. of the 25th Int'l. Joint Conf. on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wray et al., 2016b] Kyle H. Wray, Dirk Ruiken, Roderic A. Gru- pen, and Shlomo Zilberstein. Log-space harmonic function path planning. In Proc. of the 29th IEEE/RSJ Int'l. Conf. on Intelligent Robots and Systems (IROS), pages 1511-1516, 2016.","title":"Log-space harmonic function path planning","authors":"Wray","year":"2016","venue":"Proc. of the 29th IEEE/RSJ Int'l. Conf. on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yadav et al., 2015] Amulya Yadav, Leandro S. Marcolino, Eric Rice, Robin Petering, Hailey Winetrobe, Harmony Rhoades, Milind Tambe, and Heather Carmichael. Preventing HIV spread in homeless populations using PSINET. In Proc. of the 27th Conf. on Innovative Applications of Artificial Intelligence (IAAI), pages 4006-4011, 2015.","title":"Preventing HIV spread in homeless populations using PSINET","authors":"Amulya Yadav, Leandro S Marcolino, Eric Rice, Robin Petering, Hailey Winetrobe, Harmony Rhoades, Milind Tambe, Heather Carmichael","year":"2015","venue":"Proc. of the 27th Conf. on Innovative Applications of Artificial Intelligence (IAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}