{"type":"ScholarlyArticle","title":"Learning from Demonstrations with High-Level Side Information","authors":[{"name":"Min Wen","slug":"min-wen","given":"Min","family":"Wen","affiliation":"University of Pennsylvania"},{"name":"Ivan Papusha","slug":"ivan-papusha","given":"Ivan","family":"Papusha","affiliation":"University of Texas at Austin"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"University of Texas at Austin"}],"abstract":"We consider the problem of learning from demonstration, where extra side information about the demonstration is encoded as a co-safe linear temporal logic formula. We address two known limitations of existing methods that do not account for such side information. First, the policies that result from existing methods, while matching the expected features or likelihood of the demonstrations, may still be in conflict with high-level objectives not explicit in the demonstration trajectories. Second, existing methods fail to provide a priori guarantees on the out-of-sample generalization performance with respect to such high-level goals. This lack of formal guarantees can prevent the application of learning from demonstration to safety-\r\ncritical systems, especially when inference to state space regions with poor demonstration coverage is required. In this work, we show that side information, when explicitly taken into account, indeed improves the performance and safety of the learned policy with respect to task implementation. Moreover, we describe an automated procedure to systematically generate the features that encode side information expressed in temporal logic.","doi":"10.24963/ijcai.2017/426","url":"https://ijcai.org/proceedings/2017/426","pdf":"https://ijcai.org/proceedings/2017/0426.pdf","markdown":"https://ijcai.org/proceedings/2017/426.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/426","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":426,"pages":{"first":3055,"last":3061},"keywords":["Robotics and Vision: Task learning, learning from demonstration, Adaptive control","Planning and Scheduling: Markov Decisions Processes","Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Formal verification, validation and synthesis"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the twenty-first international con- ference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learn-ing","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Albert and Anderson, 1984] A. Albert and J. A. Anderson. On the existence of maximum likelihood estimates in lo- gistic regression models. Biometrika, 71(1):1, 1984.","title":"On the existence of maximum likelihood estimates in logistic regression models","authors":"A Albert, J A Anderson","year":"1984","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Babes et al., 2011] Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman. Ap- prenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 897-904, 2011.","title":"Apprenticeship learning about multiple intentions","authors":"Monica Babes, Kaushik Vukosi Marivate, Subramanian, Michael L Littman","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baier et al., 2008] Christel Baier, Joost-Pieter Katoen, and Kim Guldstrand Larsen. Principles of model checking. MIT press, 2008.","title":"Principles of model checking","authors":"Christel Baier, Joost-Pieter Katoen, Kim Guldstrand Larsen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bertsekas et al., 1995] Dimitri P Bertsekas, Dimitri P Bert- sekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control, volume 1. Athena Scientific Belmont, MA, 1995.","title":"Dynamic programming and optimal control","authors":"Dimitri P Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, Bertsekas","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bloem and Bambos, 2014] Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on, pages 4911-4916. IEEE, 2014.","title":"Infinite time horizon maximum causal entropy inverse reinforcement learning","authors":"Michael Bloem, Nicholas Bambos","year":"2014","venue":"IEEE 53rd Annual Conference on","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bobadilla et al., 2012] Leonardo Bobadilla, Oscar Sanchez, Justin Czarnowski, Katrina Gossman, and Steven M LaValle. Controlling wild bodies using linear tempo- ral logic. In Robotics: Science and systems, volume 7, page 17, 2012.","title":"Controlling wild bodies using linear temporal logic","authors":"Leonardo Bobadilla, Oscar Sanchez, Justin Czarnowski, Katrina Gossman, Steven M Lavalle","year":"2012","venue":"Robotics: Science and systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Boularias et al., 2011] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. pages 182-189, 2011.","title":"Relative entropy inverse reinforcement learning","authors":"Abdeslam Boularias, Jens Kober, Jan Peters","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dvijotham and Todorov, 2010] Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable MDPs. In International Conference on Machine Learning, pages 335-342, 2010.","title":"Inverse optimal control with linearly-solvable MDPs","authors":"Todorov Dvijotham, Krishnamurthy Dvijotham, Emanuel Todorov","year":"2010","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fainekos et al., 2005] Georgios E Fainekos, Hadas Kress- Gazit, and George J Pappas. Temporal logic motion plan- ning for mobile robots. In Robotics and Automation, 2005. ICRA 2005. Proceedings of the 2005 IEEE International Conference on, pages 2020-2025. IEEE, 2005.","title":"Temporal logic motion planning for mobile robots","authors":"Hadas Georgios E Fainekos, George J Kress-Gazit, Pappas","year":"2005","venue":"Proceedings of the 2005 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[John, 1994] George H John. When the best move isn't op- timal: Q-learning with exploration. In AAAI, page 1464.","title":"When the best move isn't optimal: Q-learning with exploration","authors":"John George","year":null,"venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kress-Gazit et al., 2011] Hadas Kress-Gazit, Tichakorn Wongpiromsarn, and Ufuk Topcu. Correct, reactive, high-level robot control. IEEE Robotics & Automation Magazine, 18(3):65-74, 2011.","title":"Correct, reactive, high-level robot control","authors":"Kress-Gazit","year":"2011","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kupferman and Vardi, 2001] Orna Kupferman and Moshe Y Vardi. Model checking of safety proper- ties. Formal Methods in System Design, 19(3):291-314, 2001.","title":"Model checking of safety properties","authors":"Vardi ; ] Orna Kupferman, Moshe Y Kupferman, Vardi","year":"2001","venue":"Formal Methods in System Design","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Macglashan and Littman, 2015] James Macglashan and Michael L Littman. Between imitation and intention learning. In International Conference on Artificial Intelligence, pages 3692-3698, 2015.","title":"Between imitation and intention learning","authors":"Littman Macglashan, James Macglashan, Michael L Littman","year":"2015","venue":"International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Neu and Szepesvári, 2007] Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse rein- forcement learning and gradient methods. In Proceedings of the Twenty-Third Conference on Uncertainty in Arti- ficial Intelligence, UAI'07, pages 295-302, Arlington, Virginia, United States, 2007. AUAI Press.","title":"Apprenticeship learning using inverse reinforcement learning and gradient methods","authors":"Szepesvári ; ] Gergely Neu, Csaba Neu, Szepesvári","year":"2007","venue":"Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, UAI'07","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In Icml, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pnueli, 1981] Amir Pnueli. The temporal semantics of concurrent programs. Theoretical computer science, 13(1):45-60, 1981.","title":"The temporal semantics of concurrent programs","authors":"Amir Pnueli, Pnueli","year":"1981","venue":"Theoretical computer science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ramachandran and Amir, 2007] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI 2007, Proceedings of the International Joint Conference on Artificial Intelligence, Hyderabad, India, January, pages 2586-2591, 2007.","title":"Bayesian inverse reinforcement learning","authors":"Deepak Ramachandran, Eyal Amir","year":"2007","venue":"IJCAI 2007, Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ratliff et al., 2006] Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Ma- chine learning, pages 729-736. ACM, 2006.","title":"Maximum margin planning","authors":"J Nathan D Ratliff, Martin A Andrew Bagnell, Zinkevich","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wolff et al., 2012] Eric Wolff, Ufuk Topcu, and Richard Murray. Optimal control with weighted average costs and temporal logic specifications. In Proceedings of Robotics: Science and Systems, Sydney, Australia, July 2012.","title":"Optimal control with weighted average costs and temporal logic specifications","authors":"Eric Wolff, Ufuk Topcu, Richard Murray","year":"2012","venue":"Proceedings of Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew Maas, J. An- drew Bagnell, and Anind K Dey. Maximum entropy in- verse reinforcement learning. In National Conference on Artificial Intelligence, pages 1433-1438, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}