{"type":"ScholarlyArticle","title":"Truly Batch Apprenticeship Learning with Deep Successor Features","authors":[{"name":"Donghun Lee","slug":"donghun-lee","given":"Donghun","family":"Lee","affiliation":"SEAS , Harvard University"},{"name":"Srivatsan Srinivasan","slug":"srivatsan-srinivasan","given":"Srivatsan","family":"Srinivasan","affiliation":"SEAS , Harvard University"},{"name":"Finale Doshi-Velez","slug":"finale-doshi-velez","given":"Finale","family":"Doshi-Velez","affiliation":"SEAS , Harvard University"}],"abstract":"We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free batch settings. Unlike existing methods that require hand-crafted features, on-policy evaluation, further data acquisition for evaluation policies or the knowledge of model dynamics, our algorithm requires only batch data (demonstrations) of the observed expert behavior.  Such settings are common in many real-world tasks---health care, finance, or industrial process control---where accurate simulators do not exist and additional data acquisition is costly.  We develop a transition-regularized imitation learning model to learn a rich feature representation and a near-expert initial policy that makes the subsequent batch inverse reinforcement learning process viable. We also introduce deep successor feature networks that perform off-policy evaluation to estimate feature expectations of candidate policies. Under the batch setting, our method achieves superior results on control benchmarks as well as a real clinical task of sepsis management in the Intensive Care Unit.","doi":"10.24963/ijcai.2019/819","url":"https://ijcai.org/proceedings/2019/819","pdf":"https://ijcai.org/proceedings/2019/0819.pdf","markdown":"https://ijcai.org/proceedings/2019/819.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/819","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":819,"pages":{"first":5909,"last":5915},"keywords":["Special Track on AI for Improving Human-Well Being: Health applications (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: AI safety (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In 21 st ICM L, page 1, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"21 st ICM L","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boularias et al., 2011] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In 14 th AISTATS, pages 182-189, 2011.","title":"Relative entropy inverse reinforcement learning","authors":"Abdeslam Boularias, Jens Kober, Jan Peters","year":"2011","venue":"14 th AISTATS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burchfiel et al., 2016] Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance minimization for reward learn- ing from scored trajectories. In 30 th AAAI, pages 3330- 3336, 2016.","title":"Distance minimization for reward learning from scored trajectories","authors":"Benjamin Burchfiel, Carlo Tomasi, Ronald Parr","year":"2016","venue":"30 th AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In 33 rd ICML, pages 1329-1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"33 rd ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2017] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv:1710.11248, 2017.","title":"Learning robust rewards with adversarial inverse reinforcement learning","authors":"Justin Fu, Katie Luo, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Henderson et al., 2017] Peter Henderson, Risashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv:1709.06560v3, 2017.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Risashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In 30 th NeurIPS, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"30 th NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Johnson et al., 2016] Alistair Johnson, Tom Pollard, Lu Shen, Lehman Li-wei, Mohammad Ghassemi, Ben- jamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035, 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"Alistair Johnson, Tom Pollard, Lu Shen, Lehman Li-Wei, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, Roger Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Klein et al., 2011] Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free appren- ticeship learning. In European Workshop on Reinforce- ment Learning, pages 285-296. Springer, 2011.","title":"Batch, off-policy and model-free apprenticeship learning","authors":"Edouard Klein, Matthieu Geist, Olivier Pietquin","year":"2011","venue":"European Workshop on Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Klein et al., 2012] Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learn- ing through structured classification. In 26 th NIPS, pages 1007-1015, 2012.","title":"Inverse reinforcement learning through structured classification","authors":"Edouard Klein, Matthieu Geist, Bilal Piot, Olivier Pietquin","year":"2012","venue":"26 th NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klein et al., 2013] Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pietquin. A cascaded supervised learn- ing approach to inverse reinforcement learning. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pages 1-16. Springer, 2013.","title":"A cascaded supervised learning approach to inverse reinforcement learning","authors":"Edouard Klein, Bilal Piot, Matthieu Geist, Olivier Pietquin","year":"2013","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulkarni et al., 2016] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep succes- sor reinforcement learning. arXiv:1606.02396, 2016.","title":"Deep successor reinforcement learning","authors":"D Tejas, Ardavan Kulkarni, Simanta Saeedi, Samuel J Gautam, Gershman","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lagoudakis and Parr, 2003] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. JMLR, 4:1107-1149, 2003.","title":"Least-squares policy iteration","authors":"G Michail, Ronald Lagoudakis, Parr","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Leike et al., 2017] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro Ortega, Laurent Orseau, and Shane Legg. Ai safety gridworlds. arXiv:1711.09883, 2017.","title":"Ai safety gridworlds","authors":"Leike","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mervyn et al., 2016] Singer Mervyn, Deutschman Clifford S., Seymour Cristopher, and et al. The third international consensus definitions for sepsis and septic shock (sepsis- 3). JAMA, 315(8):801-810, 2016.","title":"The third international consensus definitions for sepsis and septic shock (sepsis-3)","authors":"Deutschman Singer Mervyn, S Clifford, Seymour Cristopher","year":"2016","venue":"JAMA","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. volume 518, page 529. Na- ture Publishing Group, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In 17 th ICML, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"17 th ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Piot et al., 2013] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Learning from demonstrations: Is it worth es- timating a reward function? In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 17-32. Springer, 2013.","title":"Learning from demonstrations: Is it worth estimating a reward function?","authors":"Bilal Piot, Matthieu Geist, Olivier Pietquin","year":"2013","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Piot et al., 2014] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. In Proceedings of the 2014 international conference on Autonomous agents and multi- agent systems, pages 1249-1256. International Foundation for Autonomous Agents and Multiagent Systems, 2014.","title":"Boosted and reward-regularized classification for apprenticeship learning","authors":"Bilal Piot, Matthieu Geist, Olivier Pietquin","year":"2014","venue":"Proceedings of the 2014 international conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Piot et al., 2017] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Bridging the gap between imitation learning and inverse reinforcement learning. IEEE transactions on neu- ral nets and learning systems, 28(8):1814-1826, 2017.","title":"Bridging the gap between imitation learning and inverse reinforcement learning","authors":"Bilal Piot, Matthieu Geist, Olivier Pietquin","year":"2017","venue":"IEEE transactions on neural nets and learning systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raghu et al., 2017] Aniruddh Raghu, Matthieu Ko- morowski, Leo Celi Ahmed, Imran, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv:1711.09062, 2017.","title":"Deep reinforcement learning for sepsis treatment","authors":"Aniruddh Raghu, Matthieu Komorowski, Leo Celi Ahmed, Imran, Peter Szolovits, Marzyeh Ghassemi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ratliff et al., 2006] Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In 23 rd ICML, pages 729-736, 2006.","title":"Maximum margin planning","authors":"J Nathan D Ratliff, Martin A Andrew Bagnell, Zinkevich","year":"2006","venue":"23 rd ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In 14 th AIS- TATS, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey Gordon, Drew Bagnell","year":"2011","venue":"14 th AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2016] Zhao Song, Ronald Parr, Xuejun Liao, and Lawrence Carin. Linear feature encoding for rein- forcement learning. In 30 th NeurIPS, pages 4224-4232, 2016.","title":"Linear feature encoding for reinforcement learning","authors":"Zhao Song, Ronald Parr, Xuejun Liao, Lawrence Carin","year":"2016","venue":"30 th NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton et al., 1998] Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.","title":"Introduction to reinforcement learning","authors":"Andrew G Richard S Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thomas and Brunskill, 2016] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for rl. In 33 rd ICML, pages 2139-2148, 2016.","title":"Data-efficient off-policy policy evaluation for rl","authors":"Brunskill Thomas, Philip Thomas, Emma Brunskill","year":"2016","venue":"33 rd ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van Hasselt et al., 2015] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. arXiv:1509.06461, 2015.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In 23 rd AAAI, pages 1433- 1438, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"23 rd AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}