{"type":"ScholarlyArticle","title":"BRPO: Batch Residual Policy Optimization","authors":[{"name":"Sungryull Sohn","slug":"sungryull-sohn","given":"Sungryull","family":"Sohn","affiliation":"Google Research; University of Michigan"},{"name":"Yinlam Chow","slug":"yinlam-chow","given":"Yinlam","family":"Chow","affiliation":"Google Research"},{"name":"Jayden Ooi","slug":"jayden-ooi","given":"Jayden","family":"Ooi","affiliation":"Google Research"},{"name":"Ofir Nachum","slug":"ofir-nachum","given":"Ofir","family":"Nachum","affiliation":"Google Research"},{"name":"Honglak Lee","slug":"honglak-lee","given":"Honglak","family":"Lee","affiliation":"Google Research; University of Michigan"},{"name":"Ed Chi","slug":"ed-chi","given":"Ed","family":"Chi","affiliation":"Google Research"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google Research"}],"abstract":"In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that\r\nis the same at each state. This can cause batch RL to be overly conservative, unable to exploit large policy changes at frequently-visited, high-confidence states without risking poor performance at sparsely-visited states. To remedy this, we propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method,  BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance. We show that BRPO achieves the state-of-the-art performance in a number of tasks.","doi":"10.24963/ijcai.2020/391","url":"https://ijcai.org/proceedings/2020/391","pdf":"https://ijcai.org/proceedings/2020/0391.pdf","markdown":"https://ijcai.org/proceedings/2020/391.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/391","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":391,"pages":{"first":2824,"last":2830},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25009/file_25009.m3u8","length_sec":266,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2019] Y. Abbasi-Yadkori, P. Bartlett, X. Chen, and A. Malek. Large-scale markov de- cision problems via the linear programming dual. arXiv:1901.01992, 2019.","title":"Large-scale markov decision problems via the linear programming dual","authors":"Y Abbasi-Yadkori, P Bartlett, X Chen, A Malek","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amos and Kolter, 2017] B. Amos and Z. Kolter. Optnet: Differentiable optimization as a layer in neural networks. In ICML, pages 136-145, 2017.","title":"Optnet: Differentiable optimization as a layer in neural networks","authors":"B Amos, Z Kolter","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolley and Villani, 2005] F. Bolley and C. Villani. Weighted csiszár-kullback-pinsker inequalities and applications to transportation inequalities. In Annales de la Faculté des sciences de Toulouse: Mathématiques, volume 14, pages 331-352, 2005.","title":"Weighted csiszár-kullback-pinsker inequalities and applications to transportation inequalities","authors":"F Bolley, C Villani","year":"2005","venue":"Annales de la Faculté des sciences de Toulouse: Mathématiques","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boyd and Vandenberghe, 2004] S. Boyd and L. Vanden- berghe. Convex optimization. Cambridge university press, 2004.","title":"Convex optimization","authors":"S Boyd, L Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pet- tersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. [Farias and Roy, 2003]","title":"Openai gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"P. De Farias and B. Van Roy. The lin- ear programming approach to approximate dynamic pro- gramming. Operations research, 51(6):850-865, 2003.","title":"The linear programming approach to approximate dynamic programming","authors":"P De Farias, B Van Roy","year":"2003","venue":"Operations research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Faybusovich and Moore, 1997] L. Faybusovich and J. Moore. Infinite-dimensional quadratic optimization: interior-point methods and control applications. Applied Mathematics and Optimization, 36(1):43-66, 1997.","title":"Infinite-dimensional quadratic optimization: interior-point methods and control applications","authors":"L Faybusovich, J Moore","year":"1997","venue":"Applied Mathematics and Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fujimoto et al., 2018] S. Fujimoto, D. Meger, and D. Pre- cup. Off-policy deep reinforcement learning without ex- ploration. arXiv:1812.02900, 2018.","title":"Precup. Off-policy deep reinforcement learning without exploration","authors":"S Fujimoto, D Meger, D","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, Edoardo Conti, Mo- hammad Ghavamzadeh, and Joelle Pineau. Bench- marking batch deep reinforcement learning algorithms. arXiv:1910.01708, 2019.","title":"Benchmarking batch deep reinforcement learning algorithms","authors":"Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, Joelle Pineau","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghavamzadeh et al., 2016] M. Ghavamzadeh, M. Petrik, and Y. Chow. Safe policy improvement by minimizing robust baseline regret. In NIPS, 2016.","title":"Safe policy improvement by minimizing robust baseline regret","authors":"M Ghavamzadeh, M Petrik, Y Chow","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gretton et al., 2007] A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel approach to com- paring distributions. In AAAI, 2007.","title":"A kernel approach to comparing distributions","authors":"A Gretton, K Borgwardt, M Rasch, B Schölkopf, A Smola","year":"2007","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haarnoja et al., 2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"T Haarnoja, A Zhou, P Abbeel, S Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hasselt et al., 2016] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"H Van Hasselt, A Guez, D Silver","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hunter and Lange, 2004] D. Hunter and K. Lange. A tu- torial on MM algorithms. The American Statistician, 58(1):30-37, 2004.","title":"A tutorial on MM algorithms","authors":"D Hunter, K Lange","year":"2004","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaques et al., 2019] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off- policy batch deep reinforcement learning of implicit hu- man preferences in dialog. arXiv:1907.00456, 2019.","title":"Way offpolicy batch deep reinforcement learning of implicit human preferences in dialog","authors":"Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, Rosalind Picard","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Johannink et al., 2019] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. Ojea, E. Solowjow, and S.Levine. Residual reinforcement learning for robot con- trol. In ICRA, pages 6023-6029. IEEE, 2019.","title":"Residual reinforcement learning for robot control","authors":"T Johannink, S Bahl, A Nair, J Luo, A Kumar, M Loskyll, J Ojea, E Solowjow, S Levine","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kakade and Langford, 2002] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learn- ing. In ICML, pages 267-274, 2002.","title":"Approximately optimal approximate reinforcement learning","authors":"S Kakade, J Langford","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kober and Peters, 2010] J. Kober and J. Peters. Imitation and reinforcement learning. IEEE Robotics & Automation Magazine, 17(2):55-62, 2010.","title":"Imitation and reinforcement learning","authors":"Kober, ] J Peters, J Kober, Peters","year":"2010","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kumar et al., 2019] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrap- ping error reduction. arXiv:1906.00949, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"A Kumar, J Fu, G Tucker, S Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lange et al., 2012] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.","title":"Batch reinforcement learning","authors":"S Lange, T Gabel, M Riedmiller","year":"2012","venue":"Reinforcement learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Laroche and Trichelair, 2017] R. Laroche and P. Trichelair. Safe policy improvement with baseline bootstrapping. arXiv:1712.06924, 2017.","title":"Safe policy improvement with baseline bootstrapping","authors":"R Laroche, P Trichelair","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lillicrap et al., 2015] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous control with deep reinforcement learning. arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"T Lillicrap, J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mahmood et al., 2014] A. Mahmood, H. van Hasselt, and R. Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In NIPS, pages 3014-3022, 2014.","title":"Weighted importance sampling for off-policy learning with linear function approximation","authors":"A Mahmood, H Van Hasselt, R Sutton","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mnih et al., 2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Ried- miller. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Graves, I Antonoglou, D Wierstra, M Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nachum et al., 2019] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. CoRL, 2019.","title":"Multi-agent manipulation via locomotion using hierarchical sim2real","authors":"O Nachum, M Ahn, H Ponte, S Gu, V Kumar","year":"2019","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pirotta et al., 2013] Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy iteration. In ICML, pages 307-315, 2013.","title":"Safe policy iteration","authors":"Matteo Pirotta, Marcello Restelli, Alessio Pecorino, Daniele Calandriello","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rusu et al., 2015] A. Rusu, S. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv:1511.06295, 2015.","title":"Policy distillation","authors":"A Rusu, S Colmenarejo, C Gulcehre, G Desjardins, J Kirkpatrick, R Pascanu, V Mnih, K Kavukcuoglu, R Hadsell","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schulman et al., 2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"J Schulman, S Levine, P Abbeel, M Jordan, P Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shapiro et al., 2009] A. Shapiro, D. Dentcheva, and A. Ruszczyński. Lectures on stochastic programming: modeling and theory. SIAM, 2009.","title":"Lectures on stochastic programming: modeling and theory","authors":"A Shapiro, D Dentcheva, A Ruszczyński","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Silver et al., 2018] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling. Residual policy learning. arXiv:1812.06298, 2018.","title":"Residual policy learning","authors":"T Silver, K Allen, J Tenenbaum, L Kaelbling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu and Yin, 2013] Y. Xu and W. Yin. A block coordi- nate descent method for regularized multiconvex opti- mization with applications to nonnegative tensor factoriza- tion and completion. SIAM Journal on imaging sciences, 6(3):1758-1789, 2013.","title":"A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion","authors":"Yin Xu, ] Y Xu, W Yin","year":"2013","venue":"SIAM Journal on imaging sciences","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}