{"type":"ScholarlyArticle","title":"SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations","authors":[{"name":"Chan Kim","slug":"chan-kim","given":"Chan","family":"Kim","affiliation":"Seoul National University"},{"name":"Jaekyung Cho","slug":"jaekyung-cho","given":"Jaekyung","family":"Cho","affiliation":"Seoul National University"},{"name":"Christophe Bobda","slug":"christophe-bobda","given":"Christophe","family":"Bobda","affiliation":"University of Florida {chan"},{"name":"Seung-Woo Seo","slug":"seung-woo-seo","given":"Seung-Woo","family":"Seo","affiliation":"Seoul National University"},{"name":"Seong-Woo Kim","slug":"seong-woo-kim","given":"Seong-Woo","family":"Kim","affiliation":"Seoul National University"}],"abstract":"Robotic agents trained using reinforcement learning have the problem of taking unreliable actions in an out-of-distribution (OOD) state. Agents can easily become OOD in real-world environments because it is almost impossible for them to visit and learn the entire state space during training. Unfortunately, unreliable actions do not ensure that agents perform their original tasks successfully. Therefore, agents should be able to recognize whether they are in OOD states and learn how to return to the learned state distribution rather than continue to take unreliable actions. In this study, we propose a novel method for retraining agents to recover from OOD situations in a self-supervised manner when they fall into OOD states. Our in-depth experimental results demonstrate that our method substantially improves the agent’s ability to recover from OOD situations in terms of sample efficiency and restoration of the performance for the original tasks. Moreover, we show that our method can retrain the agent to recover from OOD situations even when in-distribution states are difficult to visit through exploration. Code and supplementary materials are available at https://github.com/SNUChanKim/SeRO.","doi":"10.24963/ijcai.2023/432","url":"https://ijcai.org/proceedings/2023/432","pdf":"https://ijcai.org/proceedings/2023/0432.pdf","markdown":"https://ijcai.org/proceedings/2023/432.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/432","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":432,"pages":{"first":3884,"last":3892},"keywords":["Machine Learning: ML: Self-supervised Learning","Robotics: ROB: Learning in robotics","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abdar et al., 2021] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khos- ravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.","title":"A review of uncertainty quantification in deep learning: Techniques, applications and challenges","authors":"Abdar","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akkaya et al., 2019] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.","title":"Solving rubik's cube with a robot hand","authors":"Akkaya","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in ai safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. Eysenbach et al., 2018]","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, 2018.","title":"Leave no trace: Learning to reset for safe and autonomous reinforcement learning","authors":"Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, Sergey Levine","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th Inter- national Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Represent- ing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goan and Fookes, 2020] Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. In Case Studies in Applied Bayesian Data Science, pages 45- 87. Springer International Publishing, 2020.","title":"Bayesian neural networks: An introduction and survey","authors":"Fookes Goan, Ethan Goan, Clinton Fookes","year":"2020","venue":"Case Studies in Applied Bayesian Data Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2017] Shixiang Gu, Ethan Holly, Timothy Lilli- crap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy up- dates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.","title":"Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates","authors":"Gu","year":"2017","venue":"IEEE international conference on robotics and automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861- 1870, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Heess et al., 2017] Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emer- gence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017.","title":"Emergence of locomotion behaviours in rich environments","authors":"Nicolas Heess, T B Dhruva, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Eslami","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Henaff et al., 2019] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncer- tainty regularization for driving in dense traffic. In Inter- national Conference on Learning Representations, 2019.","title":"Model-predictive policy learning with uncertainty regularization for driving in dense traffic","authors":"Mikael Henaff, Alfredo Canziani, Yann Lecun","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kahn et al., 2017] Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty- aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182, 2017.","title":"Uncertaintyaware reinforcement learning for collision avoidance","authors":"Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kang et al., 2022] Katie Kang, Paula Gradu, Jason J Choi, Michael Janner, Claire Tomlin, and Sergey Levine. Lya- punov density models: Constraining distribution shift in learning-based control. In International Conference on Machine Learning, pages 10708-10733. PMLR, 2022.","title":"Lyapunov density models: Constraining distribution shift in learning-based control","authors":"Katie Kang, Paula Gradu, Jason J Choi, Michael Janner, Claire Tomlin, Sergey Levine","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural In- formation Processing Systems, volume 33, pages 1179- 1191.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2022] Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1455-1464. PMLR, 08-11 Nov 2022.","title":"Dealing with the unknown: Pessimistic offline reinforcement learning","authors":"Li","year":"2022","venue":"Proceedings of the 5th Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lütjens et al., 2019] Björn Lütjens, Michael Everett, and Jonathan P How. Safe reinforcement learning with model uncertainty estimates. In 2019 International Conference on Robotics and Automation (ICRA), pages 8662-8668. IEEE, 2019.","title":"Safe reinforcement learning with model uncertainty estimates","authors":"Lütjens","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sastry, 1999] Shankar Sastry. Lyapunov Stability Theory, pages 182-234. Springer New York, New York, NY, 1999.","title":"Lyapunov Stability Theory","authors":"Shankar Sastry","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2016] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. In Proceedings of the International Con- ference on Learning Representations (ICLR), 2016.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2016","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2021] Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Au- tonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34:18474-18486, 2021.","title":"Autonomous reinforcement learning via subgoal curricula","authors":"Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sharma et al., 2022a] Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distribution matching approach to non-episodic reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 19645-19657. PMLR, 17-23 Jul 2022.","title":"A state-distribution matching approach to non-episodic reinforcement learning","authors":"Sharma","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sharma et al., 2022b] Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and benchmarking. In International Confer- ence on Learning Representations, 2022.","title":"Autonomous reinforcement learning: Formalism and benchmarking","authors":"Sharma","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thananjeyan et al., 2021] Brijen Thananjeyan, Ashwin Bal- akrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922, 2021.","title":"Recovery rl: Safe reinforcement learning with learned recovery zones","authors":"Thananjeyan","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2021] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for of- fline reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11319-11328. PMLR, 18-24 Jul 2021.","title":"Uncertainty weighted actor-critic for offline reinforcement learning","authors":"Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}