{"type":"ScholarlyArticle","title":"Model-Based Offline Planning with Trajectory Pruning","authors":[{"name":"Xianyuan Zhan","slug":"xianyuan-zhan","given":"Xianyuan","family":"Zhan","affiliation":"Institute for AI Industry Research (AIR) , Tsinghua University , Beijing , China"},{"name":"Xiangyu Zhu","slug":"xiangyu-zhu","given":"Xiangyu","family":"Zhu","affiliation":"Institute for AI Industry Research (AIR) , Tsinghua University , Beijing , China"},{"name":"Haoran Xu","slug":"haoran-xu","given":"Haoran","family":"Xu","affiliation":"JD iCity & JD Intelligent Cities Research , JD Technology , Beijing , China"}],"abstract":"The recent offline reinforcement learning (RL) studies have achieved much progress to make RL usable in real-world systems by learning policies from pre-collected datasets without environment interaction. Unfortunately, existing offline RL methods still face many practical challenges in real-world system control tasks, such as computational restriction during agent training and the requirement of extra control flexibility. The model-based planning framework provides an attractive alternative. However, most model-based planning algorithms are not designed for offline settings. Simply combining the ingredients of offline RL with existing methods either provides over-restrictive planning or leads to inferior performance. We propose a new light-weighted model-based offline planning framework, namely MOPP, which tackles the dilemma between the restrictions of offline learning and high-performance planning. MOPP encourages more aggressive trajectory rollout guided by the behavior policy learned from data, and prunes out problematic trajectories to avoid potential out-of-distribution samples. Experimental results show that MOPP provides competitive performance compared with existing model-based offline planning and RL approaches.","doi":"10.24963/ijcai.2022/516","url":"https://ijcai.org/proceedings/2022/516","pdf":"https://ijcai.org/proceedings/2022/0516.pdf","markdown":"https://ijcai.org/proceedings/2022/516.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/516","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":516,"pages":{"first":3716,"last":3722},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning Algorithms","Machine Learning: Deep Reinforcement Learning","Planning and Scheduling: Markov Decisions Processes"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986567","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984710","length_sec":329,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Argenson and Dulac-Arnold, 2021] Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Conference on Learning Representations, 2021.","title":"Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning","authors":"Argenson, Dulac-Arnold","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Botev et al., 2013] Zdravko I Botev, Dirk P Kroese, Reuven Y Ru- binstein, and Pierre L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35-59. Elsevier, 2013.","title":"The cross-entropy method for optimization","authors":"Dirk P Zdravko I Botev, Reuven Y Kroese, Pierre L' Rubinstein, Ecuyer","year":"2013","venue":"Handbook of statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buckman et al., 2020] Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Represen- tations, 2020.","title":"The importance of pessimism in fixed-dataset policy optimization","authors":"Buckman","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chua et al., 2018] Kurtland Chua, Roberto Calandra, Rowan McAl- lister, and Sergey Levine. Deep reinforcement learning in a hand- ful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754-4765, 2018.","title":"Deep reinforcement learning in a handful of trials using probabilistic dynamics models","authors":"Chua","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujimoto and Gu, 2021] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Ad- vances in Neural Information Processing Systems, 2021.","title":"A minimalist approach to offline reinforcement learning","authors":"Gu Fujimoto, Scott Fujimoto, Shixiang Shane Gu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Germain et al., 2015] Mathieu Germain, Karol Gregor, Iain Mur- ray, and Hugo Larochelle. Made: Masked autoencoder for dis- tribution estimation. In International Conference on Machine Learning, pages 881-889, 2015.","title":"Made: Masked autoencoder for distribution estimation","authors":"Karol Mathieu Germain, Iain Gregor, Hugo Murray, Larochelle","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghasemipour et al., 2021] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682- 3691. PMLR, 2021.","title":"Emaq: Expectedmax q-learning operator for simple yet effective offline and online rl","authors":"Seyed Kamyar, Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy max- imum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861- 1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Janner et al., 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12519-12530, 2019.","title":"When to trust your model: Model-based policy optimization","authors":"Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kidambi et al., 2020] Rahul Kidambi, Aravind Rajeswaran, Pra- neeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, pages 21810-21823, 2020.","title":"Morel: Model-based offline reinforcement learning","authors":"Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kostrikov et al., 2021] Ilya Kostrikov, Rob Fergus, Jonathan Tomp- son, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, 2021.","title":"Offline reinforcement learning with fisher divergence critic regularization","authors":"Kostrikov","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In Advances in Neural Information Processing Systems, pages 1179-1191, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le et al., 2019] Hoang M Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Confer- ence on Machine Learning, pages 3703-3712. PMLR, 2019.","title":"Batch policy learning under constraints","authors":"Cameron Hoang M Le, Yisong Voloshin, Yue","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor poli- cies. The Journal of Machine Learning Research, 17(1):1334- 1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In Advances in Neural Infor- mation Processing Systems, pages 1264-1274, 2020.","title":"Provably good batch off-policy reinforcement learning without great exploration","authors":"Yao Liu, Adith Swaminathan, Alekh Agarwal, Emma Brunskill","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lowrey et al., 2019] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, 2019.","title":"Plan online, learn offline: Efficient learning and exploration via model-based control","authors":"Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nagabandi et al., 2020] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.","title":"Deep dynamics models for learning dexterous manipulation","authors":"Anusha Nagabandi, Kurt Konolige, Sergey Levine, Vikash Kumar","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Si- monyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mas- tering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Ba, 2020] Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In International Conference on Learning Representations, 2020.","title":"Exploring model-based planning with policy networks","authors":"Wang, Ba ;, Tingwu Wang, Jimmy Ba","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Williams et al., 2017] Grady Williams, Andrew Aldrich, and Evan- gelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344-357, 2017.","title":"Model predictive path integral control: From theory to parallel computation","authors":"Grady Williams, Andrew Aldrich, Evangelos A Theodorou","year":"2017","venue":"Journal of Guidance, Control, and Dynamics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Be- havior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2021] Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:2110.07395, 2021.","title":"Offline reinforcement learning with soft behavior regularization","authors":"Haoran Xu, Xianyuan Zhan, Jianxiong Li, Honglei Yin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2022] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.","title":"Constraints penalized q-learning for safe offline reinforcement learning","authors":"Haoran Xu, Xianyuan Zhan, Xiangyu Zhu","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, pages 14129-14142, 2020.","title":"Mopo: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhan et al., 2022] Xianyuan Zhan, Haoran Xu, Yue Zhang, Xi- angyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Com- bustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, 2022.","title":"Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning","authors":"Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, Yu Zheng","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}