{"type":"ScholarlyArticle","title":"Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation","authors":[{"name":"Woo Kyung Kim","slug":"woo-kyung-kim","given":"Woo Kyung","family":"Kim","affiliation":"Department of Computer Science and Engineering , Sungkyunkwan University"},{"name":"Minjong Yoo","slug":"minjong-yoo","given":"Minjong","family":"Yoo","affiliation":"Department of Computer Science and Engineering , Sungkyunkwan University"},{"name":"Honguk Woo","slug":"honguk-woo","given":"Honguk","family":"Woo","affiliation":"Department of Computer Science and Engineering , Sungkyunkwan University"}],"abstract":"Data-driven offline reinforcement learning and imitation learning approaches have been gaining popularity in addressing sequential decision-making problems. Yet, these approaches rarely consider learning Pareto-optimal policies from a limited pool of expert datasets. This becomes particularly marked due to practical limitations in obtaining comprehensive datasets for all preferences, where multiple conflicting objectives exist and each expert might hold a unique optimization preference for these objectives. In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. This enables progressive generation of a set of policies that accommodate diverse preferences on the multiple objectives, while using only two distinct datasets, each associated with a different expert preference. In doing so, we present a Pareto IRL framework (ParIRL) that establishes a Pareto policy set from these limited datasets. In the framework, the Pareto policy set is then distilled into a single, preference-conditioned diffusion model, thus allowing users to immediately specify which expert's patterns they prefer. Through experiments, we show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks, achieving the dense approximation of the Pareto frontier. We also demonstrate the applicability of ParIRL with autonomous driving in CARLA.","doi":"10.24963/ijcai.2024/475","url":"https://ijcai.org/proceedings/2024/475","pdf":"https://ijcai.org/proceedings/2024/0475.pdf","markdown":"https://ijcai.org/proceedings/2024/475.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/475","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":475,"pages":{"first":4300,"last":4307},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Dosovitskiy et al., 2017] Alexey Dosovitskiy, German Ros, Flelipe Codevilla, Antonio López, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceed- ings of the 1st Conference on Robot Learning, pages 1-16, 2017.","title":"CARLA: An open urban driving simulator","authors":"Alexey Dosovitskiy, German Ros, Flelipe Codevilla, Antonio López, Vladlen Koltun","year":"2017","venue":"Proceedings of the 1st Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fu et al., 2018] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. In Proceedings of the 6th Interna- tional Conference on Learning Representations, 2018.","title":"Learning robust rewards with adversarial inverse reinforcement learning","authors":"Justin Fu, Katie Luo, Sergey Levine","year":"2018","venue":"Proceedings of the 6th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Garg et al., 2021] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ- Learn: Inverse soft-Q Learning for Imitation. In Proceed- ings of the 35th Conference on Neural Information Pro- cessing Systems, pages 4028-4039, 2021.","title":"IQ-Learn: Inverse soft-Q Learning for Imitation","authors":"Garg","year":"2021","venue":"Proceedings of the 35th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gleave et al., 2020] Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differ- ences in reward functions. In Proceedings of the 8th Inter- national Conference on Learning Representations, 2020.","title":"Quantifying differences in reward functions","authors":"Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, Jan Leike","year":"2020","venue":"Proceedings of the 8th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proceedings of the 30th Conference on Neural Information Processing Systems, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Proceedings of the 30th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho and Salimans, 2022] Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance, 2022.","title":"Classifier-Free Diffusion Guidance","authors":"Jonathan Ho, Tim Salimans","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[HO et al., 2020] Jonathan HO, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Proceedings of the 34th Conference on Neural Informa- tion Processing System, 2020.","title":"Denoising Diffusion Probabilistic Models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Proceedings of the 34th Conference on Neural Information Processing System","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kishikawa and Arai, 2021] Daiko Kishikawa and Sachiyo Arai. Multi-objective inverse reinforcement learning via non-negative matrix factorization. In 10th International Congress on Advanced Applied Informatics, pages 452- 457, 2021.","title":"Multi-objective inverse reinforcement learning via non-negative matrix factorization","authors":"Arai Kishikawa, Daiko Kishikawa, Sachiyo Arai","year":"2021","venue":"10th International Congress on Advanced Applied Informatics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kyriakis et al., 2022] Panagiotis Kyriakis, Jyotirmoy v. Deshmukh, and Paul Bogdan. Pareto policy adaptation. In Proceedings of the 10th International Conference on Learning Representations, 2022.","title":"Pareto policy adaptation","authors":"Panagiotis Kyriakis, Jyotirmoy V Deshmukh, Paul Bogdan","year":"2022","venue":"Proceedings of the 10th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pearce et al., 2023] Tim Pearce, Tabish Rashid, Anssi Kan- ervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momenne- jad, Katja Hofmann, and Sam Devlin. Imitating Human Behaviour with Diffusion Models. In Proceddings of the 11th Conference on Learning Representations, 2023. [Shafiullah et al., 2022] Nur Muhammad Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior Transformers: Cloning k modes with one stone. In Proceedings of the 37th Conference on Nueral Information Processing Systems, 2022.","title":"Behavior Transformers: Cloning k modes with one stone","authors":"Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, Sam Devlin, ; Shafiullah","year":"2022","venue":"Proceedings of the 37th Conference on Nueral Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Villani, 2003] Cédric Villani. Topics in optimal transporta- tion, volume 58. American Mathematical Society, 2003.","title":"Topics in optimal transportation","authors":"Cédric Villani, Villani","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Wang et al., 2024] Bingzheng Wang, Guoqiang Wu, Teng Pang, Yan Zhang, and Yilong Yin. DiffAIL: Diffusion Ad- versarial Imitation Learning. In Proceedings of the 38th Conference on Artificial Intelligence, pages 4028-4039, 2024.","title":"DiffAIL: Diffusion Adversarial Imitation Learning","authors":"Bingzheng Wang, Guoqiang Wu, Teng Pang, Yan Zhang, Yilong Yin","year":"2024","venue":"Proceedings of the 38th Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Wulfe et al., 2022] Blake Wulfe, Logan Ellis, Jean Mercat, Rowan McAllister, Adrien Gaidon, and Ashwin Balakr- ishna. Dynamics-aware comparison of learned reward functions. In Proceedings of the 10th International Con- ference on Learning Representations, 2022.","title":"Dynamics-aware comparison of learned reward functions","authors":"Blake Wulfe, Logan Ellis, Jean Mercat, Rowan Mcallister, Adrien Gaidon, Ashwin Balakrishna","year":"2022","venue":"Proceedings of the 10th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Xu et al., 2020] Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning, pages 10608-10616, 2020.","title":"Prediction-guided multi-objective reinforcement learning for continuous robot control","authors":"Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, Wojciech Matusik","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Yang et al., 2019] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. In Proceedings of the 33rd Conference on Neural Informa- tion Processing Systems, pages 14610-14621, 2019.","title":"A generalized algorithm for multiobjective reinforcement learning and policy adaptation","authors":"Yang","year":"2019","venue":"Proceedings of the 33rd Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zeng et al., 2022] Siliang Zeng, Chenliang Li, Alfredo Gar- cia, and Mingyi Hong. Maximum-Likelihood Inverse Re- inforcement Learning with Finite-Time Guarantees. In Proceedings of the 37th Conference on Nueral Informa- tion Processing Systems, 2022.","title":"Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees","authors":"Zeng","year":"2022","venue":"Proceedings of the 37th Conference on Nueral Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Zhu et al., 2023] PBaiting Zhu, Meihua Dang, and Aditya Grover. Scaling Pareto-Efficient Decision Making Via Of- fline Multi-Objective RL. In Proceedings of the 11th Inter- national Conference on Learning Representations, 2023.","title":"Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL","authors":"Zhu","year":"2023","venue":"Proceedings of the 11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}