{"type":"ScholarlyArticle","title":"Reinforcement Learning from Diverse Human Preferences","authors":[{"name":"Wanqi Xue","slug":"wanqi-xue","given":"Wanqi","family":"Xue","affiliation":"Nanyang Technological University"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"Nanyang Technological University; Skywork AI , Singapore"},{"name":"Shuicheng Yan","slug":"shuicheng-yan","given":"Shuicheng","family":"Yan","affiliation":"Skywork AI , Singapore"},{"name":"Zhongwen Xu","slug":"zhongwen-xu","given":"Zhongwen","family":"Xu","affiliation":"Tencent AI Lab"}],"abstract":"The complexity of designing reward functions has been a major obstacle to the wide application of deep reinforcement learning (RL) techniques. Describing an agent's desired behaviors and properties can be difficult, even for experts. A new paradigm called reinforcement learning from human preferences (or preference-based RL) has emerged as a promising solution, in which reward functions are learned from human preference labels among behavior trajectories. However, existing methods for preference-based RL are limited by the need for accurate oracle preference labels. This paper addresses this limitation by developing a method for learning from diverse human preferences. The key idea is to stabilize reward learning through regularization and correction in a latent space. To ensure temporal consistency, a strong constraint is imposed on the reward model that forces its latent space to be close to a non-parameterized distribution. Additionally, a confidence-based reward model ensembling method is designed to generate more stable and reliable predictions. The proposed method is tested on a variety of tasks in DMcontrol and Meta-world and has shown consistent and significant improvements over existing preference-based RL algorithms when learning from diverse feedback, paving the way for real-world applications of RL methods.","doi":"10.24963/ijcai.2024/586","url":"https://ijcai.org/proceedings/2024/586","pdf":"https://ijcai.org/proceedings/2024/0586.pdf","markdown":"https://ijcai.org/proceedings/2024/586.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/586","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":586,"pages":{"first":5298,"last":5306},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Akrour et al., 2011] Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pages 12-27. Springer, 2011.","title":"Preference-based policy learning","authors":"Akrour","year":"2011","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Biyik and Sadigh, 2018] Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward func- tions. In Conference on Robot Learning, pages 519-528.","title":"Batch active preference-based learning of reward functions","authors":"Sadigh Biyik, Erdem Biyik, Dorsa Sadigh","year":"2018","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bradley and Terry, 1952] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324- 345, 1952.","title":"Rank analysis of incomplete block designs: I. the method of paired comparisons","authors":"Terry ; Bradley, Allan Ralph, Milton E Bradley, Terry","year":"1952","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2021] Zehong Cao, KaiChiu Wong, and Chin- Teng Lin. Weak human preference supervision for deep reinforcement learning. IEEE Transactions on Neural Net- works and Learning Systems, 32(12):5369-5378, 2021.","title":"Weak human preference supervision for deep reinforcement learning","authors":"Cao","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Jan Paul F Christiano, Tom Leike, Miljan Brown, Shane Martic, Dario Legg, Amodei","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gerstgrasser et al., 2022] Matthias Gerstgrasser, Rakshit Trivedi, and David C. Parkes. Crowdplay: Crowdsourc- ing human demonstrations for offline learning. In Interna- tional Conference on Learning Representations, 2022.","title":"Crowdplay: Crowdsourcing human demonstrations for offline learning","authors":"Matthias Gerstgrasser, Rakshit Trivedi, David C Parkes","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hejna III and Sadigh, 2022] Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in- the-loop rl. In 6th Annual Conference on Robot Learning, 2022.","title":"Few-shot preference learning for human-inthe-loop rl","authors":"Iii Hejna, Donald Joseph Sadigh, Iii Hejna, Dorsa Sadigh","year":"2022","venue":"6th Annual Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ibarz et al., 2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 31, 2018.","title":"Reward learning from human preferences and demonstrations in atari","authors":"Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2021a] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.","title":"B-pref: Benchmarking preferencebased reinforcement learning","authors":"Lee","year":"2021","venue":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2021b] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforce- ment learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Con- ference on Machine Learning, pages 6152-6163, 2021.","title":"Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training","authors":"Lee","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leike et al., 2018] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.","title":"Scalable agent alignment via reward modeling: a research direction","authors":"Leike","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liang et al., 2022] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for explo- ration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022.","title":"Reward uncertainty for exploration in preference-based reinforcement learning","authors":"Xinran Liang, Katherine Shu, Kimin Lee, Pieter Abbeel","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Moravčík et al., 2017] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artifi- cial intelligence in heads-up no-limit poker. Science, 356(6337):508-513, 2017.","title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Park et al., 2022] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmenta- tion for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Repre- sentations, 2022.","title":"SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning","authors":"Park","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stiennon et al., 2020] Nisan Stiennon, Long Ouyang, Jef- frey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.","title":"Learning to summarize with human feedback","authors":"Stiennon","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tassa et al., 2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.","title":"Deepmind control suite","authors":"Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego De Las, David Casas, Abbas Budden, Josh Abdolmaleki, Andrew Merel, Lefrancq","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tunyasuvunakool et al., 2020] Saran Tunyasuvunakool, Al- istair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continu- ous control. Software Impacts, 6:100022, 2020.","title":"dm control: Software and tasks for continuous control","authors":"Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, Yuval Tassa","year":"2020","venue":"Software Impacts","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wilson et al., 2012] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in Neural Infor- mation Processing Systems, 25, 2012.","title":"A bayesian approach for policy learning from trajectory preference queries","authors":"Aaron Wilson, Alan Fern, Prasad Tadepalli","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wurman et al., 2022] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subrama- nian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing cham- pion gran turismo drivers with deep reinforcement learn- ing. Nature, 602(7896):223-228, 2022.","title":"Outracing champion gran turismo drivers with deep reinforcement learning","authors":"Samuel Peter R Wurman, Kenta Barrett, James Kawamoto, Kaushik Macglashan, Thomas J Subramanian, Roberto Walsh, Alisa Capobianco, Franziska Devlic, Florian Eckert, Fuchs","year":"2022","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xue et al., 2023] Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. Prefrec: Recommender systems with human preferences for reinforcing long-term user engage- ment. In Proceedings of the 29th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, KDD '23, page 2874-2884, 2023.","title":"Prefrec: Recommender systems with human preferences for reinforcing long-term user engagement","authors":"Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, Bo An","year":"2023","venue":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2020] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Confer- ence on Robot Learning, pages 1094-1100. PMLR, 2020.","title":"Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning","authors":"Yu","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2024] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization, 2024.","title":"Beyond one-preference-for-all: Multi-objective direct preference optimization","authors":"Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, Yu Qiao","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}