{"type":"ScholarlyArticle","title":"BATON: Aligning Text-to-Audio Model Using Human Preference Feedback","authors":[{"name":"Huan Liao","slug":"huan-liao","given":"Huan","family":"Liao","affiliation":"Tsinghua University"},{"name":"Haonan Han","slug":"haonan-han","given":"Haonan","family":"Han","affiliation":"Tsinghua University"},{"name":"Kai Yang","slug":"kai-yang","given":"Kai","family":"Yang","affiliation":"Tsinghua University"},{"name":"Tianjiao Du","slug":"tianjiao-du","given":"Tianjiao","family":"Du","affiliation":"Tsinghua University"},{"name":"Rui Yang","slug":"rui-yang","given":"Rui","family":"Yang","affiliation":"Tsinghua University"},{"name":"Qinmei Xu","slug":"qinmei-xu","given":"Qinmei","family":"Xu","affiliation":"Tsinghua University"},{"name":"Zunnan Xu","slug":"zunnan-xu","given":"Zunnan","family":"Xu","affiliation":"Tsinghua University"},{"name":"Jingquan Liu","slug":"jingquan-liu","given":"Jingquan","family":"Liu","affiliation":"Tsinghua University"},{"name":"Jiasheng Lu","slug":"jiasheng-lu","given":"Jiasheng","family":"Lu","affiliation":"Huawei Technologies Co. , Ltd."},{"name":"Xiu Li","slug":"xiu-li","given":"Xiu","family":"Li","affiliation":"Tsinghua University"}],"abstract":"With the development of AI-Generated Content (AIGC), text-to-audio models are gaining widespread attention. However, it is challenging for these models to generate audio aligned with human preference due to the inherent information density of natural language and limited model understanding ability. To alleviate this issue, we formulate the BATON, the first framework specifically designed to enhance the alignment between generated audio and text prompt using human preference feedback. Our BATON comprises three key stages: Firstly, we curated a dataset containing both prompts and the corresponding generated audio, which was then annotated based on human feedback. Secondly, we introduced a reward model using the constructed dataset, which can mimic human preference by assigning rewards to input text-audio pairs. Finally, we employed the reward model to fine-tune an off-the-shelf text-to-audio model. The experiment results demonstrate that our BATON can significantly improve the generation quality of the original text-to-audio models, concerning audio integrity, temporal relationship, and alignment with human preference. Project page is available at https://baton2024.github.io.","doi":"10.24963/ijcai.2024/502","url":"https://ijcai.org/proceedings/2024/502","pdf":"https://ijcai.org/proceedings/2024/0502.pdf","markdown":"https://ijcai.org/proceedings/2024/502.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/502","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":502,"pages":{"first":4542,"last":4550},"keywords":["Machine Learning: ML: Generative models","Natural Language Processing: NLP: Speech","Multidisciplinary Topics and Applications: MTA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.","title":"Gpt-4 technical report","authors":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anthropic, 2023] Anthropic. Introducing claude, 2023.","title":"Anthropic. Introducing claude","authors":"Anthropic","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Black et al., 2023] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023.","title":"Training diffusion models with reinforcement learning","authors":"Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Casper et al., 2023] Stephen Casper, Xander Davies, Clau- dia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental lim- itations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023.","title":"Open problems and fundamental limitations of reinforcement learning from human feedback","authors":"Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2023] Xuxin Cheng, Zhihong Zhu, Bowen Cao, Qichen Ye, and Yuexian Zou. Mrrl: Modify- ing the reference via reinforcement learning for non- autoregressive joint multiple intent detection and slot fill- ing. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 10495-10505, 2023.","title":"Mrrl: Modifying the reference via reinforcement learning for nonautoregressive joint multiple intent detection and slot filling","authors":"Cheng","year":"2023","venue":"Findings of the Association for Computational Linguistics: EMNLP 2023","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Jan Paul F Christiano, Tom Leike, Miljan Brown, Shane Martic, Dario Legg, Amodei","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.","title":"Raft: Reward ranked finetuning for generative foundation model alignment","authors":"Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghosal et al., 2023] Deepanway Ghosal, Navonil Ma- jumder, Ambuj Mehrish, and Soujanya Poria. Text-to- audio generation using instruction tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023. [Google, 2023] Google. Bard, 2023.","title":"Text-toaudio generation using instruction tuned llm and latent diffusion model","authors":"Ghosal","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2024] Zhifang Guo, Jianguo Mao, Rui Tao, Long Yan, Kazushige Ouchi, Hong Liu, and Xiangdong Wang. Audio generation with multiple conditional diffu- sion model. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 38, pages 18153-18161, 2024.","title":"Audio generation with multiple conditional diffusion model","authors":"Zhifang Guo, Jianguo Mao, Rui Tao, Long Yan, Kazushige Ouchi, Hong Liu, Xiangdong Wang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2023a] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023.","title":"Make-an-audio 2: Temporal-enhanced text-to-audio generation","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2023b] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an- audio: Text-to-audio generation with prompt-enhanced diffusion models. ICML, 2023.","title":"Make-anaudio: Text-to-audio generation with prompt-enhanced diffusion models","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kilgour et al., 2019] Kevin Kilgour, Mauricio Zuluaga, Do- minik Roblek, and Matthew Sharifi. Fréchet audio dis- tance: A reference-free metric for evaluating music en- hancement algorithms. In Interspeech 2019, Sep 2019.","title":"Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms","authors":"Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi","year":"2019","venue":"Interspeech 2019","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2019] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North, Jan 2019.","title":"Audiocaps: Generating captions for audios in the wild","authors":"Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, Gunhee Kim","year":"2019","venue":"Proceedings of the 2019 Conference of the North","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Ba, 2014] DiederikP. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: Learning,arXiv: Learning, Dec 2014.","title":"Adam: A method for stochastic optimization","authors":"Ba ; Kingma, Diederikp, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kong et al., 2020a] Jungil Kong, Jaehyeon Kim, and Jaeky- oung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.","title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis","authors":"Kong","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kong et al., 2020b] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, page 2880-2894, Jan 2020.","title":"Panns: Large-scale pretrained audio neural networks for audio pattern recognition","authors":"Kong","year":"2020","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kreuk et al., 2022] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audio- gen: Textually guided audio generation. arXiv preprint arXiv:2209.15352, 2022.","title":"Audiogen: Textually guided audio generation","authors":"Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, Yossi Adi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kreutzer et al., 2018] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa- pers), pages 1777-1788, Melbourne, Australia, July 2018. Association for Computational Linguistics.","title":"Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning","authors":"Julia Kreutzer, Joshua Uyheng, Stefan Riezler","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lee et al., 2023] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023.","title":"Aligning text-to-image models using human feedback","authors":"Lee","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2023a] Haohe Liu, Zehua Chen, Yi Yuan, Xin- hao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the Interna- tional Conference on Machine Learning, 2023.","title":"AudioLDM: Text-to-audio generation with latent diffusion models","authors":"Liu","year":"2023","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2023b] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308.05734, 2023.","title":"AudioLDM 2: Learning holistic audio generation with self-supervised pretraining","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in neural informa- tion processing systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ranzato et al., 2015] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. Computer Sci- ence, 2015.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shen et al., 2023] Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Bar- ret Zoph, William Fedus, Xinyun Chen, et al. Mixture- of-experts meets instruction tuning: A winning com- bination for large language models. arXiv preprint arXiv:2305.14705, 2023.","title":"Mixtureof-experts meets instruction tuning: A winning combination for large language models","authors":"Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stiennon et al., 2020] Nisan Stiennon, Long Ouyang, Jef- frey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.","title":"Learning to summarize with human feedback","authors":"Stiennon","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Van Den Oord et al., 2017] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu and Hu, 2018] Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learn- ing. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.","title":"Learning to extract coherent summary via deep reinforcement learning","authors":"Hu ; ] Yuxiang Wu, Baotian Wu, Hu","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2023] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dub- nov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.","title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation","authors":"Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov","year":"2023","venue":"ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xue et al., 2024] Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. arXiv preprint arXiv:2401.01044, 2024.","title":"Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation","authors":"Jinlong Xue, Yayue Deng, Yingming Gao, Ya Li","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2023a] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.","title":"Diffsound: Discrete diffusion model for text-to-sound generation","authors":"Yang","year":"2023","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2023b] Kai Yang, Jian Tao, Jiafei Lyu, Chun- jiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune dif- fusion models without any reward model. arXiv preprint arXiv:2311.13231, 2023.","title":"Using human feedback to fine-tune diffusion models without any reward model","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2023c] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self- instruction. arXiv preprint arXiv:2305.18752, 2023.","title":"Gpt4tools: Teaching large language model to use tools via selfinstruction","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yuan et al., 2024] Yi Yuan, Haohe Liu, Xubo Liu, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. Retrieval- augmented text-to-audio generation. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 581-585. IEEE, 2024.","title":"Retrievalaugmented text-to-audio generation","authors":"Yi Yuan, Haohe Liu, Xubo Liu, Qiushi Huang, Mark D Plumbley, Wenwu Wang","year":"2024","venue":"ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Ziegler et al., 2019] Daniel M Ziegler, Nisan Stiennon, Jef- frey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv:1909.08593, 2019.","title":"Fine-tuning language models from human preferences","authors":"Nisan Daniel M Ziegler, Jeffrey Stiennon, Tom B Wu, Alec Brown, Dario Radford, Paul Amodei, Geoffrey Christiano, Irving","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}