{"type":"ScholarlyArticle","title":"On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models","authors":[{"name":"Xinpeng Wang","slug":"xinpeng-wang","given":"Xinpeng","family":"Wang","affiliation":"Tongji University"},{"name":"Shitong Duan","slug":"shitong-duan","given":"Shitong","family":"Duan","affiliation":"Fudan University"},{"name":"Xiaoyuan Yi","slug":"xiaoyuan-yi","given":"Xiaoyuan","family":"Yi","affiliation":"Microsoft Research Asia"},{"name":"Jing Yao","slug":"jing-yao","given":"Jing","family":"Yao","affiliation":"Microsoft Research Asia"},{"name":"Shanlin Zhou","slug":"shanlin-zhou","given":"Shanlin","family":"Zhou","affiliation":"Tongji University"},{"name":"Zhihua Wei","slug":"zhihua-wei","given":"Zhihua","family":"Wei","affiliation":"Tongji University"},{"name":"Peng Zhang","slug":"peng-zhang","given":"Peng","family":"Zhang","affiliation":"Fudan University"},{"name":"Dongkuan Xu","slug":"dongkuan-xu","given":"Dongkuan","family":"Xu","affiliation":"North Carolina State University"},{"name":"Maosong Sun","slug":"maosong-sun","given":"Maosong","family":"Sun","affiliation":"Tsinghua University"},{"name":"Xing Xie","slug":"xing-xie","given":"Xing","family":"Xie","affiliation":"Microsoft Research Asia"}],"abstract":"Big models have achieved revolutionary breakthroughs in the field of AI, but they also pose potential ethical and societal risks to humans. Addressing such problems, alignment technologies were introduced to make these models conform to human preferences and values. Despite the considerable advancements in the past year, various challenges lie in establishing the optimal alignment strategy, such as data cost and scalable oversight, and how to align remains an open question. In this survey paper, we comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges. Following this foundation, we provide a detailed examination of existing alignment methods, which fall into three categories: RL-based Alignment, SFT-based Alignment, and Inference-Time Alignment, and demonstrate their intrinsic connections, strengths, and limitations, helping readers better understand this research area. In addition, two emerging topics, alignment goal and multimodal alignment, are also discussed as novel frontiers in the field. Looking forward, we discuss potential alignment paradigms and how they could handle remaining challenges, prospecting where future alignment will go.","doi":"10.24963/ijcai.2024/918","url":"https://ijcai.org/proceedings/2024/918","pdf":"https://ijcai.org/proceedings/2024/0918.pdf","markdown":"https://ijcai.org/proceedings/2024/918.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/918","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":918,"pages":{"first":8308,"last":8316},"keywords":["AI Ethics, Trust, Fairness: ETF: Values","Natural Language Processing: NLP: Language models","Natural Language Processing: NLP: Language generation"],"videos":[],"references":[{"position":1,"raw":"[Asilomar, 2017] A.I. Asilomar. Asilomar ai principles, 2017.","title":"Asilomar ai principles","authors":"A I Asilomar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asimov, 1942] Isaac Asimov. Runaround, astouding sci- ence fiction. New York, 1942.","title":"Isaac Asimov. Runaround, astouding science fiction","authors":"Asimov","year":"1942","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Askell et al., 2021] Amanda Askell, Yuntao Bai, Anna Chen, et al. A general language assistant as a laboratory for alignment. ArXiv, 2021.","title":"A general language assistant as a laboratory for alignment","authors":"Amanda Askell, Yuntao Bai, Anna Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aydin, 2023] Omer Aydin. Google bard generated literature review: Metaverse. SSRN, 2023.","title":"Google bard generated literature review: Metaverse","authors":"Omer Aydin, Aydin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Azar et al., 2023] Mohammad Gheshlaghi Azar, Mark Rowland, et al. A general theoretical paradigm to under- stand learning from human preferences. ArXiv, 2023.","title":"A general theoretical paradigm to understand learning from human preferences","authors":"Mohammad Gheshlaghi Azar, Mark Rowland","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bai et al., 2022a] Yuntao Bai, Andy Jones, Kamal Ndousse, et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. ArXiv, 2022.","title":"Training a helpful and harmless assistant with reinforcement learning from human feedback","authors":"Bai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bai et al., 2022b] Yuntao Bai, Saurav Kadavath, et al. Con- stitutional ai: Harmlessness from ai feedback. ArXiv, 2022.","title":"Constitutional ai: Harmlessness from ai feedback","authors":"Bai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bhardwaj and Poria, 2023] Rishabh Bhardwaj and Sou- janya Poria. Red-teaming large language models using chain of utterances for safety-alignment. ArXiv, 2023.","title":"Red-teaming large language models using chain of utterances for safety-alignment","authors":"Poria ; Bhardwaj, Soujanya Bhardwaj, Poria","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bommasani et al., 2021] Rishi Bommasani, Drew A Hud- son, Ehsan Adeli, et al. On the opportunities and risks of foundation models. ArXiv, 2021.","title":"On the opportunities and risks of foundation models","authors":"Rishi Bommasani, Drew A Hudson, Ehsan Adeli","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bostrom, 2003] Nick Bostrom. Ethical issues in advanced artificial intelligence. Science fiction and philosophy: from time travel to superintelligence, 2003.","title":"Science fiction and philosophy: from time travel to superintelligence","authors":"Bostrom","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Bowman et al., 2022] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, et al. Measuring progress on scalable over- sight for large language models. ArXiv, 2022.","title":"Measuring progress on scalable oversight for large language models","authors":"Jeeyoon Samuel R Bowman, Ethan Hyun, Perez","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Carroll, 2018] Micah Carroll. Overview of current ai align- ment approaches. 2018.","title":"Micah Carroll. Overview of current ai alignment approaches","authors":"Carroll","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Casper et al., 2023] Stephen Casper, Xander Davies, et al. Open problems and fundamental limitations of reinforce- ment learning from human feedback. ArXiv, 2023.","title":"Open problems and fundamental limitations of reinforcement learning from human feedback","authors":"Stephen Casper, Xander Davies","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Christiano et al., 2017] Paul F. Christiano, Jan Leike, Tom B. Brown, et al. Deep reinforcement learning from human preferences. NeurIPS, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Paul F Christiano, Jan Leike, Tom B Brown","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Christiano, 2018] Paul Christiano. Clarifying ai alignment, 2018.","title":"Clarifying ai alignment","authors":"Paul Christiano","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dai et al., 2023] Wenliang Dai, Junnan Li, Dongxu Li, et al. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, 2023.","title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","authors":"Wenliang Dai, Junnan Li, Dongxu Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[de Font-Reaulx, 2022] Paul de Font-Reaulx. Alignment as a dynamic process. NeurIPS ML Safety Workshop, 2022. Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, et al. BERT: Pre-training of deep bidirectional transformers for language understanding. AACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":null,"year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Everitt and Hutter, 2018] Tom Everitt and Marcus Hutter. The alignment problem for bayesian history-based rein- forcement learners. Under submission, 2018.","title":"The alignment problem for bayesian history-based reinforcement learners","authors":"Hutter Everitt, Tom Everitt, Marcus Hutter","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gabriel, 2020] Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 2020.","title":"Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines","authors":"Gabriel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ganguli et al., 2023] Deep Ganguli, Amanda Askell, Nicholas Schiefer, et al. The capacity for moral self- correction in large language models. ArXiv, 2023.","title":"The capacity for moral selfcorrection in large language models","authors":"Ganguli","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Gehman et al., 2020] Samuel Gehman, Suchin Gururangan, et al. RealToxicityPrompts: Evaluating neural toxic de- generation in language models. EMNLP Findings, 2020.","title":"RealToxicityPrompts: Evaluating neural toxic degeneration in language models","authors":"Gehman","year":"2020","venue":"EMNLP Findings","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Go et al., 2023] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, et al. Aligning language models with prefer- ences through f-divergence minimization. ArXiv, 2023.","title":"Aligning language models with preferences through f-divergence minimization","authors":"Go","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Gong et al., 2023] Tao Gong, Chengqi Lyu, Shilong Zhang, et al. Multimodal-gpt: A vision and language model for dialogue with humans. ArXiv, 2023.","title":"Multimodal-gpt: A vision and language model for dialogue with humans","authors":"Tao Gong, Chengqi Lyu, Shilong Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Gou et al., 2023] Zhibin Gou, Zhihong Shao, Yeyun Gong, et al. Critic: Large language models can self-correct with tool-interactive critiquing. ArXiv, 2023.","title":"Critic: Large language models can self-correct with tool-interactive critiquing","authors":"Zhibin Gou, Zhihong Shao, Yeyun Gong","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Gulcehre et al., 2023] Caglar Gulcehre, Tom Le Paine, Sri- vatsan Srinivasan, et al. Reinforced self-training (rest) for language modeling. ArXiv, 2023.","title":"Reinforced self-training (rest) for language modeling","authors":"Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Stu- art J Russell, Pieter Abbeel, et al. Cooperative inverse reinforcement learning. NeurIPS, 2016.","title":"Cooperative inverse reinforcement learning","authors":"Hadfield-Menell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Han, 2023] Xiaochuang Han. In-context alignment: Chat with vanilla language models before fine-tuning. ArXiv, 2023.","title":"Xiaochuang Han. In-context alignment: Chat with vanilla language models before fine-tuning","authors":"Han","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Hejna et al., 2023] Joey Hejna, Rafael Rafailov, et al. Con- trastive prefence learning: Learning from human feedback without rl. ArXiv, 2023.","title":"Contrastive prefence learning: Learning from human feedback without rl","authors":"Hejna","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Horvitz and Selman, 2008] Eric Horvitz and Bart Selman. Aaai presidential panel on long-term ai futures: Interim report from the panel chairs. AAAI, 2008.","title":"Aaai presidential panel on long-term ai futures: Interim report from the panel chairs","authors":"Selman Horvitz, Eric Horvitz, Bart Selman","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ji et al., 2024] Jiaming Ji, Boyuan Chen, Hantao Lou, et al. Aligner: Achieving efficient alignment through weak-to- strong correction. ArXiv, 2024.","title":"Aligner: Achieving efficient alignment through weak-tostrong correction","authors":"Jiaming Ji, Boyuan Chen, Hantao Lou","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kaplan et al., 2020] Jared Kaplan, Sam McCandlish, et al. Scaling laws for neural language models. ArXiv, 2020.","title":"Scaling laws for neural language models","authors":"Jared Kaplan, Sam Mccandlish","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Kenton et al., 2021] Zachary Kenton, Tom Everitt, et al. Alignment of language agents. ArXiv, 2021.","title":"Alignment of language agents","authors":"Zachary Kenton, Tom Everitt","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Kim et al., 2023] Sungdong Kim, Sanghwan Bae, Jamin Shin, et al. Aligning large language models through syn- thetic feedback. ArXiv, 2023.","title":"Aligning large language models through synthetic feedback","authors":"Kim","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lee et al., 2023] Kimin Lee, Hao Liu, et al. Aligning text- to-image models using human feedback. ArXiv, 2023.","title":"Aligning textto-image models using human feedback","authors":"Lee","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Leike et al., 2018] Jan Leike, David Krueger, Tom Everitt, et al. Scalable agent alignment via reward modeling: a research direction. ArXiv, 2018.","title":"Scalable agent alignment via reward modeling: a research direction","authors":"Jan Leike, David Krueger, Tom Everitt","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Li et al., 2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, et al. Otter: A multi-modal model with in-context instruc- tion tuning. ArXiv, 2023.","title":"Otter: A multi-modal model with in-context instruction tuning","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Li et al., 2023b] Yuhui Li, Fangyun Wei, Jinjing Zhao, et al. Rain: Your language models can align themselves without finetuning. ArXiv, 2023.","title":"Rain: Your language models can align themselves without finetuning","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Lin et al., 2022] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. ACL, 2022.","title":"TruthfulQA: Measuring how models mimic human falsehoods","authors":"Stephanie Lin, Jacob Hilton, Owain Evans","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Lin et al., 2023] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, et al. The unlocking spell on base llms: Re- thinking alignment via in-context learning. ArXiv, 2023.","title":"The unlocking spell on base llms: Rethinking alignment via in-context learning","authors":"Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Liu et al., 2022] Ruibo Liu, Chenyan Jia, Ge Zhang, et al. Second thoughts are best: Learning to re-align with human values from text edits. NeurIPS, 2022.","title":"Second thoughts are best: Learning to re-align with human values from text edits","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Liu et al., 2023a] Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. ArXiv, 2023.","title":"Chain of hindsight aligns language models with feedback","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Liu et al., 2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, et al. Visual instruction tuning. ArXiv, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Liu et al., 2023c] Ruibo Liu, Ruixin Yang, Chenyan Jia, et al. Training socially aligned language models in sim- ulated human society. ArXiv, 2023.","title":"Training socially aligned language models in simulated human society","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[McKenzie et al., 2023] Ian R McKenzie, Alexander Lyzhov, Michael Pieler, et al. Inverse scaling: When bigger isn't better. ArXiv, 2023.","title":"Inverse scaling: When bigger isn't better","authors":"Alexander Ian R Mckenzie, Michael Lyzhov, Pieler","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Nakano et al., 2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, et al. Webgpt: Browser-assisted question- answering with human feedback. ArXiv, 2021.","title":"Webgpt: Browser-assisted questionanswering with human feedback","authors":"Nakano","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Nick, 2014] Bostrom Nick. Superintelligence: Paths, dan- gers, strategies. Oxford University Press, Oxford, 2014.","title":"Superintelligence: Paths, dangers, strategies","authors":"Bostrom Nick","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Rafailov et al., 2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. Direct preference optimization: Your lan- guage model is secretly a reward model. ArXiv, 2023.","title":"Direct preference optimization: Your language model is secretly a reward model","authors":"Rafael Rafailov, Archit Sharma, Eric Mitchell","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Sheng et al., 2019] Emily Sheng, Kai-Wei Chang, et al. The woman worked as a babysitter: On biases in language gen- eration. EMNLP, 2019.","title":"The woman worked as a babysitter: On biases in language generation","authors":"Emily Sheng, Kai-Wei Chang","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Skalse et al., 2022] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, et al. Defining and characterizing reward gaming. NeurIPS, 2022.","title":"Defining and characterizing reward gaming","authors":"Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Soares and Fallenstein, 2014] Nate Soares and Benja Fall- enstein. Aligning superintelligence with human interests: A technical research agenda. MIRI technical report, 2014.","title":"Nate Soares and Benja Fallenstein. Aligning superintelligence with human interests: A technical research agenda","authors":"Fallenstein Soares","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Song et al., 2023] Feifan Song, Bowen Yu, et al. Preference ranking optimization for human alignment. ArXiv, 2023. Stiennon et al., 2020] Nisan Stiennon, Long Ouyang, Jef- frey Wu, et al. Learning to summarize with human feed- back. NeurIPS, 2020.","title":"Preference ranking optimization for human alignment","authors":"Song","year":"2020","venue":"Learning to summarize with human feedback","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Sun et al., 2023] Zhiqing Sun, Yikang Shen, Qinhong Zhou, et al. Principle-driven self-alignment of language models from scratch with minimal human supervision. ArXiv, 2023.","title":"Principle-driven self-alignment of language models from scratch with minimal human supervision","authors":"Zhiqing Sun, Yikang Shen, Qinhong Zhou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Team et al., 2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, et al. Gemini: a family of highly capable mul- timodal models. ArXiv, 2023.","title":"Gemini: a family of highly capable multimodal models","authors":"Team","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ArXiv, 2018.","title":"Behavioral cloning from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":57,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, et al. Llama: Open and efficient foundation language models. ArXiv, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Waldrop, 1987] M Mitchell Waldrop. A question of respon- sibility. AI Magazine, 1987.","title":"A question of responsibility","authors":"Mitchell Waldrop","year":"1987","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Wang et al., 2022] Yizhong Wang, Yeganeh Kordi, Swa- roop Mishra, et al. Self-instruct: Aligning language model with self generated instructions. ArXiv, 2022.","title":"Self-instruct: Aligning language model with self generated instructions","authors":"Yizhong Wang, Yeganeh Kordi, Swaroop Mishra","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Wang et al., 2023] Yidong Wang, Zhuohao Yu, Zhengran Zeng, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. ArXiv, 2023.","title":"Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization","authors":"Yidong Wang, Zhuohao Yu, Zhengran Zeng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Wei et al., 2022] Jason Wei, Yi Tay, Rishi Bommasani, et al. Emergent abilities of large language models. ArXiv, 2022.","title":"Emergent abilities of large language models","authors":"Jason Wei, Yi Tay, Rishi Bommasani","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Weidinger et al., 2022] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, et al. Taxonomy of risks posed by lan- guage models. ACM FAccT, 2022.","title":"Taxonomy of risks posed by language models","authors":"Laura Weidinger, Jonathan Uesato, Maribeth Rauh","year":"2022","venue":"ACM FAccT","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Wiener, 1960] Norbert Wiener. Some moral and technical consequences of automation: As machines learn they may develop unforeseen strategies at rates that baffle their pro- grammers. Science, 1960.","title":"Some moral and technical consequences of automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers","authors":"Norbert Wiener, Wiener","year":"1960","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Wu et al., 2023] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. ArXiv, 2023.","title":"Better aligning text-to-image models with human preference","authors":"Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Xie et al., 2021] Sang Michael Xie, Aditi Raghunathan, et al. An explanation of in-context learning as implicit bayesian inference. ICLR, 2021.","title":"An explanation of in-context learning as implicit bayesian inference","authors":"Sang Michael Xie, Aditi Raghunathan","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Xu et al., 2023] Chunpu Xu, Steffi Chern, Ethan Chern, et al. Align on the fly: Adapting chatbot behavior to estab- lished norms. ArXiv, 2023.","title":"Align on the fly: Adapting chatbot behavior to established norms","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Yuan et al., 2023] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, et al. Rrhf: Rank responses to align language models with human feedback without tears. ArXiv, 2023.","title":"Rrhf: Rank responses to align language models with human feedback without tears","authors":"Zheng Yuan, Hongyi Yuan, Chuanqi Tan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Zhao et al., 2023a] Wayne Xin Zhao, Kun Zhou, Junyi Li, et al. A survey of large language models. ArXiv, 2023.","title":"A survey of large language models","authors":"Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Zhao et al., 2023b] Yao Zhao, Rishabh Joshi, Tianqi Liu, et al. Slic-hf: Sequence likelihood calibration with human feedback. ArXiv, 2023.","title":"Slic-hf: Sequence likelihood calibration with human feedback","authors":"Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Zhou et al., 2023] Chunting Zhou, Pengfei Liu, Puxin Xu, et al. Lima: Less is more for alignment. ArXiv, 2023.","title":"Lima: Less is more for alignment","authors":"Zhou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, 2023.","title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}