{"type":"ScholarlyArticle","title":"Indirect Online Preference Optimization via Reinforcement Learning","authors":[{"name":"En Wang","slug":"en-wang","given":"En","family":"Wang","affiliation":""},{"name":"Xingyu Lin","slug":"xingyu-lin","given":"Xingyu","family":"Lin","affiliation":""},{"name":"Du Su","slug":"du-su","given":"Du","family":"Su","affiliation":""},{"name":"Chenfu Bao","slug":"chenfu-bao","given":"Chenfu","family":"Bao","affiliation":""},{"name":"Zhonghou Lv","slug":"zhonghou-lv","given":"Zhonghou","family":"Lv","affiliation":""},{"name":"Funing Yang","slug":"funing-yang","given":"Funing","family":"Yang","affiliation":""},{"name":"Yuanbo Xu","slug":"yuanbo-xu","given":"Yuanbo","family":"Xu","affiliation":""},{"name":"Wenbin Liu","slug":"wenbin-liu","given":"Wenbin","family":"Liu","affiliation":""}],"abstract":"Human preference alignment (HPA) aims to ensure Large Language Models (LLMs) responding appropriately to meet human moral and ethical requirements. Existing methods, such as RLHF and DPO, rely heavily on high-quality human annotation, which restrict the efficiency of iterative online model refinement.\r\n\t\tTo address the inefficiencies of human annotation acquisition, iterated online strategy advocates the use of fine-tuned LLMs to self-generate preference data. However, this approach is prone to distribution bias, because of differences between human and model annotations, as well as modeling errors between simulators and real-world contexts. To mitigate the impact of distribution bias, we adopt the principles of adversarial training, framing a zero-sum two-player game with a protagonist agent and an adversarial agent. With the adversarial agent challenging the alignment of protagonist agent, we continuously refine the protagonist’s performance. By utilizing min-max equilibrium and Nash equilibrium strategies, we propose Indirect Online Preference Optimization (IOPO) mechanism that enables the protagonist agent to converge without bias while maintaining linear computational complexity. Extensive experiments across three real-world datasets demonstrate that IOPO outperforms state-of-the-art alignment methods in both offline and online scenarios, evidenced by standard alignment metrics and human evaluations. This innovation reduces the time required for model iterations from months to one week, alleviates distribution shifts, and significantly cuts annotation costs.","doi":"10.24963/ijcai.2025/61","url":"https://ijcai.org/proceedings/2025/61","pdf":"https://ijcai.org/proceedings/2025/0061.pdf","markdown":"https://ijcai.org/proceedings/2025/61.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/61","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":61,"pages":{"first":538,"last":546},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Agent theories and models","AI Ethics, Trust, Fairness: ETF: AI and law, governance, regulation"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}