Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models

Zhang, Yang; Yu, Yu; Tang, Bo; Zhu, Yu; Sun, Chuxiong; Wei, Wenqiang; Hu, Jie; Xie, Zipeng; Li, Zhiyu; Xiong, Feiyu; Chung, Edward

doi:10.24963/ijcai.2025/931

Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models

Yang Zhang, Yu Yu, Bo Tang, Yu Zhu, Chuxiong Sun, Wenqiang Wei, Jie Hu, Zipeng Xie, Zhiyu Li, Feiyu Xiong, Edward Chung

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 8375-8383.

https://doi.org/10.24963/ijcai.2025/931

Keywords

AI Ethics, Trust, Fairness: ETF: Ethical, legal and societal issues
AI Ethics, Trust, Fairness: ETF: Safety and robustness
Natural Language Processing: NLP: Language models

Abstract

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are “Accepted” or “Rejected” as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.

Yang Zhang, Yu Yu, Bo Tang, Yu Zhu, Chuxiong Sun, Wenqiang Wei, Jie Hu, Zipeng Xie, Zhiyu Li, Feiyu Xiong, and Edward Chung. 2025. Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 8375–8383. https://doi.org/10.24963/ijcai.2025/931

@inproceedings{ijcai2025p931,
  title     = {Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models},
  author    = {Zhang, Yang and Yu, Yu and Tang, Bo and Zhu, Yu and Sun, Chuxiong and Wei, Wenqiang and Hu, Jie and Xie, Zipeng and Li, Zhiyu and Xiong, Feiyu and Chung, Edward},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {8375--8383},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/931},
  url       = {https://doi.org/10.24963/ijcai.2025/931},
}

TY  - CPAPER
TI  - Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models
AU  - Zhang, Yang
AU  - Yu, Yu
AU  - Tang, Bo
AU  - Zhu, Yu
AU  - Sun, Chuxiong
AU  - Wei, Wenqiang
AU  - Hu, Jie
AU  - Xie, Zipeng
AU  - Li, Zhiyu
AU  - Xiong, Feiyu
AU  - Chung, Edward
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 8375
EP  - 8383
DO  - 10.24963/ijcai.2025/931
UR  - https://doi.org/10.24963/ijcai.2025/931
ER  -