Proceedings

Block Circulant Adapter for Large Language Models

Xinyu Ding, Meiqi Wang, Siyu Liao, Zhongfeng Wang

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 5030-5038.

https://doi.org/10.24963/ijcai.2025/560

Keywords

  • Natural Language Processing: NLP: Language models
  • Computer Vision: CV: Efficiency and Optimization
  • Machine Learning: ML: Matrix/tensor methods
  • Machine Learning: ML: Deep learning architectures

Abstract

Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs.

We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs.

Experiments show that our method uses 14× less number of parameters than VeRA, 16× smaller than LoRA and 32× less FLOPs than FourierFT, while maintaining close or better task performance.

Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.

Xinyu Ding, Meiqi Wang, Siyu Liao, and Zhongfeng Wang. 2025. Block Circulant Adapter for Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 5030–5038. https://doi.org/10.24963/ijcai.2025/560
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards