{"type":"ScholarlyArticle","title":"Block Circulant Adapter for Large Language Models","authors":[{"name":"Xinyu Ding","slug":"xinyu-ding","given":"Xinyu","family":"Ding","affiliation":""},{"name":"Meiqi Wang","slug":"meiqi-wang","given":"Meiqi","family":"Wang","affiliation":""},{"name":"Siyu Liao","slug":"siyu-liao","given":"Siyu","family":"Liao","affiliation":""},{"name":"Zhongfeng Wang","slug":"zhongfeng-wang","given":"Zhongfeng","family":"Wang","affiliation":""}],"abstract":"Fine-tuning large language models (LLMs) is difficult due to their  huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. \r\nWe propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs.\r\nExperiments show that our method uses 14× less number of parameters than VeRA, 16× smaller than LoRA and 32× less FLOPs than FourierFT,  while maintaining close or better task performance.\r\nOur approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.","doi":"10.24963/ijcai.2025/560","url":"https://ijcai.org/proceedings/2025/560","pdf":"https://ijcai.org/proceedings/2025/0560.pdf","markdown":"https://ijcai.org/proceedings/2025/560.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/560","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":560,"pages":{"first":5030,"last":5038},"keywords":["Natural Language Processing: NLP: Language models","Computer Vision: CV: Efficiency and Optimization","Machine Learning: ML: Matrix/tensor methods","Machine Learning: ML: Deep learning architectures"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}