{"type":"ScholarlyArticle","title":"Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors","authors":[{"name":"Xinyu Ding","slug":"xinyu-ding","given":"Xinyu","family":"Ding","affiliation":""},{"name":"Lexuan Chen","slug":"lexuan-chen","given":"Lexuan","family":"Chen","affiliation":""},{"name":"Siyu Liao","slug":"siyu-liao","given":"Siyu","family":"Liao","affiliation":""},{"name":"Zhongfeng Wang","slug":"zhongfeng-wang","given":"Zhongfeng","family":"Wang","affiliation":""}],"abstract":"Foundation models have achieved tremendous success in different domains.\r\nHowever, their huge computation and storage complexity make these models difficult to fine-tune and also less applicable in practice. \r\nRecent study shows training in Fourier domain can be an effective fine-tuning method in terms of both model performance and number of training parameters. \r\nIn this work, we propose to further reduce the complexity by the factorization through the product of interleaved circulant and diagonal matrices. In addition, we address the case of non-square fine-tuning weights by partitioning the circulant matrix into blocks.\r\nOur method avoids the construction of weight change matrix and utilizes 1D fast Fourier transform (FFT) instead of 2D FFT. \r\nExperimental results show that our method achieves similar or better performance across various tasks with much less floating-point operations (FLOPs) and the number of trainable parameters.","doi":"10.24963/ijcai.2025/1021","url":"https://ijcai.org/proceedings/2025/1021","pdf":"https://ijcai.org/proceedings/2025/1021.pdf","markdown":"https://ijcai.org/proceedings/2025/1021.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1021","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"AI4Tech: AI Enabling Technologies","track":"AI4Tech: AI Enabling Technologies","number":1021,"pages":{"first":9185,"last":9193},"keywords":["Advanced AI4Tech: Deep AI4Tech","Advanced AI4Tech: Generative and LLMs-driven AI4Tech","Advanced AI4Tech: Creative AI4Tech"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}