{"type":"ScholarlyArticle","title":"Unlocking More Granular Control of Memory-Efficient LLM Finetuning","authors":[{"name":"Yezhen Wang","slug":"yezhen-wang","given":"Yezhen","family":"Wang","affiliation":""},{"name":"Zhouhao Yang","slug":"zhouhao-yang","given":"Zhouhao","family":"Yang","affiliation":""},{"name":"Fanyi Pu","slug":"fanyi-pu","given":"Fanyi","family":"Pu","affiliation":""},{"name":"Kenji Kawaguchi","slug":"kenji-kawaguchi","given":"Kenji","family":"Kawaguchi","affiliation":""}],"abstract":"Low-rank gradient projection (LoRP) has recently emerged as a memory-efficient alternative to low-rank adapters (LoRA) for finetuning large language models. Existing LoRP methods, however, implicitly fix the projection unit to a single gradient row, leaving the effect of grouping multiple rows (or subdividing a row) largely unexplored. In this work, we systematically investigate the impact of the projection unit on LoRP methods. Specifically, we extend existing LoRP approaches by introducing an additional degree of freedom, projection granularity, beyond the traditional rank hyperparameter. This enables a framework capable of performing Various-grained Low-Rank Projection of gradients, which we term VLoRP. Using VLoRP, we observe that, under an identical memory budget, fine-grained projections consistently deliver superior performance. Moreover, VLoRP requires no extra computation and minimal code changes, effectively providing a no-cost accuracy boost to LoRP. Finally, we provide convergence analysis on VLoRP with either SGD or an Adam-based memory-efficient optimizer, and extensive experiments are conducted to validate our findings, covering tasks such as Commonsense Reasoning, MMLU, and GSM8K","doi":"10.24963/ijcai.2026/555","url":"https://ijcai.org/proceedings/2026/555","pdf":"https://ijcai.org/proceedings/2026/0555.pdf","markdown":"https://ijcai.org/proceedings/2026/555.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/555","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Machine Learning","track":"Main Track","number":555,"pages":{"first":4985,"last":4993},"keywords":["Machine Learning: Matrix/tensor methods","Machine Learning: Optimization","Machine Learning: Other"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}