{"type":"ScholarlyArticle","title":"TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning","authors":[{"name":"Dabiao Ma","slug":"dabiao-ma","given":"Dabiao","family":"Ma","affiliation":""},{"name":"Ziming Dai","slug":"ziming-dai","given":"Ziming","family":"Dai","affiliation":""},{"name":"Zhimin Xin","slug":"zhimin-xin","given":"Zhimin","family":"Xin","affiliation":""},{"name":"Shu Wang","slug":"shu-wang","given":"Shu","family":"Wang","affiliation":""},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":""},{"name":"Haojun Fei","slug":"haojun-fei","given":"Haojun","family":"Fei","affiliation":""}],"abstract":"Current Parameter-Efficient Fine-Tuning (PEFT) methods typically operate under an implicit assumption: once a target module is selected, every token passing through it contributes equally to the downstream task and requires a parameter update.  In this paper, we challenge this convention by revealing a pervasive token-level redundancy in the fine-tuning of large models (LMs). We propose TS-PEFT, a theoretical framework utilizing proximal optimization that acts as a dynamic probe to identify token-level redundancy during the fine-tuning process.  Extensive experiments demonstrate that indiscriminately updating all tokens is not only computationally superfluous but often introduces optimization noise.  Surprisingly, by discarding 30%-70% of token updates, TS-PEFT consistently matches or exceeds the performance of dense baselines such as LoRA, DoRA. Our in-depth analysis shows that the learned token-level sparsity is a superior indicator of module importance compared to traditional weight criteria, providing a novel data-driven perspective on the intrinsic adaptation mechanism of LMs.","doi":"10.24963/ijcai.2026/514","url":"https://ijcai.org/proceedings/2026/514","pdf":"https://ijcai.org/proceedings/2026/0514.pdf","markdown":"https://ijcai.org/proceedings/2026/514.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/514","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Machine Learning","track":"Main Track","number":514,"pages":{"first":4616,"last":4624},"keywords":["Natural Language Processing: Interpretability and analysis of models for NLP","Machine Learning: Learning sparse models","Machine Learning: Multi-task and transfer learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}