{"type":"ScholarlyArticle","title":"COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens","authors":[{"name":"Eugene Kwek","slug":"eugene-kwek","given":"Eugene","family":"Kwek","affiliation":""},{"name":"Wenpeng Yin","slug":"wenpeng-yin","given":"Wenpeng","family":"Yin","affiliation":""}],"abstract":"Improving the memory efficiency, throughput, and serving cost of large language models (LLMs) is critical for edge deployment, interactive applications, and sustainable inference. Pruning is a promising approach, but existing methods have limitations: width pruning disrupts the standard transformer architecture and requires custom inference code, while depth pruning causes abrupt accuracy drops. Moreover, many approaches that work well for LLMs fail to preserve performance on small language models (SLMs). We propose COMPACT, which jointly prunes (i) rare vocabulary to shrink embedding layers and (ii) FFN intermediate channels using common-token–weighted activations aligned with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as deployment-friendliness, scale-adaptivity, competitive pruning speed, and strong inference performance. Experiments on several LLM families (0.5B–70B) show state-of-the-art downstream performance, with substantial improvements in inference throughput and GPU memory. Project code is at https://github.com/ekwek1/COMPACT.","doi":"10.24963/ijcai.2026/641","url":"https://ijcai.org/proceedings/2026/641","pdf":"https://ijcai.org/proceedings/2026/0641.pdf","markdown":"https://ijcai.org/proceedings/2026/641.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/641","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Natural Language Processing","track":"Main Track","number":641,"pages":{"first":5757,"last":5765},"keywords":["Natural Language Processing: Language models"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}