COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens

1 Introduction

LLMs have achieved strong performance across many NLP tasks, but their rapidly growing parameter counts make deployment costly. To enable real-world use cases such as ondevice inference and latency-sensitive systems, it is crucial to compress LLMs while preserving performance. Major compression approaches include quantization [Frantar et al., 2022; Lin et al., 2024] and pruning [Frantar and Alistarh, 2023; Sun et al., 2024; Yin et al., 2024; Ramachandran et al., 2025]. This work focuses on structured pruning, which removes entire rows or columns of weight matrices. Structured pruning is divided into depth and width pruning. Depth pruning removes entire transformer blocks [Kim et al., 2024; Song et al., 2024; Gromov et al., 2025], but often causes sharp performance drops. Width pruning trims FFN channels or attention heads [Ma et al., 2023; Ashkboos et al., 2024; An et al., 2024], but typically deviates from a transformer architecture and requires custom inference code. Existing methods also suffer from three key limitations: (i) They fail to consider where parameters are concentrated (embeddings, FFNs, or attention), causing poor transfer across scales, especially from LLMs to SLMs. (ii) They ignore linguistic structure, treating all tokens as equally important despite large frequency disparities. (iii) They require model-specific implementation changes, making maintenance cumbersome. These issues lead to non-robust performance.

To address this, we propose COMPACT, a pruning framework with two modules: i) Vocabulary pruning removes rare tokens and shrinks embedding layers, yielding substantial parameter and memory reductions, especially for SLMs. ii) Common-token-weighted FFN pruning scores channels using activations weighted by only the tokens retained after vocabulary pruning. Together, these modules guide pruning by parameter distribution, respect linguistic structure, preserve compatibility with existing inference frameworks, and remain architecture-agnostic across model families.

We analyze parameter distributions across model families and scales, revealing that embeddings dominate SLMs while FFNs dominate LLMs. We further observe that token frequencies follow a Zipfian distribution [Zhemchuzhina et al., 2022], implying that rare tokens contribute little to downstream performance. Removing these tokens reduces embedding size with minimal accuracy loss. These findings motivate and validate COMPACT.

We evaluate COMPACT on Qwen 2.5, LLaMA 3, and Gemma 3 models from 0.5B to 70B parameters, using seven benchmarks: MMLU [Hendrycks et al., 2021], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2020], ARC-C/E [Clark et al., 2018], PIQA [Bisk et al., 2020], and GSM8K [Cobbe et al., 2021]. We also measure pruning time, inference throughput, and GPU memory usage. Results show: (i) Scale robustness: strong performance at high pruning ratios, even for SLMs. (ii) Smooth degradation: gradual performance decline, unlike depth pruning. (iii) End-to-end efficiency: significant GPU memory savings and improved throughput.

Our contributions are threefold:

• A systematic analysis of parameter distributions across embeddings, FFNs, and attention, revealing scaledependent redundancy.
• COMPACT, a linguistically grounded, scale-adaptive, and structure-agnostic pruning method.

• State-of-the-art pruning results across LLM families and scales, with strong downstream retention and gains in efficiency, pruning time, and GPU memory usage.

2 RelatedWork

Depth Pruning removes entire transformer blocks while preserving compatibility with common inference frameworks [He et al., 2024; Lu et al., 2024]. Representative methods include Shortened LLaMA (perplexity-minimizing), SLEB (iterative recalibration), and angular-similarity pruning [Kim et al., 2024; Song et al., 2024; Gromov et al., 2025]. LLMStreamline trains a lightweight network to recover accuracy but requires significant hardware and time [Chen et al., 2025]. We therefore focus on training-free pruning instead, which runs in minutes on a single GPU.

Width Pruning reduces transformer block size [Xia et al., 2024; Gao et al., 2024; Guo et al., 2025]. Methods include LLM-Pruner/LoRAPrune (gradient-based) [Ma et al., 2023; Zhang et al., 2024], SliceGPT (orthogonal transforms + lowrank) [Ashkboos et al., 2024], FLAP (stability-based) [An et al., 2024], Bonsai (perturbation modeling) [Dery et al., 2024], SlimGPT (non-uniform) [Ling et al., 2024], MoDeGPT (matrix decomposition) [Lin et al., 2025]. Width pruning often breaks standard transformer layout, requiring custom inference code and limiting deployment. COMPACT avoids this by preserving the architecture, yielding a deployment-friendly width-pruning method that outperforms depth pruning.

Vocabulary Pruning reduces vocabulary size, often to tailor models to a specific languages/domains [Ushio et al., 2023a; Dorkin et al., 2025; Ushio et al., 2023b; Bogoychev et al., 2024; Yang et al., 2022; Yang et al., 2024b]. Others prune draft models for speculative decoding speedups [Zhao et al., 2025; Goel et al., 2025], which does not compress the base model used at inference. In contrast, we perform non-domainspecific vocabulary pruning for general-purpose LLMs, which, to our knowledge, is a novel technique.

3 ProposedMethod: COMPACT

Before designing an effective pruning strategy (Sections 3.23.4 ), we first analyze where parameters are concentrated within modern decoder-only transformers (Section 3.1).

3.1 Parameter Distribution in LLMs

Mainstream LLMs consist of three major groups of parameters: (i) vocab parameters, located in the embedding and LM head layers; (ii) attention parameters, from the self-attention blocks; and (iii) FFN parameters, from the feed-forward blocks.

Formally, the embedding and LM head layers map between the vocabulary space of size V and the hidden dimension D, giving

(or VD if tied embeddings are used). Each FFN block contains three projection matrices of size D × I, where I is the intermediate dimension, yielding

for L layers. For attention, the number of parameters depends on whether grouped query attention is used [Ainslie et al., 2023]. With H denoting the ratio of attention heads to KV heads, the count is

Asymptotically, NFFN and Nattention scale as O(LD2 )-I ≈ O(D)-while Nvocab scales only as O(D), as V is kept constant when scaling. Thus, as model size grows, vocab parameters become proportionally smaller. Conversely, for smaller models, vocab parameters can constitute a significant fraction of the total. We observe this empirically by calculating the relative proportions of each parameter group on popular model families. Figure 1 shows our empirical analysis on the Qwen 2.5 model family [Yang et al., 2024a], which validates our theoretical analysis. Proportions of other model families can be found in Appendix A.1. This motivates our strategy: vocabulary pruning is an efficient way to reduce parameters in SLMs, while FFN pruning is critical for LLMs.

3.2 Vocabulary Pruning

Byte-Pair Encoding (BPE) tokenizers follow Zipf's law [Zhemchuzhina et al., 2022], where most tokens appear extremely rarely. Since BPE builds its vocabulary by merging frequent token pairs, the rarest tokens naturally appear at the end of the vocabulary list.

Let V ′ be the target vocabulary size. We define the set S as the V -V ′ rarest tokens in the vocabulary. These tokens can be directly removed by pruning the corresponding rows in the embedding/LM head matrices and deleting the corresponding merge rules from the tokenizer. VOCAB-PRUNING is highly efficient: it requires no calibration data or forward passes.

3.3 Intermediate Pruning

Pruning vocabulary parameters alone reduces the embedding size but does not address redundancies in the FFNs, which dominate parameter count in large models. To prune FFNs, we adopt an activation-based criterion. Assuming the FFN utilizes a SiLU activation function, the standard act2 method [Muralidharan et al., 2024; Sandri et al., 2025] defines the importance of FFN intermediate channel k as Ik = ∑Ni=1 (SiLU(XiWgate)XiWup )2k , summing squared activations over a calibration dataset. Here, Xi is FFN input and

Algorithm 1 COMPACT

Require: Model M, calibration dataset D, target vocabulary size V ′ , target intermediate size I ′

• 1: Identify S ← set of V -V ′ rarest tokens in vocabulary.
• 2: Run forward passes of M on D, collect squared activations.
• 3: For each channel k, compute importance Ik using common act2 (Eq. 4).
• 4: for each layer do
• 5: Prune I - I ′ least important channels (remove rows of Wgate,Wup and columns of Wdown).
• 6: end for
• 7: Prune vocab parameters: remove final V -V ′ rows of embedding and LM head matrices; delete tokenizer merges for tokens in S.
• 8: return pruned model M ′ .

Wgate,Wup are model weights. However, this equally weights all tokens xi, including xi ∈ S. Since such tokens will never appear in the input after pruning, their activations should not guide channel importance. We therefore introduce common act2 , a weighted variant:

This ensures FFN pruning explicitly targets the remaining valid tokens. All FFN layers use the same pruning ratio.

3.4 COMPACT: Joint Pruning Pipeline

Our proposed method, COMPACT, integrates vocabulary pruning with common act2 -based FFN pruning. Importantly, embedding pruning and channel pruning are not performed sequentially in isolation: knowledge of S (rarest tokens) is first identified, then used to guide intermediate pruning, and finally both vocab and FFN parameters are removed. The full pipeline is given in Algorithm 1.

Advantages of COMPACT. i) COMPACT is scale-adaptive. COMPACT uses two different knobs for pruning: (i) vocabulary pruning on embedding layers and (ii) commontoken-weighted pruning of FFN intermediate channels. These two knobs are orthogonal, allowing COMPACT to be tunable for SLMs (emphasize vocab pruning), LLMs (emphasize intermediate pruning), or any mix to meet a target budget. This tunability preserves capacity on frequent tokens while enabling strong compression across model scales. ii) COMPACT is compatible with LLM frameworks. One weakness of most width pruning methods is that they do not maintain a standard transformer architecture. This is indeed the case with SliceGPT, which prunes the hidden size in all layers except the final one, as well as 2SSP, which prunes entire attention modules. As a result, these methods are not compatible with transformers, vLLM, or other inference engines, limiting practicality. In this regard, COMPACT is similar to depth pruning, since pruning the vocabulary and intermediate size does not affect the transformer architecture. As a result, COMPACT models are compatible with all inference engines, making it a practical approach to width pruning. The full advantages of COMPACT are summarized in Table 1.

4 Experiments

4.1 Experimental Setup

Baselines. We compare with representative and state-of-theart i) width pruning methods: SliceGPT, [Ashkboos et al., 2024], 2SSP [Sandri et al., 2025]; ii) depth pruning methods: ShortGPT [Men et al., 2024], LaCo [Yang et al., 2024c]. All methods use the default calibration set in their paper. For COMPACT, we use 256 calibration samples from the C4 dataset [Raffel et al., 2020].

LLMs to Prune. We evaluate on a diverse set of LLMs spanning architectures and scales: (i) SLMs: Qwen 2.5-0.5B, LLaMA 3.2-1B, and Gemma 3-1B. This mix covers three distinct families, enabling a robustness assessment across architectures; moreover, SLMs are particularly challenging to prune, as they are often trained beyond the Chinchilla-optimal compute-data balance, leaving limited redundancy. Nevertheless, pruning SLMs is highly valuable for edge use and/or privacy-constrained settings (healthcare, classrooms, federated clients): it shrinks memory/storage, improves end-to-end latency, lowers energy use, and reduces serving cost. (ii) LLMs: LLaMA 3.1-8B and LLaMA 3.1-70B. Together with the 1B variant above, this suite evaluates pruning effectiveness in a robust manner.

Evaluation Tasks. Following SliceGPT [Ashkboos et al., 2024], we evaluate pruned models using HellaSwag (HeSw), WinoGrande (WiGr), ARC-C, ARC-E, and PIQA. Since Jaiswal et al. [2024] shows that pruned LLMs degrade more on complex tasks, we add MMLU for general knowledge, and GSM8K for generation tasks. This gives a more complete view of model performance.

Evaluation Criteria. (A) We report the percentage of parameters that were removed (Ratio (%)), the mean score of the 7 benchmarks (Avg), and the relative mean score compared to the dense model (Avg%); (B) It is standard to evaluate pruned models on perplexity and downstream tasks. However, because we reduce vocabulary size, our perplexity naturally decreases, making comparisons to baselines unfair. Thus, we only report i) performance on downstream tasks, ii) efficiency regarding pruning time, inference time, and memory usage.

4.2 Performance on downstream tasks Results on SLMs

COMPACT outperforms baselines by large margins. Our results are summarized in Table 2. Although prior works report strong results on LLMs, they perform poorly on SLMs.

On Qwen 2.5-0.5B, GSM8K accuracy collapses for all baselines at only 10% pruning. Likewise, at 10% pruning, MMLU drops to near-random for all models and baselines, with the sole exception of LaCo on Qwen 2.5-0.5B. Because MMLU and GSM8K are the most demanding tasks in our suite, these trends indicate that existing methods fail to preserve performance on challenging benchmarks for SLMs. In contrast, COMPACT delays this collapse: it remains marginally above random on MMLU and GSM8K even at 35% pruning across all models. Across models and pruning ratios, COMPACT attains the highest mean score and leads on nearly all individual benchmarks, despite using a slightly higher pruning ratio than the baselines. Notably, at 35% pruning, COMPACT maintains similar performance to the baselines at 20%, demonstrating superior robustness under high compression.

COMPACT supports a wide variety of model architectures out-of-the-box. The official implementations of existing approaches support just a few model architectures. For instance, SliceGPT supports LLaMA/OPT/Phi, LaCo supports LLaMA 2/Baichuan, and ShortGPT only supports LLaMA. Adding support for modern model families like LLaMA 3 and Qwen 2.5 required adding architecture-specific changes, since these architectures can differ significantly in how they handle self-attention, layer normalization, etc. The strength of COMPACT is that it only prunes the vocabulary embeddings and FFN blocks, which have been standardized and remain unchanged across the vast majority of model architectures. As a consequence, COMPACT is architecture-agnostic and runs outof-the-box across many model families. This is most evident with Gemma 3, which uses QK-norm and alternating local and global attention layers-optimizations that prevented us from adapting our baselines. Accordingly, baseline results for Gemma 3 are omitted in Table 2. In contrast, COMPACT operates on Gemma 3 without any architecture-specific changes.

Results on LLMs

COMPACT is robust across scales. Our results are in Table 3. We see that COMPACT achieves state-of-the-art performance for larger models as well, with over 80% performance at a 35% ratio, indicating that our method is highly robust to a wide range of sizes. We attribute this robustness to our dual approach to width pruning. As model size increases, the proportion of vocabulary parameters decreases, which decreases the effectiveness of pruning vocabulary size. However, intermediate pruning becomes more effective as model size increases, similarly to most pruning methods [Xu et al., 2024]. These two techniques complement each other's strengths, leading to a robust hybrid pruning method. Although the performance gap between COMPACT and 2SSP narrows at the 70B size, we note that 2SSP's hybrid approach of pruning FFN channels and entire attention blocks deviates from the standard transformer architecture, while COMPACT's approach does not. This makes COMPACT more practical to deploy, while also achieving slightly higher performance.

COMPACT shows smooth degradation. On LLaMA 3.18B, we observe that ShortGPT surpasses COMPACT at a 20% pruning ratio, but not at 10% or 35%, with its 20% score even exceeding its 10% score. This aligns with the step-like behavior in Gromov et al. [2025] for depth-pruning methods, where performance remains intact up to a critical threshold and then collapses abruptly. ShortGPT also exhibits this pattern, explaining the spike at 20%. In contrast, COMPACT (a width-pruning method) shows smooth MMLU degradation as pruning increases-hence the dip relative to ShortGPT at 20%, followed by recovery and a lead at 35%. Even at 35% pruning, COMPACT remains above random on both MMLU and GSM8K. A similar effect is seen on LLaMA 3.1-70B.

Analysis: Why our pruning hurts performance minimally? Changing the vocabulary affects how text is tokenized: If a token is removed during pruning, the text associated with the token is now tokenized as multiple shorter, more common tokens. We analyze how often rare tokens occur. We use the questions in each of our benchmarks, as well as 10k random samples from the C4 dataset. Our results are in Table 4. Our pruned model reduces vocabulary size from 150k to 50k, a 67% reduction. Despite this, only 4% of words are tokenized differently from the original, regardless of text source.

These results provide insight to the effectiveness of VOCABPRUNING: significant proportions of vocabulary only affects a small fraction of text, so removing these vocabulary has little impact on performance.

4.3 Efficiency in Pruning Time, Inference Throughput, and Memory

Pruning time. The main strength of training-free methods is that they can prune large models efficiently. To test this, we report pruning times at 35% pruning on LLaMA 3.1-8B and 70B to best discriminate between methods.

Our results are in Table 5. With LLaMA 3.1-8B, COMPACT is 3 times faster than 2SSP, our strongest baseline, and comparable our depth pruning methods, with pruning times under a minute. At the 70B size, COMPACT now becomes 6 times faster than 2SSP. The low pruning times show that COMPACT has competitive efficiency to our baselines.

Inference throughput and memory usage. We evaluate two inference paradigms: Text classification and Text generation. The inference setup is in Appendix A.2.

Our results are in Table 6. Note that ShortGPT and LaCo prune the same number of layers, so they have the same inference performance. In the text classification task, our method achieves the highest memory reduction, and the second-highest throughput increase. The low memory usage is from the reduced vocabulary size. During the forward pass, logits are stored in GPU memory, which becomes very large with high batch sizes. By pruning the vocabulary size, COMPACT shrinks logit size, causing large memory reductions. This is especially important in edge computing applications, where memory is very limited and can spell the difference in whether a model can be used or not. COMPACT has higher throughput than our width pruning baselines SliceGPT/2SSP, although it falls behind depth pruning methods. This aligns with Bian et al. [2025], which showed that scaling down layer count proportionally increases throughput, but scaling layer size does not. While COMPACT is faster than other width-pruning methods, it still lags behind depth-pruning approaches. Memory and throughput trends in generation mirror those in classification.

5 Ablation Study

In our ablation study, we try to further four questions: i) Q1: how effective is COMMON-ACT2 ? ii) Q2: how to trade off VOCAB-PRUNING and FFN-PRUNING? iii) Q3: how much calibration data?

Answer to Q1: effectiveness of COMMON-ACT2 . We compare our novel COMMON-ACT2 method with ACT2 . To isolate the effect of the intermediate pruning method, we prune rare vocabulary for all models, then apply the specified intermediate pruning method. We also test another commonly used method, |act|, which is similar to ACT2 but uses the summed absolute activations instead of squared activations. Our results are summarized in Figure 2. We find that COMMONACT2 achieves the highest mean performance.

Answer to Q2: vocabulary-intermediate tradeoff COMPACT has two hyperparameters: the new vocabulary size V ′ and the new intermediate size I ′ , which can be adjusted to produce multiple configurations at a given pruning ratio. We test different configurations of Qwen 2.5-0.5B at 35% pruning (Table 7). Note that V ′ = 151936, I ′ = 2048 is identical to ACT2 . Despite its simplicity, ACT2 achieves better performance than our baselines even without VOCAB-PRUNING. However, the best result is obtained by combining both, validating COMPACT's methodology. One topic of future work is to determine how to compute the optimal choice of V ′ and I ′ more efficiently than a simple linear sweep.

Answer toQ3: calibration data size. We perform ablations over the number of calibration samples, with our results in Figure 3. COMPACT is highly robust to sample count, and similar performance can be achieved with just 16 samples, implying that the pruning time can be reduced further.

6 Conclusion

We propose COMPACT, a training-free pruning method that jointly prunes vocabulary and FFNs under the common-token distribution. Experiments demonstrate robust performance across model scales with strong efficiency and broad deployment compatibility.

A Appendix

A.1 Parameter Distribution Across Other Model Families

Figure 4 provides parameter distributions for the LLaMA 3 and Gemma 3 model families, respectively. We see that these models follow the same trend as Qwen 2.5 where SLMs have a higher proportion of vocabulary parameters, corroborating our theoretical analysis.

A.2 Inference Settings

Settings: i) Text classification: When running our pruned models on our downstream performance benchmarks, we record the maximum memory usage as well as the throughput, measured in number of questions per second. For this test, we use the HellaSwag benchmark, as it is the longest test in our benchmark suite, which allows us to better discriminate between the methods. We test on a larger model (LLaMA 3.1-8B) using a 3-run average, again to better discriminate between methods. All models are loaded in 16-bit precision on a single A100-80GB GPU, with a batch size of 256. ii) Text generation: We use the vLLM library [Kwon et al., 2023] to test the memory reduction and inference speedup of our method. As mentioned before, because SliceGPT and 2SSP are incompatible with vLLM, we omit it from our tests. Similarly to the text classification test, we use a 3-run average of LLaMA 3.1-8B in 16-bit precision on a single A100-80GB GPU, but with a batch size of 1 instead, as this is a more realistic workload for on-device text generation. Tests are conducted with 128 input tokens and 128 output tokens.

A.3 Merge Order vs. Calibration

To compare the effectiveness of using the tokenizer merge order to determine rare vocabulary, we compare it against a calibration-based approach, using 1024 calibration samples from the C4 dataset to find the rarest tokens:

We see that pruning by merge rule order is equally effective, while also being simpler and cheaper.

Acknowledgments

This work was supported by NSF DMS-2533995.