{"type":"ScholarlyArticle","title":"RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations","authors":[{"name":"Zunhai Su","slug":"zunhai-su","given":"Zunhai","family":"Su","affiliation":""},{"name":"Hanyu Wei","slug":"hanyu-wei","given":"Hanyu","family":"Wei","affiliation":""},{"name":"Zhe Chen","slug":"zhe-chen","given":"Zhe","family":"Chen","affiliation":""},{"name":"Wang Shen","slug":"wang-shen","given":"Wang","family":"Shen","affiliation":""},{"name":"Linge Li","slug":"linge-li","given":"Linge","family":"Li","affiliation":""},{"name":"Huangqi Yu","slug":"huangqi-yu","given":"Huangqi","family":"Yu","affiliation":""},{"name":"Kehong Yuan","slug":"kehong-yuan","given":"Kehong","family":"Yuan","affiliation":""}],"abstract":"Key-Value (KV) cache facilitates efficient large language models (LLMs) inference by avoiding recomputation of past KVs. \r\nAs the batch size and context length increase, the oversized KV caches become a significant memory bottleneck, highlighting the need for efficient compression.\r\nExisting KV quantization rely on fine-grained quantization or the retention of a significant portion of high bit-widths caches, both of which compromise compression ratio and often fail to maintain robustness at extremely low average bit-widths.\r\nIn this work, we explore the potential of rotation technique for 2-bit KV quantization and propose RotateKV, which achieves accurate and robust performance through the following innovations:\r\n(i) Outlier-Aware Rotation, which utilizes channel-reordering to adapt the rotations to varying channel-wise outlier distributions without sacrificing the computational efficiency of the fast Walsh-Hadamard transform (FWHT);\r\n(ii) Pre-RoPE Grouped-Head Rotation, which mitigates the impact of rotary position embedding (RoPE) on proposed outlier-aware rotation and further smooths outliers across heads;\r\n(iii) Attention-Sink-Aware Quantization, which leverages the massive activations to precisely identify and protect attention sinks.\r\nRotateKV achieves less than 0.3 perplexity (PPL) degradation with 2-bit quantization on WikiText-2 using LLaMA-2-13B, maintains strong CoT reasoning and long-context capabilities, with less than 1.7% degradation on GSM8K, outperforming existing methods even at lower average bit-widths.\r\nRotateKV also showcases a 3.97× reduction in peak memory usage, supports 5.75× larger batch sizes, and achieves a 2.32× speedup in decoding stage.","doi":"10.24963/ijcai.2025/690","url":"https://ijcai.org/proceedings/2025/690","pdf":"https://ijcai.org/proceedings/2025/0690.pdf","markdown":"https://ijcai.org/proceedings/2025/690.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/690","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":690,"pages":{"first":6200,"last":6208},"keywords":["Natural Language Processing: NLP: Language generation","Machine Learning: ML: Optimization"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}