Proceedings

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

Sehyeon Oh, Yongin Kwon, Jemin Lee

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 1496-1504.

https://doi.org/10.24963/ijcai.2026/167

Keywords

  • Computer Vision: Efficiency and Optimization

Abstract

FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult.

We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison.

To address these challenges, we propose QFlash, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel.

On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73x speedup over I-ViT and up to 8.69x speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization.

Our code is publicly available at https://github.com/EfficientCompLab/qflash.

Sehyeon Oh, Yongin Kwon, and Jemin Lee. 2026. QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 1496–1504. https://doi.org/10.24963/ijcai.2026/167
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards