---
title: "QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention"
authors:
  - "Sehyeon Oh"
  - "Yongin Kwon"
  - "Jemin Lee"
doi: "10.24963/ijcai.2026/167"
url: "https://ijcai.org/proceedings/2026/167"
pdf: "https://ijcai.org/proceedings/2026/0167.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "1496-1504"
track: "Main Track"
keywords:
  - "Computer Vision: Efficiency and Optimization"
---

# QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

Sehyeon Oh, Yongin Kwon, Jemin Lee

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 1496-1504. DOI: [10.24963/ijcai.2026/167](https://doi.org/10.24963/ijcai.2026/167)

## Abstract

FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult.
We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison.
To address these challenges, we propose QFlash, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel.
On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73x speedup over I-ViT and up to 8.69x speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization.
Our code is publicly available at https://github.com/EfficientCompLab/qflash.