{"type":"ScholarlyArticle","title":"QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention","authors":[{"name":"Sehyeon Oh","slug":"sehyeon-oh","given":"Sehyeon","family":"Oh","affiliation":""},{"name":"Yongin Kwon","slug":"yongin-kwon","given":"Yongin","family":"Kwon","affiliation":""},{"name":"Jemin Lee","slug":"jemin-lee","given":"Jemin","family":"Lee","affiliation":""}],"abstract":"FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult.\r\nWe identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison.\r\nTo address these challenges, we propose QFlash, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel.\r\nOn seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73x speedup over I-ViT and up to 8.69x speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization.\r\nOur code is publicly available at https://github.com/EfficientCompLab/qflash.","doi":"10.24963/ijcai.2026/167","url":"https://ijcai.org/proceedings/2026/167","pdf":"https://ijcai.org/proceedings/2026/0167.pdf","markdown":"https://ijcai.org/proceedings/2026/167.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/167","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Computer Vision","track":"Main Track","number":167,"pages":{"first":1496,"last":1504},"keywords":["Computer Vision: Efficiency and Optimization"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}