{"type":"ScholarlyArticle","title":"SatFormer: Saliency-Guided Abnormality-Aware Transformer for Retinal Disease Classification in Fundus Image","authors":[{"name":"Yankai Jiang","slug":"yankai-jiang","given":"Yankai","family":"Jiang","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Ke Xu","slug":"ke-xu","given":"Ke","family":"Xu","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Xinyue Wang","slug":"xinyue-wang","given":"Xinyue","family":"Wang","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Yuan Li","slug":"yuan-li","given":"Yuan","family":"Li","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Hongguang Cui","slug":"hongguang-cui","given":"Hongguang","family":"Cui","affiliation":"The First Affiliated Hospital , Zhejiang University School of Medicine , Hangzhou , China {jyk1996ver , 3180103434 ,"},{"name":"Yubo Tao","slug":"yubo-tao","given":"Yubo","family":"Tao","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Hai Lin","slug":"hai-lin","given":"Hai","family":"Lin","affiliation":"State Key Laboratory of CAD&CG , College of Computer Science and Technology , Zhejiang University , Hangzhou , China"}],"abstract":"Automatic and accurate retinal disease diagnosis is critical to guide proper therapy and prevent potential vision loss. Previous works simply exploit the most discriminative features while ignoring the pathological visual clues of scattered subtle lesions. Therefore, without a comprehensive understanding of features from different lesion regions, they are vulnerable to noise from complex backgrounds and suffer from misclassification failures. In this paper, we address these limitations with a novel saliency-guided abnormality-aware transformer which explicitly captures the correlation between different lesion features from a global perspective with enhanced pathological semantics. The model has several merits. First, we propose a saliency enhancement module (SEM) which adaptively integrates disease related semantics and highlights potentially salient lesion regions. Second, to the best of our knowledge, this is the first work to explore comprehensive lesion feature dependencies via a tailored efficient self-attention. Third, with the saliency enhancement module and abnormality-aware attention, we propose a new variant of Vision Transformer models, called SatFormer, which outperforms the state-of-the-art methods on two public retinal disease classification benchmarks. Ablation study shows that the proposed components can be easily embedded into any Vision Transformers via a plug-and-play manner and effectively boost the performance.","doi":"10.24963/ijcai.2022/138","url":"https://ijcai.org/proceedings/2022/138","pdf":"https://ijcai.org/proceedings/2022/0138.pdf","markdown":"https://ijcai.org/proceedings/2022/138.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/138","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":138,"pages":{"first":987,"last":994},"keywords":["Machine Learning: Convolutional Networks","Computer Vision: Biomedical Image Analysis","Machine Learning: Classification","Machine Learning: Attention Models","Multidisciplinary Topics and Applications: Bioinformatics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986676","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984819","length_sec":348,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bi et al., 2021] Qi Bi, Shuang Yu, Wei Ji, Cheng Bian, Li- jun Gong, Hanruo Liu, Kai Ma, and Yefeng Zheng. Local- global dual perception based deep multiple instance learn- ing for retinal disease classification. In MICCAI, pages 55-64. Springer, 2021.","title":"Localglobal dual perception based deep multiple instance learning for retinal disease classification","authors":"Qi Bi, Shuang Yu, Wei Ji, Cheng Bian, Lijun Gong, Hanruo Liu, Kai Ma, Yefeng Zheng","year":"2021","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bourne et al., 2017] Rupert RA Bourne, Seth R Flaxman, Tasanee Braithwaite, et al. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. The Lancet Global Health, 5(9):e888- e897, 2017.","title":"Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis","authors":"Seth R Rupert Ra Bourne, Tasanee Flaxman, Braithwaite","year":"2017","venue":"The Lancet Global Health","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural In- formation Processing Systems, 34, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cuadros and Bresnick, 2009] Jorge Cuadros and George Bresnick. Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening. Journal of diabetes science and technology, 3(3):509-516, 2009.","title":"Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening","authors":"Cuadros, Bresnick, Jorge Cuadros, George Bresnick","year":"2009","venue":"Journal of diabetes science and technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2021] Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. In ICCV, pages 732-741, 2021.","title":"Rethinking channel dimensions for efficient model design","authors":"Han","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2019] Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng. Canet: cross- disease attention network for joint diabetic retinopathy and diabetic macular edema grading. IEEE transactions on medical imaging, 39(5):1483-1493, 2019.","title":"Canet: crossdisease attention network for joint diabetic retinopathy and diabetic macular edema grading","authors":"Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, Pheng-Ann Heng","year":"2019","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707, 2021.","title":"Localvit: Bringing locality to vision transformers","authors":"Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, Luc Van Gool","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2018] Zhiwen Lin, Ruoqian Guo, Yanjie Wang, Bian Wu, Tingting Chen, Wenzhe Wang, Danny Z Chen, and Jian Wu. A framework for identifying diabetic retinopathy based on anti-noise detection and attention- based fusion. In MICCAI, pages 74-82. Springer, 2018.","title":"A framework for identifying diabetic retinopathy based on anti-noise detection and attentionbased fusion","authors":"Zhiwen Lin, Ruoqian Guo, Yanjie Wang, Bian Wu, Tingting Chen, Wenzhe Wang, Danny Z Chen, Jian Wu","year":"2018","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pachade et al., 2021] Samiksha Pachade, Prasanna Porwal, et al. Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research. Data, 6(2):14, 2021.","title":"Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research","authors":"Samiksha Pachade, Prasanna Porwal","year":"2021","venue":"Data","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Phene et al., 2019] Sonia Phene, R Carter Dunn, Naama Hammel, Yun Liu, Jonathan Krause, Naho Kitade, Mike Schaekermann, Rory Sayres, Derek J Wu, Ashish Bora, et al. Deep learning and glaucoma specialists: the rel- ative importance of optic disc features to predict glau- coma referral in fundus photographs. Ophthalmology, 126(12):1627-1639, 2019.","title":"Deep learning and glaucoma specialists: the relative importance of optic disc features to predict glaucoma referral in fundus photographs","authors":"Sonia Phene, R Carter Dunn, Naama Hammel, Yun Liu, Jonathan Krause, Naho Kitade, Mike Schaekermann, Rory Sayres, Derek J Wu, Ashish Bora","year":"2019","venue":"Ophthalmology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sun et al., 2021] Rui Sun, Yihao Li, Tianzhu Zhang, Zhen- dong Mao, Feng Wu, and Yongdong Zhang. Lesion-aware transformers for diabetic retinopathy grading. In CVPR, pages 10938-10947, 2021.","title":"Lesion-aware transformers for diabetic retinopathy grading","authors":"Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. Efficient- net: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114. PMLR, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2017] Zhe Wang, Yanxin Yin, Jianping Shi, Wei Fang, Hongsheng Li, and Xiaogang Wang. Zoom- in-net: Deep mining lesions for diabetic retinopathy de- tection. In MICCAI, pages 267-275. Springer, 2017.","title":"Zoomin-net: Deep mining lesions for diabetic retinopathy detection","authors":"Zhe Wang, Yanxin Yin, Jianping Shi, Wei Fang, Hongsheng Li, Xiaogang Wang","year":"2017","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021a] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In ICCV, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021b] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Cross- former: A versatile vision transformer hinging on cross- scale attention. arXiv preprint arXiv:2108.00154, 2021.","title":"Crossformer: A versatile vision transformer hinging on crossscale attention","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2022] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, pages 1-10, 2022.","title":"Pvt v2: Improved baselines with pyramid vision transformer","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2022","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, pages 3-19, 2018.","title":"Cbam: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021a] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, pages 22-31, 2021.","title":"Cvt: Introducing convolutions to vision transformers","authors":"Wu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021b] Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2t: Pyramid pooling transformer for scene understanding. arXiv preprint arXiv:2106.12011, 2021.","title":"P2t: Pyramid pooling transformer for scene understanding","authors":"Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2021] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. In ICCV, pages 9981-9990, 2021.","title":"Co-scale conv-attentional image transformers","authors":"Weijian Xu, Yifan Xu, Tyler Chang, Zhuowen Tu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2021] Shuang Yu, Kai Ma, Qi Bi, Cheng Bian, Munan Ning, Nanjun He, Yuexiang Li, Hanruo Liu, and Yefeng Zheng. Mil-vt: Multiple instance learning en- hanced vision transformer for fundus image classification. In MICCAI, pages 45-54. Springer, 2021.","title":"Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification","authors":"Shuang Yu, Kai Ma, Qi Bi, Cheng Bian, Munan Ning, Nanjun He, Yuexiang Li, Hanruo Liu, Yefeng Zheng","year":"2021","venue":"MICCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}