{"type":"ScholarlyArticle","title":"SCTrans: Multi-scale scRNA-seq Sub-vector Completion Transformer for Gene-selective Cell Type Annotation","authors":[{"name":"Lu Lin","slug":"lu-lin","given":"Lu","family":"Lin","affiliation":"School of Computer Science and Engineering , South China University of Technology"},{"name":"Wen Xue","slug":"wen-xue","given":"Wen","family":"Xue","affiliation":"School of Computer Science and Engineering , South China University of Technology"},{"name":"Xindian Wei","slug":"xindian-wei","given":"Xindian","family":"Wei","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Wenjun Shen","slug":"wenjun-shen","given":"Wenjun","family":"Shen","affiliation":"Department of Bioinformatics , Shantou University Medical College"},{"name":"Cheng Liu","slug":"cheng-liu","given":"Cheng","family":"Liu","affiliation":"Department of Computer Science , Shantou University"},{"name":"Si Wu","slug":"si-wu","given":"Si","family":"Wu","affiliation":"School of Computer Science and Engineering , South China University of Technology"},{"name":"Hau San Wong","slug":"hau-san-wong","given":"Hau San","family":"Wong","affiliation":"Department of Computer Science , City University of Hong Kong"}],"abstract":"Cell type annotation is pivotal to single-cell RNA sequencing data (scRNA-seq)-based biological and medical analysis, e.g., identifying biomarkers, exploring cellular heterogeneity, and understanding disease mechanisms. The previous annotation methods typically learn a nonlinear mapping to infer cell type from gene expression vectors, and thus fall short in discovering and associating salient genes with specific cell types. To address this issue, we propose a multi-scale scRNA-seq Sub-vector Completion Transformer, and our model is referred to as SCTrans. Considering that the expressiveness of gene sub-vectors is richer than that of individual genes, we perform multi-scale partitioning on gene vectors followed by masked sub-vector completion, conditioned on unmasked ones. Toward this end, the multi-scale sub-vectors are tokenized, and the intrinsic contextual relationships are modeled via self-attention computation and conditional contrastive regularization imposed on an encoding transformer. By performing mutual learning between the encoder and an additional lightweight counterpart, the salient tokens can be distinguished from the others. As a result, we can perform gene-selective cell type annotation, which contributes to our superior performance over state-of-the-art annotation methods.","doi":"10.24963/ijcai.2024/658","url":"https://ijcai.org/proceedings/2024/658","pdf":"https://ijcai.org/proceedings/2024/0658.pdf","markdown":"https://ijcai.org/proceedings/2024/658.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/658","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":658,"pages":{"first":5954,"last":5962},"keywords":["Data Mining: DM: Applications","Multidisciplinary Topics and Applications: MTA: Bioinformatics","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Abdelaal et al., 2019] Tamim Abdelaal, Lieke Michielsen, Davy Cats, Dylan Hoogduin, Hailiang Mei, Marcel JT Reinders, and Ahmed Mahfouz. A comparison of au- tomatic cell identification methods for single-cell rna se- quencing data. Genome biology, 20:1-19, 2019.","title":"A comparison of automatic cell identification methods for single-cell rna sequencing data","authors":"Abdelaal","year":"2019","venue":"Genome biology","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abnar and Zuidema, 2020] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928, 2020.","title":"Quantifying attention flow in transformers","authors":"Zuidema Abnar, Samira Abnar, Willem Zuidema","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alsaigh et al., 2022] Tom Alsaigh, Doug Evans, David Frankel, and Ali Torkamani. Decoding the transcriptome of calcified atherosclerotic plaque at single-cell resolution. Communications biology, 5(1):1084, 2022.","title":"Decoding the transcriptome of calcified atherosclerotic plaque at single-cell resolution","authors":"Alsaigh","year":"2022","venue":"Communications biology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aran et al., 2019] Dvir Aran, Agnieszka P Looney, Leqian Liu, Esther Wu, Valerie Fong, Austin Hsu, Suzanna Chak, Ram P Naikawadi, Paul J Wolters, Adam R Abate, et al. Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage. Nature im- munology, 20(2):163-172, 2019.","title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage","authors":"Dvir Aran, P Agnieszka, Leqian Looney, Esther Liu, Valerie Wu, Austin Fong, Suzanna Hsu, Ram P Chak, Paul J Naikawadi, Wolters, Adam R Abate","year":"2019","venue":"Nature immunology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brbić et al., 2020] Maria Brbić, Marinka Zitnik, Sheng Wang, Angela O Pisco, Russ B Altman, Spyros Darma- nis, and Jure Leskovec. Mars: discovering novel cell types across heterogeneous single-cell experiments. Na- ture methods, 17(12):1200-1206, 2020.","title":"Mars: discovering novel cell types across heterogeneous single-cell experiments","authors":"Maria Brbić, Marinka Zitnik, Sheng Wang, Angela O Pisco, Spyros Russ B Altman, Jure Darmanis, Leskovec","year":"2020","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2023] Jiawei Chen, Hao Xu, Wanyu Tao, Zhaoxiong Chen, Yuxuan Zhao, and Jing-Dong J Han. Transformer for one stop interpretable cell type annota- tion. Nature Communications, 14(1):223, 2023.","title":"Transformer for one stop interpretable cell type annotation","authors":"Jiawei Chen, Hao Xu, Wanyu Tao, Zhaoxiong Chen, Yuxuan Zhao, Jing-Dong J Han","year":"2023","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Connell et al., 2022] William Connell, Umair Khan, and Michael J Keiser. A single-cell gene expression language model. arXiv preprint arXiv:2210.14330, 2022.","title":"A single-cell gene expression language model","authors":"William Connell, Umair Khan, Michael J Keiser","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cortal et al., 2021] Akira Cortal, Loredana Martignetti, Emmanuelle Six, and Antonio Rausell. Gene signature extraction and cell identity recognition at the single-cell level with cell-id. Nature biotechnology, 39(9):1095- 1102, 2021.","title":"Akira Cortal, Loredana Martignetti, Emmanuelle Six, and Antonio Rausell. Gene signature extraction and cell identity recognition at the single-cell level with cell-id","authors":"Cortal","year":"2021","venue":"Nature biotechnology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2021] Alexy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: trans- formers for image recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: transformers for image recognition at scale","authors":"Alexy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hao et al., 2021] Yuhan Hao, Stephanie Hao, Erica Andersen-Nissen, William M Mauck, Shiwei Zheng, Andrew Butler, Maddie J Lee, Aaron J Wilk, Charlotte Darby, Michael Zager, et al. Integrated analysis of multimodal single-cell data. Cell, 184(13):3573-3587, 2021.","title":"Integrated analysis of multimodal single-cell data","authors":"Yuhan Hao, Stephanie Hao, Erica Andersen-Nissen, William M Mauck, Shiwei Zheng, Andrew Butler, Maddie J Lee, Aaron J Wilk, Charlotte Darby, Michael Zager","year":"2021","venue":"Cell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hao et al., 2023] Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large scale foundation model on single-cell transcriptomics. bioRxiv, pages 2023-05, 2023.","title":"Large scale foundation model on single-cell transcriptomics","authors":"Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, Xuegong Zhang","year":"2023","venue":"bioRxiv","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2022] Hang Hu, Zhong Li, Xiangjie Li, Minzhe Yu, and Xiutao Pan. Sccaes: deep clustering of single- cell rna-seq via convolutional autoencoder embedding and soft k-means. Briefings in Bioinformatics, 23(1):bbab321, 2022.","title":"Sccaes: deep clustering of singlecell rna-seq via convolutional autoencoder embedding and soft k-means","authors":"Hang Hu, Zhong Li, Xiangjie Li, Minzhe Yu, Xiutao Pan","year":"2022","venue":"Briefings in Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ianevski et al., 2022] Aleksandr Ianevski, Anil K Giri, and Tero Aittokallio. Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data. Nature communications, 13(1):1246, 2022.","title":"Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data","authors":"Aleksandr Ianevski, Anil K Giri, Tero Aittokallio","year":"2022","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kimmel and Kelley, 2021] Jacob C Kimmel and David R Kelley. Semisupervised adversarial neural networks for single-cell classification. Genome research, 31(10):1781- 1793, 2021.","title":"Semisupervised adversarial neural networks for single-cell classification","authors":"C Jacob, David R Kimmel, Kelley","year":"2021","venue":"Genome research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kiselev et al., 2017] Vladimir Yu Kiselev, Kristina Kirschner, Michael T Schaub, Tallulah Andrews, Andrew Yiu, Tamir Chandra, Kedar N Natarajan, Wolf Reik, Mauricio Barahona, Anthony R Green, et al. Sc3: consensus clustering of single-cell rna-seq data. Nature methods, 14(5):483-486, 2017.","title":"Sc3: consensus clustering of single-cell rna-seq data","authors":"Vladimir Yu Kiselev, Kristina Kirschner, T Michael, Tallulah Schaub, Andrew Andrews, Tamir Yiu, Chandra, Wolf Kedar N Natarajan, Mauricio Reik, Barahona, Anthony R Green","year":"2017","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kiselev et al., 2018] Vladimir Yu Kiselev, Andrew Yiu, and Martin Hemberg. scmap: projection of single-cell rna- seq data across data sets. Nature methods, 15(5):359-362, 2018.","title":"scmap: projection of single-cell rnaseq data across data sets","authors":"Vladimir Yu Kiselev, Andrew Yiu, Martin Hemberg","year":"2018","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kiselev et al., 2019] Vladimir Yu Kiselev, Tallulah S An- drews, and Martin Hemberg. Challenges in unsupervised clustering of single-cell rna-seq data. Nature Reviews Ge- netics, 20(5):273-282, 2019.","title":"Challenges in unsupervised clustering of single-cell rna-seq data","authors":"Vladimir Yu Kiselev, Tallulah S Andrews, Martin Hemberg","year":"2019","venue":"Nature Reviews Genetics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2020a] Chenwei Li, Baolin Liu, Boxi Kang, Zedao Liu, Yedan Liu, Changya Chen, Xianwen Ren, and Zemin Zhang. Scibet as a portable and fast single cell type iden- tifier. Nature communications, 11(1):1818, 2020.","title":"Scibet as a portable and fast single cell type identifier","authors":"Li","year":"2020","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2020b] Xiangjie Li, Kui Wang, Yafei Lyu, Huize Pan, Jingxiao Zhang, Dwight Stambolian, Katalin Susz- tak, Muredach P Reilly, Gang Hu, and Mingyao Li. Deep learning enables accurate clustering with batch effect re- moval in single-cell rna-seq analysis. Nature communica- tions, 11(1):2338, 2020.","title":"Deep learning enables accurate clustering with batch effect removal in single-cell rna-seq analysis","authors":"Li","year":"2020","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ma et al., 2023] Anjun Ma, Xiaoying Wang, Jingxian Li, Cankun Wang, Tong Xiao, Yuntao Liu, Hao Cheng, Juexin Wang, Yang Li, Yuzhou Chang, et al. Single-cell biolog- ical network inference using a heterogeneous graph trans- former. Nature Communications, 14(1):964, 2023.","title":"Single-cell biological network inference using a heterogeneous graph transformer","authors":"Anjun Ma, Xiaoying Wang, Jingxian Li, Cankun Wang, Tong Xiao, Yuntao Liu, Hao Cheng, Juexin Wang, Yang Li, Yuzhou Chang","year":"2023","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Meng et al., 2023] Rui Meng, Shuaidong Yin, Jianqiang Sun, Huan Hu, and Qi Zhao. scaaga: Single cell data analysis framework using asymmetric autoencoder with gene attention. Computers in Biology and Medicine, 165:107414, 2023.","title":"scaaga: Single cell data analysis framework using asymmetric autoencoder with gene attention","authors":"Rui Meng, Shuaidong Yin, Jianqiang Sun, Huan Hu, Qi Zhao","year":"2023","venue":"Computers in Biology and Medicine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shao et al., 2021] Xin Shao, Haihong Yang, Xiang Zhuang, Jie Liao, Penghui Yang, Junyun Cheng, Xiaoyan Lu, Hua- jun Chen, and Xiaohui Fan. scdeepsort: a pre-trained cell- type annotation method for single-cell transcriptomics us- ing deep learning with a weighted graph neural network. Nucleic acids research, 49(21):e122-e122, 2021.","title":"scdeepsort: a pre-trained celltype annotation method for single-cell transcriptomics using deep learning with a weighted graph neural network","authors":"Xin Shao, Haihong Yang, Xiang Zhuang, Jie Liao, Penghui Yang, Junyun Cheng, Xiaoyan Lu, Huajun Chen, Xiaohui Fan","year":"2021","venue":"Nucleic acids research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tian et al., 2021] Tian Tian, Jie Zhang, Xiang Lin, Zhi Wei, and Hakon Hakonarson. Model-based deep embedding for constrained clustering analysis of single cell rna-seq data. Nature communications, 12(1):1873, 2021.","title":"Model-based deep embedding for constrained clustering analysis of single cell rna-seq data","authors":"Tian Tian, Jie Zhang, Xiang Lin, Zhi Wei, Hakon Hakonarson","year":"2021","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Neural Information Processing Systems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proc. Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wan et al., 2022] Hui Wan, Liang Chen, and Minghua Deng. scname: neighborhood contrastive clustering with ancillary mask estimation for scrna-seq data. Bioinformat- ics, 38(6):1575-1583, 2022.","title":"scname: neighborhood contrastive clustering with ancillary mask estimation for scrna-seq data","authors":"Liang Hui Wan, Minghua Chen, Deng","year":"2022","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2017] Bo Wang, Junjie Zhu, Emma Pierson, Daniele Ramazzotti, and Serafim Batzoglou. Visualization and analysis of single-cell rna-seq data by kernel-based similarity learning. Nature methods, 14(4):414-416, 2017.","title":"Visualization and analysis of single-cell rna-seq data by kernel-based similarity learning","authors":"Bo Wang, Junjie Zhu, Emma Pierson, Daniele Ramazzotti, Serafim Batzoglou","year":"2017","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Jiacheng Wang, Quan Zou, and Chen Lin. A comparison of deep learning-based pre-processing and clustering approaches for single-cell rna sequencing data. Briefings in Bioinformatics, 23(1):bbab345, 2022.","title":"A comparison of deep learning-based pre-processing and clustering approaches for single-cell rna sequencing data","authors":"Wang","year":"2022","venue":"Briefings in Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2021] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvR: introducing convolutions to vision transformers. arXiv:2103.15808, 2021.","title":"CvR: introducing convolutions to vision transformers","authors":"Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2019] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. June 2019.","title":"Deep flow-guided video inpainting","authors":"Rui Xu, Xiaoxiao Li, Bolei Zhou, Chen Change Loy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2022] Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jian- hua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4(10):852-866, 2022.","title":"scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data","authors":"Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, Jianhua Yao","year":"2022","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhai et al., 2023] Yuyao Zhai, Liang Chen, and Minghua Deng. Realistic cell type annotation and discovery for single-cell rna-seq data. In Proceedings of the Thirty- Second International Joint Conference on Artificial Intel- ligence, pages 4967-4974, 2023.","title":"Realistic cell type annotation and discovery for single-cell rna-seq data","authors":"Zhai","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/552"},{"position":35,"raw":"[Ziegenhain et al., 2017] Christoph Ziegenhain, Beate Vieth, Swati Parekh, Björn Reinius, Amy Guillaumet-Adkins, Martha Smets, Heinrich Leonhardt, Holger Heyn, Ines Hellmann, and Wolfgang Enard. Comparative analysis of single-cell rna sequencing methods. Molecular cell, 65(4):631-643, 2017.","title":"Comparative analysis of single-cell rna sequencing methods","authors":"Christoph Ziegenhain, Beate Vieth, Swati Parekh, Björn Reinius, Amy Guillaumet-Adkins, Martha Smets, Heinrich Leonhardt, Holger Heyn, Ines Hellmann, Wolfgang Enard","year":"2017","venue":"Molecular cell","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}