{"type":"ScholarlyArticle","title":"Appearance Prompt Vision Transformer for Connectome Reconstruction","authors":[{"name":"Rui Sun","slug":"rui-sun","given":"Rui","family":"Sun","affiliation":"University of Science and Technology of China"},{"name":"Naisong Luo","slug":"naisong-luo","given":"Naisong","family":"Luo","affiliation":"University of Science and Technology of China"},{"name":"Yuwen Pan","slug":"yuwen-pan","given":"Yuwen","family":"Pan","affiliation":"University of Science and Technology of China"},{"name":"Huayu Mai","slug":"huayu-mai","given":"Huayu","family":"Mai","affiliation":"University of Science and Technology of China"},{"name":"Tianzhu Zhang","slug":"tianzhu-zhang","given":"Tianzhu","family":"Zhang","affiliation":"University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center; Deep Space Exploration Lab"},{"name":"Zhiwei Xiong","slug":"zhiwei-xiong","given":"Zhiwei","family":"Xiong","affiliation":"University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center"},{"name":"Feng Wu","slug":"feng-wu","given":"Feng","family":"Wu","affiliation":"University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center"}],"abstract":"Neural connectivity reconstruction aims to understand the function of biological reconstruction and promote basic scientific research. The intricate morphology and densely intertwined branches make it an extremely challenging task. Most previous best-performing methods adopt affinity learning or metric learning. Nevertheless, they either neglect to model explicit voxel semantics caused by implicit optimization or are hysteresis to spatial information. Furthermore, the inherent locality of 3D CNNs limits modeling long-range dependencies, leading to sub-optimal results. In this work, we propose a coherent and unified Appearance Prompt Vision Transformer (APViT) to integrate affinity and metric learning to exploit the complementarity by learning long-range spatial dependencies. The proposed APViT enjoys several merits. First, the extension continuity-aware attention module aims at constructing hierarchical attention customized for neuron extensibility and slice continuity to learn instance voxel semantic context from a global perspective and utilize continuity priors to enhance voxel spatial awareness. Second, the appearance prompt modulator is responsible for leveraging voxel-adaptive appearance knowledge conditioned on affinity rich in spatial information to instruct instance voxel semantics, exploiting the potential of affinity learning to complement metric learning. Extensive experimental results on multiple challenging benchmarks demonstrate that our APViT achieves consistent improvements with huge flexibility under the same post-processing strategy.","doi":"10.24963/ijcai.2023/158","url":"https://ijcai.org/proceedings/2023/158","pdf":"https://ijcai.org/proceedings/2023/0158.pdf","markdown":"https://ijcai.org/proceedings/2023/158.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/158","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":158,"pages":{"first":1423,"last":1431},"keywords":["Computer Vision: CV: Biomedical image analysis","Computer Vision: CV: Segmentation"],"videos":[],"references":[{"position":1,"raw":"[Arganda-Carreras et al., 2015] Ignacio Arganda-Carreras, Srinivas C Turaga, and Berger. Crowdsourcing the cre- ation of image segmentation algorithms for connectomics. Frontiers in neuroanatomy, page 142, 2015.","title":"Crowdsourcing the creation of image segmentation algorithms for connectomics","authors":"Ignacio Arganda-Carreras, Srinivas C Turaga, Berger","year":"2015","venue":"Frontiers in neuroanatomy","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ascoli, 2002] Giorgio A Ascoli. Computational neu- roanatomy: Principles and methods. Springer Science & Business Media, 2002.","title":"Computational neuroanatomy: Principles and methods","authors":"Giorgio A Ascoli","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beier et al., 2017] Thorsten Beier, Constantin Pape, Nasim Rahaman, Timo Prange, Stuart Berg, Davi D Bock, Albert Cardona, Graham W Knott, Stephen M Plaza, Louis K Scheffer, et al. Multicut brings automated neurite seg- mentation closer to human performance. Nature methods, 14(2):101-102, 2017.","title":"Multicut brings automated neurite segmentation closer to human performance","authors":"Thorsten Beier, Constantin Pape, Nasim Rahaman, Timo Prange, Stuart Berg, D Davi, Albert Bock, Graham W Cardona, Stephen M Knott, Louis K Plaza, Scheffer","year":"2017","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berning et al., 2015] Manuel Berning, Kevin M Boergens, and Moritz Helmstaedter. Segem: efficient image analysis for high-resolution connectomics. Neuron, 87(6):1193- 1206, 2015.","title":"Segem: efficient image analysis for high-resolution connectomics","authors":"Manuel Berning, Kevin M Boergens, Moritz Helmstaedter","year":"2015","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Lian, 2022] Zenggui Chen and Zhouhui Lian. Semi-supervised semantic segmentation via prototypical contrastive learning. In Proceedings of the 30th ACM In- ternational Conference on Multimedia, pages 6696-6705, 2022.","title":"Semi-supervised semantic segmentation via prototypical contrastive learning","authors":"Chen, Lian ;, Zenggui Chen, Zhouhui Lian","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donohue and Ascoli, 2011] Duncan E Donohue and Gior- gio A Ascoli. Automated reconstruction of neuronal mor- phology: an overview. Brain research reviews, 67(1- 2):94-102, 2011.","title":"Automated reconstruction of neuronal morphology: an overview","authors":"E Duncan, Giorgio A Donohue, Ascoli","year":"2011","venue":"Brain research reviews","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy and Beyer, 2020] Alexey Dosovitskiy and Beyer. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"Alexey Dosovitskiy and Beyer. An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Beyer Dosovitskiy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Funke et al., ] Jan Funke, Eric Perlman, Srini Turaga, Davi Bock, and Stephan Saalfeld. Cremi challenge leaderboard, as of 2017/22/09.","title":"Cremi challenge leaderboard","authors":"Funke","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Funke et al., 2018] Jan Funke, Fabian Tschopp, William Grisaitis, Arlo Sheridan, Chandan Singh, Stephan Saalfeld, and Srinivas C Turaga. Large scale image seg- mentation with structured loss based deep learning for connectome reconstruction. TPAMI, 41(7):1669-1680, 2018.","title":"Large scale image segmentation with structured loss based deep learning for connectome reconstruction","authors":"Funke","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2021] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few- shot learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL, pages 3816-3830. ACL, 2021.","title":"Making pre-trained language models better fewshot learners","authors":"Tianyu Gao, Adam Fisch, Danqi Chen","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2022] Yuxian Gu, Xu Han, Zhiyuan Liu, and Min- lie Huang. PPT: pre-trained prompt tuning for few-shot learning. In ACL, pages 8410-8423. Association for Com- putational Linguistics, 2022.","title":"PPT: pre-trained prompt tuning for few-shot learning","authors":"Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hatamizadeh et al., 2022] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Trans- formers for 3d medical image segmentation. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 574-584, 2022.","title":"Unetr: Transformers for 3d medical image segmentation","authors":"Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Daguang Holger R Roth, Xu","year":"2022","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2022] Wei Huang, Shiyu Deng, Chang Chen, Xueyang Fu, and Zhiwei Xiong. Learning to model pixel- embedded affinity for homogeneous instance segmenta- tion. In AAAI, 2022.","title":"Learning to model pixelembedded affinity for homogeneous instance segmentation","authors":"Wei Huang, Shiyu Deng, Chang Chen, Xueyang Fu, Zhiwei Xiong","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Januszewski et al., 2018] Michał Januszewski, Jörgen Ko- rnfeld, Peter H Li, Art Pope, Tim Blakely, Larry Lind- sey, Jeremy Maitin-Shepard, Mike Tyka, Winfried Denk, and Viren Jain. High-precision automated reconstruction of neurons with flood-filling networks. Nature methods, 15(8):605-610, 2018.","title":"High-precision automated reconstruction of neurons with flood-filling networks","authors":"Michał Januszewski, Jörgen Kornfeld, H Peter, Art Li, Tim Pope, Larry Blakely, Jeremy Lindsey, Mike Maitin-Shepard, Winfried Tyka, Viren Denk, Jain","year":"2018","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119, 2022.","title":"Visual prompt tuning","authors":"Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2017] Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, and H Sebastian Seung. Superhuman accuracy on the snemi3d connectomics challenge. arXiv preprint arXiv:1706.00120, 2017.","title":"Superhuman accuracy on the snemi3d connectomics challenge","authors":"Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2021] Kisuk Lee, Ran Lu, Kyle Luther, and H Se- bastian Seung. Learning and segmenting dense voxel em- beddings for 3d neuron reconstruction. TMI, 40(12):3801- 3811, 2021.","title":"Learning and segmenting dense voxel embeddings for 3d neuron reconstruction","authors":"Lee","year":"2021","venue":"TMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Li and Liang, ] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL. [Liu et al., 2021]","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Liang Li, ] Xiang, Lisa Li, Percy Liang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586, 2021.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Livet and Weissman, 2007] Jean Livet and Tamily Weiss- man. Transgenic strategies for combinatorial expression of fluorescent proteins in the nervous system. Nature, 450(7166):56-62, 2007.","title":"Transgenic strategies for combinatorial expression of fluorescent proteins in the nervous system","authors":"Weissman Livet","year":"2007","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Luo et al., 2023] Naisong Luo, Yuwen Pan, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Camou- flaged instance segmentation via explicit de-camouflaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.","title":"Camouflaged instance segmentation via explicit de-camouflaging","authors":"Luo","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mai et al., 2023] Huayu Mai, Rui Sun, Tianzhu Zhang, Zhi- wei Xiong, and Feng Wu. Dualrel: semi-supervised mito- chondria segmentation from a prototype perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.","title":"Dualrel: semi-supervised mitochondria segmentation from a prototype perspective","authors":"Mai","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Meirovitch et al., 2016] Yaron Meirovitch, Alexander Matveev, Hayk Saribekyan, David Budden, David Rolnick, Gergely Odor, Seymour Knowles-Barley, Thouis Raymond Jones, Hanspeter Pfister, Jeff William Lichtman, et al. A multi-pass approach to large-scale connectomics. arXiv preprint arXiv:1612.02120, 2016.","title":"A multi-pass approach to large-scale connectomics","authors":"Yaron Meirovitch, Alexander Matveev, Hayk Saribekyan, David Budden, David Rolnick, Gergely Odor, Seymour Knowles-Barley, Raymond Thouis, Hanspeter Jones, Jeff Pfister, William Lichtman","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Meirovitch et al., 2019a] Yaron Meirovitch, Lu Mi, Hayk Saribekyan, Alexander Matveev, David Rolnick, and Nir Shavit. Cross-classification clustering: an efficient multi- object tracking technique for 3-d instance segmentation in connectomics. In CVPR, pages 8425-8435, 2019.","title":"Cross-classification clustering: an efficient multiobject tracking technique for 3-d instance segmentation in connectomics","authors":"Meirovitch","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Meirovitch et al., 2019b] Yaron Meirovitch, Lu Mi, Hayk Saribekyan, Alexander Matveev, David Rolnick, and Nir Shavit. Cross-classification clustering: an efficient multi- object tracking technique for 3-d instance segmentation in connectomics. In CVPR, pages 8425-8435, 2019.","title":"Cross-classification clustering: an efficient multiobject tracking technique for 3-d instance segmentation in connectomics","authors":"Meirovitch","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2021] Rui Sun, Yihao Li, Tianzhu Zhang, Zhen- dong Mao, Feng Wu, and Yongdong Zhang. Lesion-aware transformers for diabetic retinopathy grading. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10938-10947, 2021.","title":"Lesion-aware transformers for diabetic retinopathy grading","authors":"Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022] Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In Computer Vision-ECCV 2022: 17th Eu- ropean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 36-52. Springer, 2022.","title":"Adaptive agent transformer for few-shot segmentation","authors":"Yuan Wang, Rui Sun, Zhe Zhang, Tianzhu Zhang","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2023] Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmen- tation: a buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.","title":"Rethinking the correlation in few-shot segmentation: a buoys view","authors":"Yuan Wang, Rui Sun, Tianzhu Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wolf et al., 2018] Steffen Wolf, Constantin Pape, Alberto Bailoni, Nasim Rahaman, Anna Kreshuk, Ullrich Kothe, and FredA Hamprecht. The mutex watershed: efficient, parameter-free image partitioning. In ECCV, pages 546- 562, 2018.","title":"The mutex watershed: efficient, parameter-free image partitioning","authors":"Steffen Wolf, Constantin Pape, Alberto Bailoni, Nasim Rahaman, Anna Kreshuk, Ullrich Kothe, Freda Hamprecht","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}