{"type":"ScholarlyArticle","title":"MNet: Rethinking 2D/3D Networks for Anisotropic Medical Image Segmentation","authors":[{"name":"Zhangfu Dong","slug":"zhangfu-dong","given":"Zhangfu","family":"Dong","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,"},{"name":"Yuting He","slug":"yuting-he","given":"Yuting","family":"He","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,"},{"name":"Xiaoming Qi","slug":"xiaoming-qi","given":"Xiaoming","family":"Qi","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,"},{"name":"Yang Chen","slug":"yang-chen","given":"Yang","family":"Chen","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,; Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing; Centre de Recherche en Information Biomédicale Sino-Franc ¸ais (CRIBs)"},{"name":"Huazhong Shu","slug":"huazhong-shu","given":"Huazhong","family":"Shu","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,; Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing; Centre de Recherche en Information Biomédicale Sino-Franc ¸ais (CRIBs)"},{"name":"Jean-Louis Coatrieux","slug":"jean-louis-coatrieux","given":"Jean-Louis","family":"Coatrieux","affiliation":"Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing; Centre de Recherche en Information Biomédicale Sino-Franc ¸ais (CRIBs)"},{"name":"Guanyu Yang","slug":"guanyu-yang","given":"Guanyu","family":"Yang","affiliation":"LIST , Key Laboratory of Computer Network and Information Integration ( Southeast University) ,; Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing; Centre de Recherche en Information Biomédicale Sino-Franc ¸ais (CRIBs); Ministry of Education , Nanjing , China"},{"name":"Shuo Li","slug":"shuo-li","given":"Shuo","family":"Li","affiliation":"Dept. of Medical Biophysics , University of Western Ontario , London , ON , Canada"}],"abstract":"The nature of thick-slice scanning causes severe inter-slice discontinuities of 3D medical images, and the vanilla 2D/3D convolutional neural networks (CNNs) fail to represent sparse inter-slice information and dense intra-slice information in a balanced way, leading to severe underfitting to inter-slice features (for vanilla 2D CNNs) and overfitting to noise from long-range slices (for vanilla 3D CNNs). In this work, a novel mesh network (MNet) is proposed to balance the spatial representation inter axes via learning. 1) Our MNet latently fuses plenty of representation processes by embedding multi-dimensional convolutions deeply into basic modules, making the selections of representation processes flexible, thus balancing representation for sparse inter-slice information and dense intra-slice information adaptively. 2) Our MNet latently fuses multi-dimensional features inside each basic module, simultaneously taking the advantages of 2D (high segmentation accuracy of the easily recognized regions in 2D view) and 3D (high smoothness of 3D organ contour) representations, thus obtaining more accurate modeling for target regions. Comprehensive experiments are performed on four public datasets (CT\\&MR), the results consistently demonstrate the proposed MNet outperforms the other methods. The code and datasets are available at: https://github.com/zfdong-code/MNet","doi":"10.24963/ijcai.2022/122","url":"https://ijcai.org/proceedings/2022/122","pdf":"https://ijcai.org/proceedings/2022/0122.pdf","markdown":"https://ijcai.org/proceedings/2022/122.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/122","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":122,"pages":{"first":870,"last":876},"keywords":["Computer Vision: Segmentation","Computer Vision: Biomedical Image Analysis","Computer Vision: 3D Computer Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986735","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984878","length_sec":227,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2018] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine In- telligence, 40(4):834-848, 2018.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Özgün C ¸ic ¸ek, Ahmed Abdulkadir, So- eren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U-Net: Learning dense volumetric segmentation from sparse annotation. In MICCAI, pages 424-432, 2016.","title":"3D U-Net: Learning dense volumetric segmentation from sparse annotation","authors":"Ahmed Özgün C ¸ic ¸ek, Abdulkadir, S Soeren, Thomas Lienkamp, Olaf Brox, Ronneberger","year":"2016","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goldman, 2007] Lee W. Goldman. Principles of CT and CT technology. Journal of Nuclear Medicine Technology, 35(3):115-128, 2007.","title":"Principles of CT and CT technology","authors":"Lee W Goldman","year":"2007","venue":"Journal of Nuclear Medicine Technology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely con- nected convolutional networks. In CVPR, pages 4700- 4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Isensee et al., 2021] Fabian Isensee, Paul F Jaeger, Si- mon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnU-Net: A self-configuring method for deep learning- based biomedical image segmentation. Nature methods, 18(2):203-211, 2021.","title":"nnU-Net: A self-configuring method for deep learningbased biomedical image segmentation","authors":"Fabian Isensee, Paul F Jaeger, Simon Aa Kohl, Jens Petersen, Klaus H Maier-Hein","year":"2021","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2021] Yankai Jiang, Shufeng Xu, Hongjie Fan, Jiahong Qian, Weizhi Luo, Shihui Zhen, Yubo Tao, Ji- hong Sun, and Hai Lin. ALA-Net: Adaptive lesion-aware attention network for 3D colorectal tumor segmentation. IEEE Transactions on Medical Imaging, 40(12):3627- 3640, 2021.","title":"ALA-Net: Adaptive lesion-aware attention network for 3D colorectal tumor segmentation","authors":"Yankai Jiang, Shufeng Xu, Hongjie Fan, Jiahong Qian, Weizhi Luo, Shihui Zhen, Yubo Tao, Jihong Sun, Hai Lin","year":"2021","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lee et al., 2015] Kisuk Lee, Aleksandar Zlateski, Vish- wanathan Ashwin, and H. Sebastian Seung. Recursive training of 2D-3D convolutional networks for neuronal boundary prediction. In NIPS, pages 3573-3581, 2015.","title":"Recursive training of 2D-3D convolutional networks for neuronal boundary prediction","authors":"Lee","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lei et al., 2020] Tao Lei, Risheng Wang, Yong Wan, Bing- tao Zhang, Hongying Meng, and Asoke K Nandi. Medical image segmentation using deep learning: A survey. arXiv preprint arXiv:2009.13120, 2020.","title":"Medical image segmentation using deep learning: A survey","authors":"Tao Lei, Risheng Wang, Yong Wan, Bingtao Zhang, Hongying Meng, Asoke K Nandi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018] Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. H- DenseUNet: Hybrid densely connected unet for liver and tumor segmentation from CT volumes. IEEE Transactions on Medical Imaging, 37(12):2663-2674, 2018.","title":"H-DenseUNet: Hybrid densely connected unet for liver and tumor segmentation from CT volumes","authors":"Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, Pheng-Ann Heng","year":"2018","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Milletari et al., 2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neu- ral networks for volumetric medical image segmentation. In 3DV, pages 565-571, 2016.","title":"V-Net: Fully convolutional neural networks for volumetric medical image segmentation","authors":"Fausto Milletari, Nassir Navab, Seyed-Ahmad Ahmadi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pang et al., 2021] Shumao Pang, Chunlan Pang, Lei Zhao, Yangfan Chen, Zhihai Su, Yujia Zhou, Meiyan Huang, Wei Yang, Hai Lu, and Qianjin Feng. SpineParseNet: Spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representa- tion. IEEE Transactions on Medical Imaging, 40(1):262- 273, 2021.","title":"SpineParseNet: Spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation","authors":"Pang","year":"2021","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shen et al., 2020] Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, and Xiangyang Xue. Ob- ject detection from scratch with deep supervision. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 42(2):398-412, 2020.","title":"Object detection from scratch with deep supervision","authors":"Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, Xiangyang Xue","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Taha and Hanbury, 2015] Abdel Aziz Taha and Allan Han- bury. Metrics for evaluating 3D medical image segmenta- tion: Analysis, selection, and tool. BMC medical imaging, 15(1):1-28, 2015.","title":"Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool","authors":"Hanbury ; Taha, Aziz Abdel, Allan Taha, Hanbury","year":"2015","venue":"BMC medical imaging","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tajbakhsh et al., 2020] Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey N. Chiang, Zhihao Wu, and Xiaowei Ding. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis, 63:101693, 2020.","title":"Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation","authors":"Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey N Chiang, Zhihao Wu, Xiaowei Ding","year":"2020","venue":"Medical Image Analysis","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2019] Guotai Wang, Jonathan Shapey, Wenqi Li, Reuben Dorent, Alexis Dimitriadis, Sotirios Bisdas, Ian Paddick, Robert Bradford, Shaoting Zhang, Sébastien Ourselin, et al. Automatic segmentation of vestibular schwannoma from T2-Weighted MRI by deep spatial at- tention with hardness-weighted loss. In MICCAI, pages 264-272, 2019.","title":"Automatic segmentation of vestibular schwannoma from T2-Weighted MRI by deep spatial attention with hardness-weighted loss","authors":"Guotai Wang, Jonathan Shapey, Wenqi Li, Reuben Dorent, Alexis Dimitriadis, Sotirios Bisdas, Ian Paddick, Robert Bradford, Shaoting Zhang, Sébastien Ourselin","year":"2019","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2020] Zhengyang Wang, Na Zou, Dinggang Shen, and Shuiwang Ji. Non-Local U-Nets for biomedical image segmentation. In AAAI, pages 6315-6322, 2020.","title":"Non-Local U-Nets for biomedical image segmentation","authors":"Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xia et al., 2018] Yingda Xia, Lingxi Xie, Fengze Liu, Zhuotun Zhu, Elliot K Fishman, and Alan L Yuille. Bridg- ing the gap between 2D and 3D organ segmentation with volumetric fusion net. In MICCAI, pages 445-453, 2018.","title":"Bridging the gap between 2D and 3D organ segmentation with volumetric fusion net","authors":"Yingda Xia, Lingxi Xie, Fengze Liu, Zhuotun Zhu, Elliot K Fishman, Alan L Yuille","year":"2018","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zheng et al., 2019] Hao Zheng, Yizhe Zhang, Lin Yang, Peixian Liang, Zhuo Zhao, Chaoli Wang, and Danny Z Chen. A new ensemble learning framework for 3D biomedical image segmentation. In AAAI, pages 5909- 5916, 2019.","title":"A new ensemble learning framework for 3D biomedical image segmentation","authors":"Yizhe Hao Zheng, Lin Zhang, Peixian Yang, Zhuo Liang, Chaoli Zhao, Danny Z Wang, Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhou et al., 2020] Zongwei Zhou, Md Mahfuzur Rah- man Siddiquee, Nima Tajbakhsh, and Jianming Liang. UNet++: Redesigning skip connections to exploit multi- scale features in image segmentation. IEEE Transactions on Medical Imaging, 39(6):1856-1867, 2020.","title":"UNet++: Redesigning skip connections to exploit multiscale features in image segmentation","authors":"Zhou","year":"2020","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zotti et al., 2019] Clement Zotti, Zhiming Luo, Alain La- lande, and Pierre-Marc Jodoin. Convolutional neural net- work with shape prior applied to cardiac MRI segmenta- tion. IEEE Journal of Biomedical and Health Informatics, 23(3):1119-1128, 2019.","title":"Convolutional neural network with shape prior applied to cardiac MRI segmentation","authors":"Clement Zotti, Zhiming Luo, Alain Lalande, Pierre-Marc Jodoin","year":"2019","venue":"IEEE Journal of Biomedical and Health Informatics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}