{"type":"ScholarlyArticle","title":"Nuclei Segmentation via a Deep Panoptic Model with Semantic Feature Fusion","authors":[{"name":"Dongnan Liu","slug":"dongnan-liu","given":"Dongnan","family":"Liu","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Donghao Zhang","slug":"donghao-zhang","given":"Donghao","family":"Zhang","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Yang Song","slug":"yang-song","given":"Yang","family":"Song","affiliation":"School of Computer Science and Engineering , University of New South Wales , Australia"},{"name":"Chaoyi Zhang","slug":"chaoyi-zhang","given":"Chaoyi","family":"Zhang","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Fan Zhang","slug":"fan-zhang","given":"Fan","family":"Zhang","affiliation":"Brigham and Women's Hospital , Harvard Medical School , USA"},{"name":"Lauren O'Donnell","slug":"lauren-o-donnell","given":"Lauren","family":"O'Donnell","affiliation":"Brigham and Women's Hospital , Harvard Medical School , USA"},{"name":"Weidong Cai","slug":"weidong-cai","given":"Weidong","family":"Cai","affiliation":"School of Computer Science , University of Sydney , Australia"}],"abstract":"Automated detection and segmentation of individual nuclei in histopathology images is important for cancer diagnosis and prognosis. Due to the high variability of nuclei appearances and numerous overlapping objects, this task still remains challenging. Deep learning based semantic and instance segmentation models have been proposed to address the challenges, but these methods tend to concentrate on either the global or local features and hence still suffer from information loss. In this work, we propose a panoptic segmentation model which incorporates an auxiliary semantic segmentation branch with the instance branch to integrate global and local features. Furthermore, we design a feature map fusion mechanism in the instance branch and a new mask generator to prevent information loss. Experimental results on three different histopathology datasets demonstrate that our method outperforms the state-of-the-art nuclei segmentation methods and popular semantic and instance segmentation models by a large margin.","doi":"10.24963/ijcai.2019/121","url":"https://ijcai.org/proceedings/2019/121","pdf":"https://ijcai.org/proceedings/2019/0121.pdf","markdown":"https://ijcai.org/proceedings/2019/121.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/121","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":121,"pages":{"first":861,"last":868},"keywords":["Machine Learning Applications: Bio;Medicine","Computer Vision: Biomedical Image Understanding","Machine Learning Applications: Applications of Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Chankong et al., 2014] Thanatip Chankong, Nipon Theera- Umpon, and Sansanee Auephanwiriyakul. Automatic cervical cell segmentation and classification in pap smears. Computer Methods and Programs in Biomedicine, 113(2):539-556, 2014.","title":"Thanatip Chankong, Nipon Theera-Umpon, and Sansanee Auephanwiriyakul. Automatic cervical cell segmentation and classification in pap smears","authors":"Chankong","year":"2014","venue":"Computer Methods and Programs in Biomedicine","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Hao Chen, Xiaojuan Qi, Lequan Yu, Qi Dou, Jing Qin, and Pheng-Ann Heng. DCAN: Deep contour-aware networks for object instance segmentation from histology images. Medical Image Analysis, 36:135- 146, 2017.","title":"DCAN: Deep contour-aware networks for object instance segmentation from histology images","authors":"Xiaojuan Hao Chen, Lequan Qi, Qi Yu, Jing Dou, Pheng-Ann Qin, Heng","year":"2017","venue":"Medical Image Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Geus et al., 2018] Daan de Geus, Panagiotis Meletis, and Gijs Dubbelman. Panoptic segmentation with a joint se- mantic and instance segmentation network. arXiv preprint arXiv:1809.02110, 2018.","title":"Panoptic segmentation with a joint semantic and instance segmentation network","authors":"Panagiotis Daan De Geus, Gijs Meletis, Dubbelman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2980-2988. IEEE, 2017.","title":"Mask R-CNN","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, pages 5967-5976. IEEE, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kirillov et al., 2018] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic seg- mentation. arXiv preprint arXiv:1801.00868, 2018.","title":"Panoptic segmentation","authors":"Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kirillov et al., 2019] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. arXiv preprint arXiv:1901.02446, 2019.","title":"Panoptic feature pyramid networks","authors":"Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kong et al., 2013] Hui Kong, Hatice Cinar Akakin, and Sanjay E Sarma. A generalized laplacian of gaussian filter for blob detection and its applications. IEEE Transactions on Cybernetics, 43(6):1719-1733, 2013.","title":"A generalized laplacian of gaussian filter for blob detection and its applications","authors":"Hui Kong, Hatice Cinar Akakin, Sanjay E Sarma","year":"2013","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kumar et al., 2017] Neeraj Kumar, Ruchika Verma, Sanuj Sharma, Surabhi Bhargava, Abhishek Vahadane, and Amit Sethi. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans- actions on Medical Imaging, 36(7):1550-1560, 2017.","title":"A dataset and a technique for generalized nuclear segmentation for computational pathology","authors":"Neeraj Kumar, Ruchika Verma, Sanuj Sharma, Surabhi Bhargava, Abhishek Vahadane, Amit Sethi","year":"2017","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018a] Dongnan Liu, Donghao Zhang, Siqi Liu, Yang Song, Haozhe Jia, Dagan Feng, Yong Xia, and Weidong Cai. Densely connected large kernel convolu- tional network for semantic membrane segmentation in microscopy images. In ICIP, pages 2461-2465. IEEE, 2018.","title":"Densely connected large kernel convolutional network for semantic membrane segmentation in microscopy images","authors":"Liu","year":"2018","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018b] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, pages 8759-8768. IEEE, 2018.","title":"Path aggregation network for instance segmentation","authors":"Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Naik et al., 2008] Shivang Naik, Scott Doyle, Shannon Ag- ner, Anant Madabhushi, Michael Feldman, and John Tomaszewski. Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology. In ISBI, pages 284-287. IEEE, 2008.","title":"Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology","authors":"Shivang Naik, Scott Doyle, Shannon Agner, Anant Madabhushi, Michael Feldman, John Tomaszewski","year":"2008","venue":"ISBI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Naylor et al., 2018] Peter Naylor, Marick Laé, Fabien Reyal, and Thomas Walter. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Transactions on Medical Imaging, 2018.","title":"Segmentation of nuclei in histopathology images by deep regression of the distance map","authors":"Naylor","year":"2018","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oda et al., 2018] Hirohisa Oda, Holger R Roth, Kosuke Chiba, Jure Sokolić, Takayuki Kitasaka, Masahiro Oda, Akinari Hinoki, Hiroo Uchida, Julia A Schnabel, and Ken- saku Mori. Besnet: Boundary-enhanced segmentation of cells in histopathological images. In MICCAI, pages 228- 236. Springer, 2018.","title":"Besnet: Boundary-enhanced segmentation of cells in histopathological images","authors":"Hirohisa Oda, Kosuke Holger R Roth, Jure Chiba, Takayuki Sokolić, Masahiro Kitasaka, Akinari Oda, Hiroo Hinoki, Julia A Uchida, Kensaku Schnabel, Mori","year":"2018","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2017] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel mat- ters-improve semantic segmentation by global convolu- tional network. In CVPR, pages 1743-1751. IEEE, 2017.","title":"Large kernel matters-improve semantic segmentation by global convolutional network","authors":"Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raza et al., 2019] Shan E Ahmed Raza, Linda Cheung, Muhammad Shaban, Simon Graham, David Epstein, Stella Pelengaris, Michael Khan, and Nasir M Rajpoot. Micro-net: A unified model for segmentation of various objects in microscopy images. Medical image analysis, 52:160-173, 2019.","title":"Micro-net: A unified model for segmentation of various objects in microscopy images","authors":"Shan E, Ahmed Raza, Linda Cheung, Muhammad Shaban, Simon Graham, David Epstein, Stella Pelengaris, Michael Khan, Nasir M Rajpoot","year":"2019","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saleh et al., 2018] Fatemeh Sadat Saleh, Moham- mad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, and Jose M Alvarez. Effective use of synthetic data for urban scene semantic segmentation. In ECCV, pages 86-103. Springer, 2018.","title":"Effective use of synthetic data for urban scene semantic segmentation","authors":"Sadat Fatemeh, Mohammad Saleh, Mathieu Sadegh Aliakbarian, Lars Salzmann, Jose M Petersson, Alvarez","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Veta et al., 2014] Mitko Veta, Josien PW Pluim, Paul J Van Diest, and Max A Viergever. Breast cancer histopathology image analysis: A review. IEEE Transac- tions on Biomedical Engineering, 61(5):1400-1411, 2014.","title":"Breast cancer histopathology image analysis: A review","authors":"Mitko Veta, P W Josien, Paul J Pluim, Max A Van Diest, Viergever","year":"2014","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vu et al., 2018] Quoc Dang Vu, Simon Graham, Minh Nguyen Nhat To, Muhammad Shaban, Talha Qaiser, Navid Alemi Koohbanani, Syed Ali Khurram, Tahsin Kurc, Keyvan Farahani, Tianhao Zhao, et al. Methods for segmentation and classification of digital microscopy tis- sue images. arXiv preprint arXiv:1810.13230, 2018.","title":"Methods for segmentation and classification of digital microscopy tissue images","authors":"Quoc Dang Vu, Simon Graham, Minh Nguyen Nhat, Muhammad To, Talha Shaban, Qaiser, Alemi Navid, Koohbanani, Ali Syed, Tahsin Khurram, Keyvan Kurc, Tianhao Farahani, Zhao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cot- trell. Understanding convolution for semantic segmenta- tion. In WACV, pages 1451-1460. IEEE, 2018.","title":"Understanding convolution for semantic segmentation","authors":"Wang","year":"2018","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2019] Heng Wang, Donghao Zhang, Yang Song, Siqi Liu, Dagan Feng, Yue Wang, Hanchuan Peng, and Weidong Cai. Segmenting neuronal structure in 3d optical microscope images via knowledge distillation with teacher-student network. In ISBI, pages 228-231. IEEE, 2019.","title":"Segmenting neuronal structure in 3d optical microscope images via knowledge distillation with teacher-student network","authors":"Heng Wang, Donghao Zhang, Yang Song, Siqi Liu, Dagan Feng, Yue Wang, Hanchuan Peng, Weidong Cai","year":"2019","venue":"ISBI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wienert et al., 2012] Stephan Wienert, Daniel Heim, Kai Saeger, Albrecht Stenzinger, Michael Beil, Peter Huf- nagl, Manfred Dietel, Carsten Denkert, and Frederick Klauschen. Detection and segmentation of cell nuclei in virtual microscopy images: a minimum-model approach. Scientific Reports, 2:503, 2012.","title":"Detection and segmentation of cell nuclei in virtual microscopy images: a minimum-model approach","authors":"Stephan Wienert, Daniel Heim, Kai Saeger, Albrecht Stenzinger, Michael Beil, Peter Hufnagl, Manfred Dietel, Carsten Denkert, Frederick Klauschen","year":"2012","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiong et al., 2019] Yuwen Xiong, Renjie Liao, Heng- shuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation net- work. arXiv preprint arXiv:1901.03784, 2019.","title":"Upsnet: A unified panoptic segmentation network","authors":"Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2014] Ling Zhang, Hui Kong, Chien Ting Chin, Shaoxiong Liu, Zhi Chen, Tianfu Wang, and Siping Chen. Segmentation of cytoplasm and nuclei of abnormal cells in cervical cytology using global and local graph cuts. Computerized Medical Imaging and Graphics, 38(5):369- 380, 2014.","title":"Segmentation of cytoplasm and nuclei of abnormal cells in cervical cytology using global and local graph cuts","authors":"Ling Zhang, Hui Kong, Chien Ting Chin, Shaoxiong Liu, Zhi Chen, Tianfu Wang, Siping Chen","year":"2014","venue":"Computerized Medical Imaging and Graphics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018a] Donghao Zhang, Yang Song, Dongnan Liu, Haozhe Jia, Siqi Liu, Yong Xia, Heng Huang, and Weidong Cai. Panoptic segmentation with an end-to-end cell R-CNN for pathology image analysis. In MICCAI, pages 237-244. Springer, 2018.","title":"Panoptic segmentation with an end-to-end cell R-CNN for pathology image analysis","authors":"Zhang","year":"2018","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018b] Donghao Zhang, Yang Song, Siqi Liu, Dagan Feng, Yue Wang, and Weidong Cai. Nuclei instance segmentation with dual contour-enhanced adversarial net- work. In ISBI, pages 409-412. IEEE, 2018.","title":"Nuclei instance segmentation with dual contour-enhanced adversarial network","authors":"Zhang","year":"2018","venue":"ISBI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018c] Zhenli Zhang, Xiangyu Zhang, Chao Peng, Xiangyang Xue, and Jian Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In ECCV, pages 269-284, 2018.","title":"Exfuse: Enhancing feature fusion for semantic segmentation","authors":"Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2015] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object de- tectors emerge in deep scene cnns. ICLR, 2015.","title":"Object detectors emerge in deep scene cnns","authors":"Zhou","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}