{"type":"ScholarlyArticle","title":"Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss","authors":[{"name":"Qi Dou","slug":"qi-dou","given":"Qi","family":"Dou","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong"},{"name":"Cheng Ouyang","slug":"cheng-ouyang","given":"Cheng","family":"Ouyang","affiliation":"Department of Electrical Engineering and Computer Science , University of Michigan"},{"name":"Cheng Chen","slug":"cheng-chen","given":"Cheng","family":"Chen","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong; ×3 Conv-BN-ReLU , 3×3 Conv -BN-ReLU , 3×3"},{"name":"Hao Chen","slug":"hao-chen","given":"Hao","family":"Chen","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong; Imsight Medical Technology Inc. , Shenzhen , China"},{"name":"Pheng-Ann Heng","slug":"pheng-ann-heng","given":"Pheng-Ann","family":"Heng","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong"}],"abstract":"Convolutional networks (ConvNets) have achieved great successes in various challenging vision tasks. However, the performance of ConvNets would degrade when encountering the domain shift. The domain adaptation is more significant while challenging in the field of biomedical image analysis, where cross-modality data have largely different distributions. Given that annotating the medical data is especially expensive, the supervised transfer learning approaches are not quite optimal. In this paper, we propose an unsupervised domain adaptation framework with adversarial learning for cross-modality biomedical image segmentations. Specifically, our model is based on a dilated fully convolutional network for pixel-wise prediction.  Moreover, we build a plug-and-play domain adaptation module (DAM) to map the target input to features which are aligned with source domain feature space. A domain critic module (DCM) is set up for discriminating the feature space of both domains. We optimize the DAM and DCM via an adversarial loss without using any target domain label.\r\nOur proposed method is validated by adapting a ConvNet trained with MRI images to unpaired CT data for cardiac structures segmentations, and achieved very promising results.","doi":"10.24963/ijcai.2018/96","url":"https://ijcai.org/proceedings/2018/96","pdf":"https://ijcai.org/proceedings/2018/0096.pdf","markdown":"https://ijcai.org/proceedings/2018/96.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/96","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":96,"pages":{"first":691,"last":697},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning","Computer Vision: Biomedical Image Understanding","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.","title":"Wasserstein gan","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bousmalis et al., 2017] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Kr- ishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017.","title":"Unsupervised pixel-level domain adaptation with generative adversarial networks","authors":"Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, Dilip Krishnan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dou et al., 2017] Qi Dou, Lequan Yu, Hao Chen, Yueming Jin, Xin Yang, Jing Qin, and Pheng-Ann Heng. 3d deeply supervised network for automated segmentation of volu- metric medical images. Medical image analysis, 41:40- 54, 2017.","title":"3d deeply supervised network for automated segmentation of volumetric medical images","authors":"Qi Dou, Lequan Yu, Hao Chen, Yueming Jin, Xin Yang, Jing Qin, Pheng-Ann Heng","year":"2017","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Esteva et al., 2017] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115-118, 2017.","title":"Dermatologist-level classification of skin cancer with deep neural networks","authors":"Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, Sebastian Thrun","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghafoorian et al., 2017] Mohsen Ghafoorian, Alireza Mehrtash, Tina Kapur, Nico Karssemeijer, Elena Mar- chiori, Mehran Pesteie, Charles RG Guttmann, Frank-Erik de Leeuw, Clare M Tempany, Bram van Ginneken, et al. Transfer learning for domain adaptation in mri: Application in brain lesion segmentation. In MICCAI, pages 516-524, 2017.","title":"Transfer learning for domain adaptation in mri: Application in brain lesion segmentation","authors":"Mohsen Ghafoorian, Alireza Mehrtash, Tina Kapur, Nico Karssemeijer, Elena Marchiori, Mehran Pesteie, Frank-Erik Charles Rg Guttmann, Clare M De Leeuw, Bram Tempany, Van Ginneken","year":"2017","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gretton et al., 2009] Arthur Gretton, Alexander J Smola, Ji- ayuan Huang, Marcel Schmittfull, Karsten M Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. 2009.","title":"Covariate shift by kernel mean matching","authors":"Arthur Gretton, Alexander J Smola, Jiayuan Huang, Marcel Schmittfull, Karsten M Borgwardt, Bernhard Schölkopf","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kamnitsas et al., 2017] Konstantinos Kamnitsas, Christian Baumgartner, Christian Ledig, Virginia Newcombe, Joanna Simpson, Andrew Kane, David Menon, Aditya Nori, Antonio Criminisi, Daniel Rueckert, et al. Unsuper- vised domain adaptation in brain lesion segmentation with adversarial networks. In International Conference on In- formation Processing in Medical Imaging, pages 597-609. Springer, 2017.","title":"Unsupervised domain adaptation in brain lesion segmentation with adversarial networks","authors":"Konstantinos Kamnitsas, Christian Baumgartner, Christian Ledig, Virginia Newcombe, Joanna Simpson, Andrew Kane, David Menon, Aditya Nori, Antonio Criminisi, Daniel Rueckert","year":"2017","venue":"International Conference on Information Processing in Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Milletari et al., 2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neu- ral networks for volumetric medical image segmentation. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 565-571. IEEE, 2016.","title":"V-net: Fully convolutional neural networks for volumetric medical image segmentation","authors":"Fausto Milletari, Nassir Navab, Seyed-Ahmad Ahmadi","year":"2016","venue":"Fourth International Conference on","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowl- edge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Patel et al., 2015] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adapta- tion: A survey of recent advances. IEEE signal processing magazine, 32(3):53-69, 2015.","title":"Visual domain adaptation: A survey of recent advances","authors":"M Vishal, Raghuraman Patel, Ruonan Gopalan, Rama Li, Chellappa","year":"2015","venue":"IEEE signal processing magazine","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Payer et al., 2017] Christian Payer, Darko Štern, Horst Bischof, and Martin Urschler. Multi-label whole heart seg- mentation using cnns and anatomical label configurations. pages 190-198, 2017.","title":"Christian Payer","authors":"Payer","year":"2017","venue":"Multi-label whole heart segmentation using cnns and anatomical label configurations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shimodaira, 2000] Hidetoshi Shimodaira. Improving pre- dictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.","title":"Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function","authors":"Shimodaira","year":"2000","venue":"Journal of statistical planning and inference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun and Saenko, 2016] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adap- tation. In Proceedings of the ECCV Workshops, pages 443-450. Springer, 2016.","title":"Deep coral: Correlation alignment for deep domain adaptation","authors":"Saenko ; Sun, Baochen Sun, Kate Saenko","year":"2016","venue":"Proceedings of the ECCV Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Torralba and Efros, 2011] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pages 1521-1528, 2011.","title":"Unbiased look at dataset bias","authors":"Efros Torralba, Antonio Torralba, Alexei A Efros","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.","title":"Deep domain confusion: Maximizing for domain invariance","authors":"Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pages 2962-2971, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2017] Yifei Wang, Wen Li, Dengxin Dai, and Luc Van Gool. Deep domain adaptation by geodesic dis- tance minimization. In Proceedings of the IEEE Interna- tional Conference on Computer Vision Workshops, 2017.","title":"Deep domain adaptation by geodesic distance minimization","authors":"Yifei Wang, Wen Li, Dengxin Dai, Luc Van Gool","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, pages 3320-3328, 2014.","title":"How transferable are features in deep neural networks?","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2017] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, pages 636-644, 2017.","title":"Dilated residual networks","authors":"Fisher Yu, Vladlen Koltun, Thomas Funkhouser","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhuang and Shen, 2016] Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Medical image analysis, 31:77- 87, 2016.","title":"Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri","authors":"Shen Zhuang","year":"2016","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}