{"type":"ScholarlyArticle","title":"Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation","authors":[{"name":"Zekang Chen","slug":"zekang-chen","given":"Zekang","family":"Chen","affiliation":"School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Jia Wei","slug":"jia-wei","given":"Jia","family":"Wei","affiliation":"School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Rui Li","slug":"rui-li","given":"Rui","family":"Li","affiliation":"Golisano College of Computing and Information Sciences , Rochester Institute of Technology , Rochester , NY 14623 , USA"}],"abstract":"In clinical practice, well-aligned multi-modal images, such as Magnetic Resonance (MR) and Computed Tomography (CT), together can provide complementary information for image-guided therapies. Multi-modal image registration is essential for the accurate alignment of these multi-modal images. However, it remains a very challenging task due to complicated and unknown spatial correspondence between different modalities. In this paper, we propose a novel translation-based unsupervised deformable image registration approach to convert the multi-modal registration problem to a mono-modal one. Specifically, our approach incorporates a discriminator-free translation network to facilitate the training of the registration network and a patchwise contrastive loss to encourage the translation network to preserve object shapes. Furthermore, we propose to replace an adversarial loss, that is widely used in previous multi-modal image registration methods, with a pixel loss in order to integrate the output of translation into the target modality. This leads to an unsupervised method requiring no ground-truth deformation or pairs of aligned images for training. We evaluate four variants of our approach on the public Learn2Reg 2021 datasets. The experimental results demonstrate that the proposed architecture achieves state-of-the-art performance. Our code is available at https://github.com/heyblackC/DFMIR.","doi":"10.24963/ijcai.2022/117","url":"https://ijcai.org/proceedings/2022/117","pdf":"https://ijcai.org/proceedings/2022/0117.pdf","markdown":"https://ijcai.org/proceedings/2022/117.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/117","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":117,"pages":{"first":834,"last":840},"keywords":["Machine Learning: Multi-modal learning","Machine Learning: Unsupervised Learning","Computer Vision: Biomedical Image Analysis"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986606","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984749","length_sec":312,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arar et al., 2020] Moab Arar, Yiftach Ginger, Dov Danon, Amit H Bermano, and Daniel Cohen-Or. Unsupervised multi-modal im- age registration via geometry preserving image-to-image transla- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13410-13419, 2020.","title":"Unsupervised multi-modal image registration via geometry preserving image-to-image translation","authors":"Arar","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balakrishnan et al., 2019] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging, 38(8):1788-1800, 2019.","title":"Voxelmorph: a learning framework for deformable medical image registration","authors":"Guha Balakrishnan, Amy Zhao, R Mert, John Sabuncu, Adrian V Guttag, Dalca","year":"2019","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Casamitjana et al., 2021] Adrià Casamitjana, Matteo Mancini, and Juan Eugenio Iglesias. Synth-by-reg (sbr): Contrastive learning for synthesis-based registration of paired images. In Interna- tional Workshop on Simulation and Synthesis in Medical Imag- ing, pages 44-54. Springer, 2021.","title":"Synth-by-reg (sbr): Contrastive learning for synthesis-based registration of paired images","authors":"Adrià Casamitjana, Matteo Mancini, Juan Eugenio Iglesias","year":"2021","venue":"International Workshop on Simulation and Synthesis in Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de Vos et al., 2017] Bob D de Vos, Floris F Berendsen, Max A Viergever, Marius Staring, and Ivana Išgum. End-to-end unsuper- vised deformable image registration with a convolutional neural network. In Deep learning in medical image analysis and mul- timodal learning for clinical decision support, pages 204-212. Springer, 2017.","title":"End-to-end unsupervised deformable image registration with a convolutional neural network","authors":"Bob D De Vos, Floris F Berendsen, Max A Viergever, Marius Staring, Ivana Išgum","year":"2017","venue":"Deep learning in medical image analysis and multimodal learning for clinical decision support","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dice, 1945] Lee R Dice. Measures of the amount of ecologic as- sociation between species. Ecology, 26(3):297-302, 1945.","title":"Measures of the amount of ecologic association between species","authors":"Lee R Dice","year":"1945","venue":"Ecology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fan et al., 2019] Jingfan Fan, Xiaohuan Cao, Qian Wang, Pew- Thian Yap, and Dinggang Shen. Adversarial learning for mono- or multi-modal registration. Medical image analysis, 58:101545, 2019.","title":"Adversarial learning for monoor multi-modal registration","authors":"Jingfan Fan, Xiaohuan Cao, Qian Wang, Pew-Thian Yap, Dinggang Shen","year":"2019","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Heinrich et al., 2012] Mattias P Heinrich, Mark Jenkinson, Manav Bhushan, Tahreema Matin, Fergus V Gleeson, Michael Brady, and Julia A Schnabel. Mind: Modality independent neighbour- hood descriptor for multi-modal deformable registration. Medi- cal image analysis, 16(7):1423-1435, 2012.","title":"Mind: Modality independent neighbourhood descriptor for multi-modal deformable registration","authors":"Mattias P Heinrich, Mark Jenkinson, Manav Bhushan, Tahreema Matin, Fergus V Gleeson, Michael Brady, Julia A Schnabel","year":"2012","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hering et al., 2021] Alessa Hering, Lasse Hansen, Tony CW Mok, Albert Chung, Hanna Siebert, Stephanie Häger, Annkristin Lange, Sven Kuckertz, Stefan Heldmann, Wei Shao, et al. Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. arXiv preprint arXiv:2112.04489, 2021.","title":"Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning","authors":"Alessa Hering, Lasse Hansen, Tony Cw Mok, Albert Chung, Hanna Siebert, Stephanie Häger, Annkristin Lange, Sven Kuckertz, Stefan Heldmann, Wei Shao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoopes et al., 2021] Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca. Hypermorph: Amor- tized hyperparameter learning for image registration. In Interna- tional Conference on Information Processing in Medical Imag- ing, pages 3-17. Springer, 2021.","title":"Hypermorph: Amortized hyperparameter learning for image registration","authors":"Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, Adrian V Dalca","year":"2021","venue":"International Conference on Information Processing in Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2018] Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Ban- dula, Caroline M Moore, Mark Emberton, et al. Weakly- supervised convolutional neural networks for multimodal image registration. Medical image analysis, 49:1-13, 2018.","title":"Weaklysupervised convolutional neural networks for multimodal image registration","authors":"Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M Moore, Mark Emberton","year":"2018","venue":"Medical image analysis","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2018] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image trans- lation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.","title":"Multimodal unsupervised image-to-image translation","authors":"Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huttenlocher et al., 1993] Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9):850-863, 1993.","title":"Comparing images using the hausdorff distance","authors":"Gregory A Daniel P Huttenlocher, William J Klanderman, Rucklidge","year":"1993","venue":"IEEE Transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kong et al., 2021] Lingke Kong, Chenyu Lian, Detian Huang, ZhenJiang Li, Yanle Hu, and Qichao Zhou. Breaking the dilemma of medical image-to-image translation. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.","title":"Breaking the dilemma of medical image-to-image translation","authors":"Kong","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maes et al., 2003] Frederik Maes, Dirk Vandermeulen, and Paul Suetens. Medical image registration using mutual information. Proceedings of the IEEE, 91(10):1699-1722, 2003.","title":"Medical image registration using mutual information","authors":"Frederik Maes, Dirk Vandermeulen, Paul Suetens","year":"2003","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marstal et al., 2016] Kasper Marstal, Floris Berendsen, Marius Staring, and Stefan Klein. Simpleelastix: A user-friendly, multi- lingual library for medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 134-142, 2016.","title":"Simpleelastix: A user-friendly, multilingual library for medical image registration","authors":"Floris Kasper Marstal, Marius Berendsen, Stefan Staring, Klein","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oh and Kim, 2017] Seungjong Oh and Siyong Kim. Deformable image registration in radiation therapy. Radiation oncology jour- nal, 35(2):101, 2017.","title":"Seungjong Oh and Siyong Kim. Deformable image registration in radiation therapy","authors":"Kim Oh","year":"2017","venue":"Radiation oncology journal","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive cod- ing. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Park et al., 2020] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to- image translation. In European Conference on Computer Vision, pages 319-345. Springer, 2020.","title":"Contrastive learning for unpaired image-toimage translation","authors":"Park","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pielawski et al., 2020] Nicolas Pielawski, Elisabeth Wetzer, Johan Öfverstedt, Jiahao Lu, Carolina Wählby, Joakim Lindblad, and Nataša Sladoje. Comir: Contrastive multimodal image represen- tation for registration. arXiv preprint arXiv:2006.06325, 2020.","title":"Comir: Contrastive multimodal image representation for registration","authors":"Nicolas Pielawski, Elisabeth Wetzer, Johan Öfverstedt, Jiahao Lu, Carolina Wählby, Joakim Lindblad, Nataša Sladoje","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qin et al., 2019] Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, and Ali Kamen. Unsupervised deformable registration for multi-modal images via disentangled representa- tions. In International Conference on Information Processing in Medical Imaging, pages 249-261. Springer, 2019.","title":"Unsupervised deformable registration for multi-modal images via disentangled representations","authors":"Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, Ali Kamen","year":"2019","venue":"International Conference on Information Processing in Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computer-assisted intervention","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Uzunova et al., 2017] Hristina Uzunova, Matthias Wilms, Heinz Handels, and Jan Ehrhardt. Training cnns for image registra- tion from few samples with model-based data augmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 223-231. Springer, 2017.","title":"Training cnns for image registration from few samples with model-based data augmentation","authors":"Hristina Uzunova, Matthias Wilms, Heinz Handels, Jan Ehrhardt","year":"2017","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021] Chengjia Wang, Guang Yang, Giorgos Pa- panastasiou, Sotirios A Tsaftaris, David E Newby, Calum Gray, Gillian Macnaught, and Tom J MacGillivray. Dicyc: Gan-based deformation invariant cross-domain information fusion for med- ical image synthesis. Information Fusion, 67:147-160, 2021.","title":"Dicyc: Gan-based deformation invariant cross-domain information fusion for medical image synthesis","authors":"Chengjia Wang, Guang Yang, Giorgos Papanastasiou, A Sotirios, David E Tsaftaris, Calum Newby, Gillian Gray, Tom J Macnaught, Macgillivray","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2020] Zhe Xu, Jie Luo, Jiangpeng Yan, Ritvik Pulya, Xiu Li, William Wells, and Jayender Jagadeesan. Adversar- ial uni-and multi-modal stream networks for multimodal im- age registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 222-232. Springer, 2020.","title":"Adversarial uni-and multi-modal stream networks for multimodal image registration","authors":"Zhe Xu, Jie Luo, Jiangpeng Yan, Ritvik Pulya, Xiu Li, William Wells, Jayender Jagadeesan","year":"2020","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223- 2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}