{"type":"ScholarlyArticle","title":"Multimodal Image Matching Based on Cross-Modality Completion Pre-training","authors":[{"name":"Meng Yang","slug":"meng-yang","given":"Meng","family":"Yang","affiliation":""},{"name":"Fan Fan","slug":"fan-fan","given":"Fan","family":"Fan","affiliation":""},{"name":"Jun Huang","slug":"jun-huang","given":"Jun","family":"Huang","affiliation":""},{"name":"Yong Ma","slug":"yong-ma","given":"Yong","family":"Ma","affiliation":""},{"name":"Xiaoguang Mei","slug":"xiaoguang-mei","given":"Xiaoguang","family":"Mei","affiliation":""},{"name":"Zhanchuan Cai","slug":"zhanchuan-cai","given":"Zhanchuan","family":"Cai","affiliation":""},{"name":"Jiayi Ma","slug":"jiayi-ma","given":"Jiayi","family":"Ma","affiliation":""}],"abstract":"The differences in imaging devices cause multimodal images to have modal differences and geometric distortions, complicating the matching task. Deep learning-based matching methods struggle with multimodal images due to the lack of large annotated multimodal datasets. To address these challenges, we propose XCP-Match based on cross-modality completion pre-training. XCP-Match has two phases. (1) Self-supervised cross-modality completion pre-training based on real multimodal image dataset. We develop a novel pre-training model to learn cross-modal semantic features. The pre-training uses masked image modeling method for cross-modality completion, and introduces an attention-weighted contrastive loss to emphasize matching in overlapping areas. (2) Supervised fine-tuning for multimodal image matching based on the augmented MegaDepth dataset. XCP-Match constructs a complete matching framework to overcome geometric distortions and achieve precise matching. Two-phase training encourages the model to learn deep cross-modal semantic information, improving adaptation to modal differences without needing large annotated datasets. Experiments demonstrate that XCP-Match outperforms existing algorithms on public datasets.","doi":"10.24963/ijcai.2025/246","url":"https://ijcai.org/proceedings/2025/246","pdf":"https://ijcai.org/proceedings/2025/0246.pdf","markdown":"https://ijcai.org/proceedings/2025/246.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/246","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":246,"pages":{"first":2206,"last":2214},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: Machine learning for vision","Computer Vision: CV: Multimodal learning","Computer Vision: CV: Low-level Vision"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}