{"type":"ScholarlyArticle","title":"SSML-QNet: Scale-Separative Metric Learning Quadruplet Network for Multi-modal Image Patch Matching","authors":[{"name":"Xiuwei Zhang","slug":"xiuwei-zhang","given":"Xiuwei","family":"Zhang","affiliation":"School of Computer Science and Technology , Northwestern Polytechnical University , Xi'an , China"},{"name":"Yi Sun","slug":"yi-sun","given":"Yi","family":"Sun","affiliation":"School of Cybersecurity , Northwestern Polytechnical University , Xi'an , China"},{"name":"Yamin Han","slug":"yamin-han","given":"Yamin","family":"Han","affiliation":"College of Information Engineering , Northwest A&F University , Yangling , China"},{"name":"Yanping Li","slug":"yanping-li","given":"Yanping","family":"Li","affiliation":"School of Computer Science and Technology , Northwestern Polytechnical University , Xi'an , China"},{"name":"Hanlin Yin","slug":"hanlin-yin","given":"Hanlin","family":"Yin","affiliation":"School of Computer Science and Technology , Northwestern Polytechnical University , Xi'an , China"},{"name":"Yinghui Xing","slug":"yinghui-xing","given":"Yinghui","family":"Xing","affiliation":"School of Computer Science and Technology , Northwestern Polytechnical University , Xi'an , China"},{"name":"Yanning Zhang","slug":"yanning-zhang","given":"Yanning","family":"Zhang","affiliation":"School of Computer Science and Technology , Northwestern Polytechnical University , Xi'an , China"}],"abstract":"Multi-modal image matching is very challenging due to the significant diversities in visual appearance of different modal images. Typically, the existing well-performed methods mainly focus on learning invariant and discriminative features for measuring the relation between multi-modal image pairs. However, these methods often take the features as a whole and largely overlook the fact that different scale features for a same image pair may have different similarity, which may lead to sub-optimal results only. In this work, we propose a Scale-Separative Metric Learning Quadruplet network (SSML-QNet) for multi-modal image patch matching. Specifically, SSML-QNet can extract both relevant and irrelevant features of imaging modality with the proposed quadruplet network architecture. Then, the proposed Scale-Separative Metric Learning module separately encodes the similarity of different scale features with the pyramid structure. And for each scale, cross-modal consistent features are extracted and measured by coordinate and channel-wise attention sequentially. This makes our network robust to appearance divergence caused by different imaging mechanism. Experiments on the benchmark dataset (VIS-NIR, VIS-LWIR, Optical-SAR, and Brown) have verified that the proposed SSML-QNet is able to outperform other state-of-the-art methods. Furthermore, the cross-dataset transferring experiments on these four datasets also have shown that the proposed method has powerful ability of cross-dataset transferring.","doi":"10.24963/ijcai.2023/511","url":"https://ijcai.org/proceedings/2023/511","pdf":"https://ijcai.org/proceedings/2023/0511.pdf","markdown":"https://ijcai.org/proceedings/2023/511.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/511","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":511,"pages":{"first":4593,"last":4601},"keywords":["Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Classification","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[{"position":1,"raw":"[Aguilera et al., 2012] Cristhian Aguilera, Fernando Bar- rera, Felipe Lumbreras, Angel D Sappa, and Ricardo Toledo. Multispectral image feature points. Sensors, 12(9):12661-12672, 2012.","title":"Multispectral image feature points","authors":"Cristhian Aguilera, Fernando Barrera, Felipe Lumbreras, Angel D Sappa, Ricardo Toledo","year":"2012","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aguilera et al., 2015] Cristhian A Aguilera, Angel D Sappa, and Ricardo Toledo. Lghd: A feature descriptor for match- ing across non-linear intensity variations. In 2015 IEEE International Conference on Image Processing (ICIP), pages 178-181. IEEE, 2015.","title":"Lghd: A feature descriptor for matching across non-linear intensity variations","authors":"A Cristhian, Angel D Aguilera, Ricardo Sappa, Toledo","year":"2015","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arandjelovic, 2012] R. Arandjelovic. Three things every- one should know to improve object retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.","title":"Three things everyone should know to improve object retrieval","authors":"R Arandjelovic","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balntas et al., 2016a] Vassileios Balntas, Edward Johns, Lilian Tang, and Krystian Mikolajczyk. Pn-net: Conjoined triple deep network for learning local image descriptors. arXiv preprint arXiv:1601.05030, 2016.","title":"Pn-net: Conjoined triple deep network for learning local image descriptors","authors":"Balntas","year":"2016","venue":"Vassileios Balntas","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Balntas et al., 2016b] Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc, volume 1, page 3, 2016.","title":"Learning local feature descriptors with triplets and shallow convolutional neural networks","authors":"Balntas","year":"2016","venue":"Bmvc","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barnea and Silverman, 1972] Daniel I Barnea and Harvey F Silverman. A class of algorithms for fast digital image reg- istration. IEEE transactions on Computers, 100(2):179- 186, 1972.","title":"A class of algorithms for fast digital image registration","authors":"Silverman ; Barnea, I Daniel, Harvey F Barnea, Silverman","year":"1972","venue":"IEEE transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Baruch and Keller, 2021] Elad Ben Baruch and Yosi Keller. Joint detection and matching of feature points in multi- modal images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.","title":"Joint detection and matching of feature points in multimodal images","authors":"Keller ; ] Baruch, Ben Elad, Yosi Baruch, Keller","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bay et al., 2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pages 404-417. Springer, 2006.","title":"Surf: Speeded up robust features","authors":"Herbert Bay, Tinne Tuytelaars, Luc Van Gool","year":"2006","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brown and Süsstrunk, 2011] Matthew Brown and Sabine Süsstrunk. Multi-spectral sift for scene category recog- nition. In CVPR 2011, pages 177-184. IEEE, 2011.","title":"Multi-spectral sift for scene category recognition","authors":"Matthew Brown, Sabine Süsstrunk","year":"2011","venue":"CVPR 2011","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Brown et al., 2010] Matthew Brown, Gang Hua, and Simon Winder. Discriminative learning of local image descrip- tors. IEEE transactions on pattern analysis and machine intelligence, 33(1):43-57, 2010.","title":"Discriminative learning of local image descriptors","authors":"Matthew Brown, Gang Hua, Simon Winder","year":"2010","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Firmenichy et al., 2011] Damien Firmenichy, Matthew Brown, and Sabine Süsstrunk. Multispectral interest points for rgb-nir image registration. In 2011 18th IEEE international conference on image processing, pages 181-184. IEEE, 2011.","title":"Multispectral interest points for rgb-nir image registration","authors":"Damien Firmenichy, Matthew Brown, Sabine Süsstrunk","year":"2011","venue":"18th IEEE international conference on image processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2015] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg. Match- net: Unifying feature and metric learning for patch-based matching. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 3279-3286, 2015.","title":"Matchnet: Unifying feature and metric learning for patch-based matching","authors":"Han","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hou et al., 2021] Qibin Hou, Daquan Zhou, and Jiashi Feng. Coordinate attention for efficient mobile network design. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13713- 13722, 2021.","title":"Coordinate attention for efficient mobile network design","authors":"Hou","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ke and Sukthankar, 2004] Yan Ke and Rahul Sukthankar. Pca-sift: A more distinctive representation for local im- age descriptors. In Proceedings of the 2004 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II-II. IEEE, 2004.","title":"Pca-sift: A more distinctive representation for local image descriptors","authors":"Yan Ke, Rahul Sukthankar","year":"2004","venue":"Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar BG et al., 2016] Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Learning local image de- scriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 5385-5394, 2016.","title":"Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions","authors":"Vijay Kumar, B G, Gustavo Carneiro, Ian Reid","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2008] Li Liu, FY Peng, Kun Zhao, and YP Wan. Simplified sift algorithm for fast image matching. Infrared and Laser Engineering, 37(1):181-184, 2008.","title":"Simplified sift algorithm for fast image matching","authors":"Li Liu, Kun Peng, Zhao, Wan","year":"2008","venue":"Infrared and Laser Engineering","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mishchuk et al., 2017] Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. Advances in neural information processing systems, 30, 2017.","title":"Working hard to know your neighbor's margins: Local descriptor learning loss","authors":"Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, Jiri Matas","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Moreshet and Keller, 2021] Aviad Moreshet and Yosi Keller. Paying attention to multiscale feature maps in multimodal image matching. arXiv preprint arXiv:2103.11247, 2021.","title":"Paying attention to multiscale feature maps in multimodal image matching","authors":"Keller ; ] Aviad Moreshet, Yosi Moreshet, Keller","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Own and Hassanien, 2002] Hala S Own and Aboul Ella Hassanien. Multiresolution image registration algorithm in wavelet transform domain. In 2002 14th Interna- tional Conference on Digital Signal Processing Proceed- ings. DSP 2002 (Cat. No. 02TH8628), volume 2, pages 889-892. IEEE, 2002.","title":"Multiresolution image registration algorithm in wavelet transform domain","authors":"S Hala, Own, Ella Aboul, Hassanien","year":"2002","venue":"14th International Conference on Digital Signal Processing Proceedings. DSP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Quan et al., 2018] Dou Quan, Shuai Fang, Xuefeng Liang, Shuang Wang, and Licheng Jiao. Cross-spectral image patch matching by learning features of the spatially con- nected patches in a shared space. In Asian Conference on Computer Vision, pages 115-130. Springer, 2018.","title":"Cross-spectral image patch matching by learning features of the spatially connected patches in a shared space","authors":"Dou Quan, Shuai Fang, Xuefeng Liang, Shuang Wang, Licheng Jiao","year":"2018","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Quan et al., 2019] Dou Quan, Xuefeng Liang, Shuang Wang, Shaowei Wei, Yanfeng Li, Ning Huyan, and Licheng Jiao. Afd-net: Aggregated feature difference learning for cross-spectral image patch matching. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 3017-3026, 2019.","title":"Afd-net: Aggregated feature difference learning for cross-spectral image patch matching","authors":"Dou Quan, Xuefeng Liang, Shuang Wang, Shaowei Wei, Yanfeng Li, Ning Huyan, Licheng Jiao","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Quan et al., 2021] Dou Quan, Shuang Wang, Yi Li, Bowu Yang, Ning Huyan, Jocelyn Chanussot, Biao Hou, and Licheng Jiao. Multi-relation attention network for image patch matching. IEEE Transactions on Image Processing, 30:7127-7142, 2021.","title":"Multi-relation attention network for image patch matching","authors":"Dou Quan, Shuang Wang, Yi Li, Bowu Yang, Ning Huyan, Jocelyn Chanussot, Biao Hou, Licheng Jiao","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Savinov et al., 2017] Nikolay Savinov, Akihito Seki, Lu- bor Ladicky, Torsten Sattler, and Marc Pollefeys. Quad- networks: unsupervised learning to rank for interest point detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 1822-1830, 2017.","title":"Quadnetworks: unsupervised learning to rank for interest point detection","authors":"Savinov","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schmitt et al., 2019] Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, and Xiao Xiang Zhu. Sen12ms-a curated dataset of georeferenced multi-spectral sentinel- 1/2 imagery for deep learning and data fusion. arXiv preprint arXiv:1906.07789, 2019.","title":"Sen12ms-a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion","authors":"Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, Xiao Xiang Zhu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shechtman and Irani, 2007] Eli Shechtman and Michal Irani. Matching local self-similarities across images and videos. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007.","title":"Eli Shechtman and Michal Irani. Matching local self-similarities across images and videos","authors":"Irani Shechtman","year":"2007","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Simo-Serra et al., 2015] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. Discriminative learning of deep convo- lutional feature point descriptors. In Proceedings of the IEEE international conference on computer vision, pages 118-126, 2015.","title":"Discriminative learning of deep convolutional feature point descriptors","authors":"Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, Francesc Moreno-Noguer","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan et al., 2014] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Learning local feature descrip- tors using convex optimisation. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 36(8):1573-1585, 2014.","title":"Learning local feature descriptors using convex optimisation","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tian et al., 2017] Yurun Tian, Bin Fan, and Fuchao Wu. L2- net: Deep learning of discriminative patch descriptor in euclidean space. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 661- 669, 2017.","title":"L2net: Deep learning of discriminative patch descriptor in euclidean space","authors":"Yurun Tian, Bin Fan, Fuchao Wu","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Trzcinski et al., 2012] T. Trzcinski, Mario Christoudias, V. Lepetit, and Pascal V Fua. Learning image descriptors with the boosting-trick. In Neural Information Processing Systems, 2012.","title":"Learning image descriptors with the boosting-trick","authors":"T Trzcinski, Mario Christoudias, V Lepetit, Pascal V Fua","year":"2012","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2022] Chuang Yu, Yunpeng Liu, Chenxi Li, Lin Qi, Xin Xia, Tianci Liu, and Zhuhua Hu. Multi-branch feature difference learning network for cross-spectral im- age patch matching. IEEE Transactions on Geoscience and Remote Sensing, 2022.","title":"Multi-branch feature difference learning network for cross-spectral image patch matching","authors":"Chuang Yu, Yunpeng Liu, Chenxi Li, Lin Qi, Xin Xia, Tianci Liu, Zhuhua Hu","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zagoruyko and Komodakis, 2015] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4353-4361, 2015.","title":"Learning to compare image patches via convolutional neural networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2021] Hu Zhang, Keke Zu, Jian Lu, Yuru Zou, and Deyu Meng. Epsanet: An efficient pyramid squeeze attention block on convolutional neural network. arXiv preprint arXiv:2105.14447, 2021.","title":"Epsanet: An efficient pyramid squeeze attention block on convolutional neural network","authors":"Hu Zhang, Keke Zu, Jian Lu, Yuru Zou, Deyu Meng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2022] Xiuwei Zhang, Mu Tian, Yinghui Xing, Yuanzeng Yue, Yanping Li, Hanlin Yin, Runliang Xia, Jin Jin, and Yanning Zhang. Adhr-cdnet: Attentived differen- tial high-resolution change detection network for remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 2022.","title":"Adhr-cdnet: Attentived differential high-resolution change detection network for remote sensing images","authors":"Zhang","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}