{"type":"ScholarlyArticle","title":"U-Match: Two-view Correspondence Learning with Hierarchy-aware Local Context Aggregation","authors":[{"name":"Zizhuo Li","slug":"zizhuo-li","given":"Zizhuo","family":"Li","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Shihua Zhang","slug":"shihua-zhang","given":"Shihua","family":"Zhang","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Jiayi Ma","slug":"jiayi-ma","given":"Jiayi","family":"Ma","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"}],"abstract":"Local context capturing has become the core factor for achieving leading performance in two-view correspondence learning. Recent advances have devised various local context extractors whereas typically adopting explicit neighborhood relation modeling that is restricted and inflexible. To address this issue, we introduce U-Match, an attentional graph neural network that has the flexibility to enable implicit local context awareness at multiple levels. Specifically, a hierarchy-aware graph representation (HAGR) module is designed and fleshed out by local context pooling and unpooling operations. The former encodes local context by adaptively sampling a set of nodes to form a coarse-grained graph, while the latter decodes local context by recovering the coarsened graph back to its original size. Moreover, an orthogonal fusion module is proposed for the collaborative use of HAGR module, which integrates complementary local and global information into compact feature representations without redundancy. Extensive experiments on different visual tasks prove that our method significantly surpasses the state-of-the-arts. In particular, U-Match attains an AUC at 5 degree threshold of 60.53% on the challenging YFCC100M dataset without RANSAC, outperforming the strongest prior model by 8.61 absolute percentage points. Our code is publicly available at https://github.com/ZizhuoLi/U-Match.","doi":"10.24963/ijcai.2023/130","url":"https://ijcai.org/proceedings/2023/130","pdf":"https://ijcai.org/proceedings/2023/0130.pdf","markdown":"https://ijcai.org/proceedings/2023/130.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/130","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":130,"pages":{"first":1169,"last":1176},"keywords":["Computer Vision: CV: Image and video retrieval","Computer Vision: CV: Motion and tracking"],"videos":[],"references":[{"position":1,"raw":"[Balntas et al., 2017] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A bench- mark and evaluation of handcrafted and learned local de- scriptors. In Proc. CVPR, pages 5173-5182, 2017.","title":"Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors","authors":"Balntas","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barath et al., 2020] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas. Magsac++, a fast, reliable and accurate robust estimator. In Proc. CVPR, pages 1304- 1312, 2020.","title":"Magsac++, a fast, reliable and accurate robust estimator","authors":"Daniel Barath, Jana Noskova, Maksym Ivashechkin, Jiri Matas","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bian et al., 2017] JiaWang Bian, Wen-Yan Lin, Yasuyuki Matsushita, Sai-Kit Yeung, Tan-Dat Nguyen, and Ming- Ming Cheng. Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence. In Proc. CVPR, pages 4181-4190, 2017.","title":"Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence","authors":"Jiawang Bian, Wen-Yan Lin, Yasuyuki Matsushita, Sai-Kit Yeung, Tan-Dat Nguyen, Ming-Ming Cheng","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2022] Luanyuan Dai, Yizhang Liu, Jiayi Ma, Li- fang Wei, Taotao Lai, Changcai Yang, and Riqing Chen. Ms2dg-net: Progressive correspondence learning via mul- tiple sparse semantics dynamic graph. In Proc. CVPR, pages 8973-8982, 2022.","title":"Ms2dg-net: Progressive correspondence learning via multiple sparse semantics dynamic graph","authors":"Dai","year":"2022","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[DeTone et al., 2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised in- terest point detection and description. In Proc. CVPR, pages 224-236, 2018.","title":"Superpoint: Self-supervised interest point detection and description","authors":"Tomasz Daniel Detone, Andrew Malisiewicz, Rabinovich","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fischler and Bolles, 1981] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381-395, 1981.","title":"Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography","authors":"A Martin, Robert C Fischler, Bolles","year":"1981","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hartley and Zisserman, 2003] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2003.","title":"Multiple View Geometry in Computer Vision","authors":"Zisserman Hartley, Richard Hartley, Andrew Zisserman","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2021] Yuan Liu, Lingjie Liu, Cheng Lin, Zhen Dong, and Wenping Wang. Learnable motion coherence for correspondence pruning. In Proc. CVPR, pages 3237- 3246, 2021.","title":"Learnable motion coherence for correspondence pruning","authors":"Yuan Liu, Lingjie Liu, Cheng Lin, Zhen Dong, Wenping Wang","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis., 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ma et al., 2014] Jiayi Ma, Ji Zhao, Jinwen Tian, Alan L Yuille, and Zhuowen Tu. Robust point matching via vector field consensus. IEEE Trans. Image Process., 23(4):1706- 1721, 2014.","title":"Robust point matching via vector field consensus","authors":"Jiayi Ma, Ji Zhao, Jinwen Tian, Alan L Yuille, Zhuowen Tu","year":"2014","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2019] Jiayi Ma, Ji Zhao, Junjun Jiang, Huabing Zhou, and Xiaojie Guo. Locality preserving matching. Int. J. Comput. Vis., 127(5):512-531, 2019.","title":"Locality preserving matching","authors":"Jiayi Ma, Ji Zhao, Junjun Jiang, Huabing Zhou, Xiaojie Guo","year":"2019","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":"/proceedings/2017/627"},{"position":12,"raw":"[Ma et al., 2021] Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Jun- jun Jiang, and Junchi Yan. Image matching from hand- crafted to deep features: A survey. Int. J. Comput. Vis., 129(1):23-79, 2021.","title":"Image matching from handcrafted to deep features: A survey","authors":"Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, Junchi Yan","year":"2021","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proc. CVPR, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Raguram et al., 2012] Rahul Raguram, Ondrej Chum, Marc Pollefeys, Jiri Matas, and Jan-Michael Frahm. Usac: A universal framework for random sample consensus. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):2022-2038, 2012.","title":"Usac: A universal framework for random sample consensus","authors":"Rahul Raguram, Ondrej Chum, Marc Pollefeys, Jiri Matas, Jan-Michael Frahm","year":"2012","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sarlin et al., 2019] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In Proc. CVPR, pages 12716-12725, 2019.","title":"From coarse to fine: Robust hierarchical localization at large scale","authors":"Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, Marcin Dymczyk","year":"2019","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sarlin et al., 2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proc. CVPR, pages 4938-4947, 2020.","title":"Superglue: Learning feature matching with graph neural networks","authors":"Paul-Edouard Sarlin, Daniel Detone, Tomasz Malisiewicz, Andrew Rabinovich","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sattler et al., 2018] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localiza- tion in changing conditions. In Proc. CVPR, pages 8601- 8610, 2018.","title":"Benchmarking 6dof outdoor visual localization in changing conditions","authors":"Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schonberger and Frahm, 2016] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proc. CVPR, pages 4104-4113, 2016.","title":"Structure-from-motion revisited","authors":"Frahm ; Schonberger, L Johannes, Jan-Michael Schonberger, Frahm","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2020] Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi. Acne: Attentive context normalization for robust permutation-equivariant learning. In Proc. CVPR, pages 11286-11295, 2020.","title":"Acne: Attentive context normalization for robust permutation-equivariant learning","authors":"Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, Kwang Moo, Yi","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2022] Linfeng Tang, Yuxin Deng, Yong Ma, Jun Huang, and Jiayi Ma. Superfusion: A versatile image registration and fusion network with semantic awareness. IEEE/CAA J. Autom. Sinica, 9(12):2121-2137, 2022.","title":"Superfusion: A versatile image registration and fusion network with semantic awareness","authors":"Tang","year":"2022","venue":"IEEE/CAA J. Autom. Sinica","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thomee et al., 2016] Bart Thomee, David A Shamma, Ger- ald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Commun. ACM, 59(2):64-73, 2016.","title":"Yfcc100m: The new data in multimedia research","authors":"Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, Li-Jia Li","year":"2016","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30:1-11, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xiao et al., 2013] Jianxiong Xiao, Andrew Owens, and An- tonio Torralba. Sun3d: A database of big spaces recon- structed using sfm and object labels. In Proc. ICCV, pages 1625-1632, 2013.","title":"Sun3d: A database of big spaces reconstructed using sfm and object labels","authors":"Jianxiong Xiao, Andrew Owens, Antonio Torralba","year":"2013","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yi et al., 2018] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proc. CVPR, pages 2666-2674, 2018.","title":"Learning to find good correspondences","authors":"Kwang Moo, Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, Pascal Fua","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning two-view correspon- dences and geometry using order-aware network. In Proc. ICCV, pages 5845-5854, 2019.","title":"Learning two-view correspondences and geometry using order-aware network","authors":"Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, Hongen Liao","year":"2019","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2021] Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive correspondence pruning by consensus learning. In Proc. ICCV, pages 6464-6473, 2021.","title":"Progressive correspondence pruning by consensus learning","authors":"Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, Mathieu Salzmann","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}