{"type":"ScholarlyArticle","title":"H-Net: Neural Network for Cross-domain Image Patch Matching","authors":[{"name":"Weiquan Liu","slug":"weiquan-liu","given":"Weiquan","family":"Liu","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , School of Information Science and Engineering , Xiamen University , Xiamen , China"},{"name":"Xuelun Shen","slug":"xuelun-shen","given":"Xuelun","family":"Shen","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , School of Information Science and Engineering , Xiamen University , Xiamen , China"},{"name":"Cheng Wang","slug":"cheng-wang","given":"Cheng","family":"Wang","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , School of Information Science and Engineering , Xiamen University , Xiamen , China"},{"name":"Zhihong Zhang","slug":"zhihong-zhang","given":"Zhihong","family":"Zhang","affiliation":"Software School , Xiamen University , Xiamen , China"},{"name":"Chenglu Wen","slug":"chenglu-wen","given":"Chenglu","family":"Wen","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , School of Information Science and Engineering , Xiamen University , Xiamen , China"},{"name":"Jonathan Li","slug":"jonathan-li","given":"Jonathan","family":"Li","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , School of Information Science and Engineering , Xiamen University , Xiamen , China; Department of Geography and Environmental Management , University of Waterloo , Waterloo , Canada"}],"abstract":"Describing the same scene with different imaging style or rendering image from its 3D model gives us different domain images.  Different domain images tend to have a gap and different local appearances, which raise the main challenge on the cross-domain image patch matching. In this paper, we propose to incorporate AutoEncoder into the Siamese network, named as H-Net, of which the structural shape resembles the letter H. The H-Net achieves state-of-the-art performance on the cross-domain image patch matching. Furthermore, we improved H-Net to H-Net++.  The H-Net++ extracts invariant feature descriptors in cross-domain image patches and achieves state-of-the-art performance by feature retrieval in Euclidean space. As there is no benchmark dataset including cross-domain images, we made a cross-domain image dataset which consists of camera images, rendering images from UAV 3D model, and images generated by CycleGAN algorithm. Experiments show that the proposed H-Net and H-Net++ outperform the existing algorithms. Our code and cross-domain image dataset are available at https://github.com/Xylon-Sean/H-Net.","doi":"10.24963/ijcai.2018/119","url":"https://ijcai.org/proceedings/2018/119","pdf":"https://ijcai.org/proceedings/2018/0119.pdf","markdown":"https://ijcai.org/proceedings/2018/119.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/119","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":119,"pages":{"first":856,"last":863},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Machine Learning Applications: Applications of Supervised Learning","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: 2D and 3D Computer Vision","Machine Learning Applications: Networks"],"videos":[],"references":[{"position":1,"raw":"[Altwaijry et al., 2016] Hani Altwaijry, Eduard Trulls, James Hays, Pascal Fua, and Serge Belongie. Learning to match aerial images with deep attentive architectures. In CVPR, pages 3539-3547, 2016.","title":"Learning to match aerial images with deep attentive architectures","authors":"Altwaijry","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bailer et al., 2017] Christian Bailer, Kiran Varanasi, and Didier Stricker. Cnn-based patch matching for optical flow with thresholded hinge embedding loss. In CVPR, vol- ume 2, page 5, 2017.","title":"Cnn-based patch matching for optical flow with thresholded hinge embedding loss","authors":"Bailer","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balntas et al., 2016] Vassileios Balntas, Edward Johns, Lil- ian Tang, and Krystian Mikolajczyk. Pn-net: conjoined triple deep network for learning local image descriptors. arXiv preprint arXiv:1601.05030, 2016.","title":"Pn-net: conjoined triple deep network for learning local image descriptors","authors":"Balntas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Hsin-Yi Chen, Yen-Yu Lin, and Bing-Yu Chen. Co-segmentation guided hough transform for robust feature matching. IEEE TPAMI, 37(12):2388-2401, 2015.","title":"Co-segmentation guided hough transform for robust feature matching","authors":"Hsin-Yi Chen, Yen-Yu Lin, Bing-Yu Chen","year":"2015","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg. Match- net: Unifying feature and metric learning for patch-based matching. In CVPR, pages 3279-3286, 2015.","title":"Matchnet: Unifying feature and metric learning for patch-based matching","authors":"Han","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton and Sala-khutdinov, 2006] Geoffrey E Hinton and Ruslan R Sala-khutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.","title":"Reducing the dimensionality of data with neural networks","authors":"Geoffrey E Hinton, Ruslan R Sala-Khutdinov","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsu et al., 2015] Kuang-Jui Hsu, Yen-Yu Lin, and Yung-Yu Chuang. Robust image alignment with multiple feature de- scriptors and matching-guided neighborhoods. In CVPR, pages 1921-1930, 2015.","title":"Robust image alignment with multiple feature descriptors and matching-guided neighborhoods","authors":"Kuang-Jui Hsu, Yen-Yu Lin, Yung-Yu Chuang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu and Lin, 2016] Yuan-Ting Hu and Yen-Yu Lin. Progres- sive feature matching with alternate descriptor selection and correspondence enrichment. In CVPR, pages 346- 354, 2016.","title":"Progressive feature matching with alternate descriptor selection and correspondence enrichment","authors":"Yuan-Ting Hu, Yen-Yu Lin","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klambauer et al., 2017] Günter Klambauer, Thomas Un- terthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. NIPS, pages 972-981, 2017.","title":"Selfnormalizing neural networks","authors":"Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar et al., 2016] BG Kumar, Gustavo Carneiro, Ian Reid, et al. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In CVPR, pages 5385-5394, 2016.","title":"Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions","authors":"Kumar","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Leutenegger et al., 2011] Stefan Leutenegger, Margarita Chli, and Roland Y Siegwart. Brisk: Binary robust in- variant scalable keypoints. In ICCV, pages 2548-2555, 2011.","title":"Brisk: Binary robust invariant scalable keypoints","authors":"Leutenegger","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Allinson, 2008] Jing Li and Nigel M Allinson. A comprehensive review of current local features for com- puter vision. Neurocomputing, 71(10):1771-1787, 2008.","title":"A comprehensive review of current local features for computer vision","authors":"Jing Li, Nigel M Allinson","year":"2008","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2015] Tsung-Yi Lin, Yin Cui, Serge Belongie, and James Hays. Learning deep representations for ground-to-aerial geolocalization. In CVPR, pages 5007- 5015, 2015.","title":"Learning deep representations for ground-to-aerial geolocalization","authors":"Tsung-Yi Lin, Yin Cui, Serge Belongie, James Hays","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Matas et al., 2004] Jiri Matas, Ondrej Chum, Martin Urban, and Tomás Pajdla. Robust wide-baseline stereo from max- imally stable extremal regions. Image and vision comput- ing, 22(10):761-767, 2004.","title":"Robust wide-baseline stereo from maximally stable extremal regions","authors":"Matas","year":"2004","venue":"Image and vision computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Melekhov et al., 2016] Iaroslav Melekhov, Juho Kannala, and Esa Rahtu. Image patch matching using convolutional descriptors with euclidean distance. In ACCV, pages 638- 653, 2016.","title":"Image patch matching using convolutional descriptors with euclidean distance","authors":"Iaroslav Melekhov, Juho Kannala, Esa Rahtu","year":"2016","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rublee et al., 2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In ICCV, pages 2564-2571, 2011.","title":"Orb: An efficient alternative to sift or surf","authors":"Rublee","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simo-Serra et al., 2015] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. Discriminative learning of deep convolu- tional feature point descriptors. In ICCV, pages 118-126, 2015.","title":"Discriminative learning of deep convolutional feature point descriptors","authors":"Simo-Serra","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tian et al., 2017] Yurun Tian, Bin Fan, Fuchao Wu, et al. L2-net: Deep learning of discriminative patch descriptor in euclidean space. In CVPR, volume 2, 2017.","title":"L2-net: Deep learning of discriminative patch descriptor in euclidean space","authors":"Yurun Tian, Bin Fan, Fuchao Wu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vo and Hays, 2016] Nam N Vo and James Hays. Localiz- ing and orienting street views using overhead imagery. In ECCV, pages 494-509. Springer, 2016.","title":"Localizing and orienting street views using overhead imagery","authors":"Hays Vo, Nam N Vo, James Hays","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Tsun-Yi Yang, Jo-Han Hsu, Yen-Yu Lin, and Yung-Yu Chuang. Deepcd: Learning deep comple- mentary descriptors for patch representations. In CVPR, pages 3314-3322, 2017.","title":"Deepcd: Learning deep complementary descriptors for patch representations","authors":"Tsun-Yi Yang, Jo-Han Hsu, Yen-Yu Lin, Yung-Yu Chuang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zagoruyko and Komodakis, 2015] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In CVPR, pages 4353- 4361, 2015.","title":"Learning to compare image patches via convolutional neural networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2017] Liang Zheng, Yi Yang, and Qi Tian. Sift meets cnn: A decade survey of instance retrieval. IEEE TPAMI, 2017.","title":"Sift meets cnn: A decade survey of instance retrieval","authors":"Liang Zheng, Yi Yang, Qi Tian","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}