{"type":"ScholarlyArticle","title":"Image Matching via Loopy RNN","authors":[{"name":"Donghao Luo","slug":"donghao-luo","given":"Donghao","family":"Luo","affiliation":"University"},{"name":"Bingbing Ni","slug":"bingbing-ni","given":"Bingbing","family":"Ni","affiliation":"University"},{"name":"Yichao Yan","slug":"yichao-yan","given":"Yichao","family":"Yan","affiliation":"University"},{"name":"Xiaokang Yang","slug":"xiaokang-yang","given":"Xiaokang","family":"Yang","affiliation":""}],"abstract":"Most existing matching algorithms are one-off algorithms, i.e., they usually measure the distance between the two image feature representation vectors for only one time. In contrast, human's vision system achieves this task, i.e., image matching, by recursively looking at specific/related parts of both images and then making the final judgement. Towards this end, we propose a novel loopy recurrent neural network (Loopy RNN), which is capable of aggregating relationship information of two input images in a progressive/iterative manner and outputting the consolidated matching score in the final iteration. A Loopy RNN features two uniqueness. First, built on conventional long short-term memory (LSTM) nodes, it links the output gate of the tail node to the input gate of the head node, thus it brings up symmetry property required for matching. Second, a monotonous loss designed for the proposed network guarantees increasing confidence during the recursive matching process. Extensive experiments on several image matching benchmarks demonstrate the great potential of the proposed method.","doi":"10.24963/ijcai.2017/335","url":"https://ijcai.org/proceedings/2017/335","pdf":"https://ijcai.org/proceedings/2017/0335.pdf","markdown":"https://ijcai.org/proceedings/2017/335.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/335","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":335,"pages":{"first":2407,"last":2413},"keywords":["Machine Learning: Deep Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Arandjelovic et al., 2016] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In CVPR, 2016.","title":"Netvlad: Cnn architecture for weakly supervised place recognition","authors":"Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, Josef Sivic","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bay et al., 2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In ECCV, 2006.","title":"Surf: Speeded up robust features","authors":"Herbert Bay, Tinne Tuytelaars, Luc Van Gool","year":"2006","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertinetto et al., 2016] Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional siamese networks for object tracking. In ECCV, 2016.","title":"Fully-convolutional siamese networks for object tracking","authors":"Luca Bertinetto, Jack Valmadre, F João, Andrea Henriques, Philip Hs Vedaldi, Torr","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bromley et al., 1993] Jane Bromley, James W Bentz, Bot- tou Léon, Isabelle Guyon, Yann LeCun, Cliff Moore, Ed- uard Säckinger, and Roopak Shah. Signature verification using a siamese time delay neural network. IJPRAI, pages 669-688, 1993.","title":"Signature verification using a siamese time delay neural network","authors":"Jane Bromley, James W Bentz, Bottou Léon, Isabelle Guyon, Yann Lecun, Cliff Moore, Eduard Säckinger, Roopak Shah","year":"1993","venue":"IJPRAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2011] Matthew Brown, Gang Hua, and Simon Winder. Discriminative learning of local image descrip- tors. IEEE TPAMI, pages 43-57, 2011.","title":"Discriminative learning of local image descriptors","authors":"Matthew Brown, Gang Hua, Simon Winder","year":"2011","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2014] Jian Cheng, Cong Leng, Jiaxiang Wu, Hainan Cui, and Hanqing Lu. Fast and accurate image matching with cascade hashing for 3d reconstruction. In CVPR, 2014.","title":"Fast and accurate image matching with cascade hashing for 3d reconstruction","authors":"Jian Cheng, Cong Leng, Jiaxiang Wu, Hainan Cui, Hanqing Lu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dorffner, 1996] Georg Dorffner. Neural networks for time series processing. In Neural network world, 1996.","title":"Neural networks for time series processing","authors":"Georg Dorffner, Dorffner","year":"1996","venue":"Neural network world","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fischer et al., 2014] Philipp Fischer, Alexey Dosovitskiy, and Thomas Brox. Descriptor matching with convolution- al neural networks: a comparison to sift. arXiv preprint arXiv:1405.5769, 2014.","title":"Descriptor matching with convolutional neural networks: a comparison to sift","authors":"Philipp Fischer, Alexey Dosovitskiy, Thomas Brox","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2015] Xufeng Han, Thomas Leung, Yangqing Ji- a, Rahul Sukthankar, and Alexander C Berg. Matchnet: U- nifying feature and metric learning for patch-based match- ing. In CVPR, 2015.","title":"Matchnet: Unifying feature and metric learning for patch-based matching","authors":"Han","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, pages 1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jain et al., 2012] Prateek Jain, Brian Kulis, Jason V Davis, and Inderjit S Dhillon. Metric and kernel learning using a linear transformation. JMLR, pages 519-547, 2012.","title":"Metric and kernel learning using a linear transformation","authors":"Prateek Jain, Brian Kulis, Jason V Davis, Inderjit S Dhillon","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia and Darrell, 2011] Yangqing Jia and Trevor Darrell. Heavy-tailed distances for gradient based image descrip- tors. In NIPS, 2011.","title":"Heavy-tailed distances for gradient based image descriptors","authors":"Darrell ; Jia, Yangqing Jia, Trevor Darrell","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM interna- tional conference on Multimedia, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, pages 91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2016] Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in lstms for activity detection and early detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Learning activity progression in lstms for activity detection and early detection","authors":"Ma","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolajczyk and Schmid, 2005] Krystian Mikolajczyk and Cordelia Schmid. A performance evaluation of local de- scriptors. IEEE TPAMI, 27(10):1615-1630, 2005.","title":"A performance evaluation of local descriptors","authors":"Krystian Mikolajczyk, Cordelia Schmid","year":"2005","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paulin et al., 2015] Mattis Paulin, Matthijs Douze, Zaid Harchaoui, Julien Mairal, Florent Perronin, and Cordelia Schmid. Local convolutional features with unsupervised training for image retrieval. In CVPR, 2015.","title":"Local convolutional features with unsupervised training for image retrieval","authors":"Mattis Paulin, Matthijs Douze, Zaid Harchaoui, Julien Mairal, Florent Perronin, Cordelia Schmid","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2017] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Li- u, Xiaokang Yang, and Hongyuan Zha. Unsupervised deep learning for optical flow estimation. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.","title":"Unsupervised deep learning for optical flow estimation","authors":"Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, Hongyuan Zha","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rublee et al., 2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In ICCV, 2011.","title":"Orb: An efficient alternative to sift or surf","authors":"Rublee","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shyam et al., 2017] Pranav Shyam, Shubham Gupta, and Amdebkar Dukkipati. Attentive recurrent comparators. arXiv preprint arXiv:1703.00767, 2017.","title":"Attentive recurrent comparators","authors":"Pranav Shyam, Shubham Gupta, Amdebkar Dukkipati","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tola et al., 2008] Engin Tola, Vincent Lepetit, and Pascal Fua. A fast local descriptor for dense matching. In CVPR, 2008.","title":"A fast local descriptor for dense matching","authors":"Engin Tola, Vincent Lepetit, Pascal Fua","year":"2008","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Trzcinski et al., 2012] Tomasz Trzcinski, Mario Chris- toudias, Vincent Lepetit, and Pascal Fua. Learning image descriptors with the boosting-trick. In NIPS, 2012.","title":"Learning image descriptors with the boosting-trick","authors":"Tomasz Trzcinski, Mario Christoudias, Vincent Lepetit, Pascal Fua","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Winder et al., 2009] Simon Winder, Gang Hua, and Matthew Brown. Picking the best daisy. In CVPR, 2009.","title":"Picking the best daisy","authors":"Simon Winder, Gang Hua, Matthew Brown","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2015a] Junchi Yan, Jun Wang, Hongyuan Zha, Xiaokang Yang, and Stephen Chu. Consistency-driven alternating optimization for multigraph matching: A uni- fied approach. IEEE Transactions on Image Processing, 24(3):994-1009, 2015.","title":"Consistency-driven alternating optimization for multigraph matching: A unified approach","authors":"Yan","year":"2015","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yan et al., 2015b] Junchi Yan, Chao Zhang, Hongyuan Zha, Wei Liu, Xiaokang Yang, and Stephen M Chu. Dis- crete hyper-graph matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.","title":"Discrete hyper-graph matching","authors":"Yan","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yan et al., 2016a] Junchi Yan, Minsu Cho, Hongyuan Zha, Xiaokang Yang, and Stephen M Chu. Multi-graph match- ing via affinity optimization with graduated consistency regularization. IEEE transactions on pattern analysis and machine intelligence, 38(6):1228-1242, 2016.","title":"Multi-graph matching via affinity optimization with graduated consistency regularization","authors":"Yan","year":"2016","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yan et al., 2016b] Yichao Yan, Bingbing Ni, Zhichao Song, Chao Ma, Yan Yan, and Xiaokang Yang. Person re- identification via recurrent feature aggregation. In ECCV, 2016.","title":"Person reidentification via recurrent feature aggregation","authors":"Yan","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yi et al., 2014] Dong Yi, Zhen Lei, Shengcai Liao, Stan Z Li, et al. Deep metric learning for person re-identification. In ICPR, 2014.","title":"Deep metric learning for person re-identification","authors":"Dong Yi, Zhen Lei, Shengcai Liao, Stan Z Li","year":"2014","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zagoruyko and Komodakis, 2015] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In CVPR, 2015.","title":"Learning to compare image patches via convolutional neural networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}