{"type":"ScholarlyArticle","title":"Learning to Hash Naturally Sorts","authors":[{"name":"Jiaguo Yu","slug":"jiaguo-yu","given":"Jiaguo","family":"Yu","affiliation":"Nanjing University of Science and Technology"},{"name":"Yuming Shen","slug":"yuming-shen","given":"Yuming","family":"Shen","affiliation":"University of Oxford"},{"name":"Menghan Wang","slug":"menghan-wang","given":"Menghan","family":"Wang","affiliation":"eBay"},{"name":"Haofeng Zhang","slug":"haofeng-zhang","given":"Haofeng","family":"Zhang","affiliation":"Nanjing University of Science and Technology"},{"name":"Philip H.S. Torr","slug":"philip-h-s-torr","given":"Philip H.S.","family":"Torr","affiliation":"University of Oxford"}],"abstract":"Learning to hash pictures a list-wise sorting problem. Its testing metrics, e.g., mean-average precision, count on a sorted candidate list ordered by pair-wise code similarity. However, scarcely does one train a deep hashing model with the sorted results end-to-end because of the non-differentiable nature of the sorting operation. This inconsistency in the objectives of training and test may lead to sub-optimal performance since the training loss often fails to reflect the actual retrieval metric. In this paper, we tackle this problem by introducing Naturally-Sorted Hashing (NSH). We sort the Hamming distances of samples' hash codes and accordingly gather their latent representations for self-supervised training. Thanks to the recent advances in differentiable sorting approximations, the hash head receives gradients from the sorter so that the hash encoder can be optimized along with the training procedure. Additionally, we describe a novel Sorted Noise-Contrastive Estimation (SortedNCE) loss that selectively picks positive and negative samples for contrastive learning, which allows NSH to mine data semantic relations during training in an unsupervised manner. Our extensive experiments show the proposed NSH model significantly outperforms the existing unsupervised hashing methods on three benchmarked datasets.","doi":"10.24963/ijcai.2022/221","url":"https://ijcai.org/proceedings/2022/221","pdf":"https://ijcai.org/proceedings/2022/0221.pdf","markdown":"https://ijcai.org/proceedings/2022/221.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/221","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":221,"pages":{"first":1587,"last":1593},"keywords":["Computer Vision: Image and Video retrieval","Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Representation Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987011","length_sec":63,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985154","length_sec":505,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Blondel et al., 2020] Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and rank- ing. In ICML, 2020.","title":"Fast differentiable sorting and ranking","authors":"Mathieu Blondel, Olivier Teboul, Quentin Berthet, Josip Djolonga","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In ICML, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real- world web image database from national university of singapore. In CIVR, 2009.","title":"Nus-wide: a realworld web image database from national university of singapore","authors":"Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng","year":"2009","venue":"CIVR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2017] Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. Stochastic generative hashing. In ICML, 2017.","title":"Stochastic generative hashing","authors":"Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, Le Song","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erin Liong et al., 2015] Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre Moulin, and Jie Zhou. Deep hashing for compact binary codes learning. In CVPR, 2015.","title":"Deep hashing for compact binary codes learning","authors":"Erin Liong","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghasedi Dizaji et al., 2018] Kamran Ghasedi Dizaji, Feng Zheng, Najmeh Sadoughi, Yanhua Yang, Cheng Deng, and Heng Huang. Unsupervised deep generative adversarial hashing network. In CVPR, 2018.","title":"Unsupervised deep generative adversarial hashing network","authors":"Kamran Ghasedi Dizaji, Feng Zheng, Najmeh Sadoughi, Yanhua Yang, Cheng Deng, Heng Huang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gionis et al., 1999] Aristides Gionis, Piotr Indyk, Rajeev Mot- wani, et al. Similarity search in high dimensions via hashing. In VLDB, 1999.","title":"Similarity search in high dimensions via hashing","authors":"Gionis","year":"1999","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gong et al., 2013] Y. Gong, S. Lazebnik, A. Gordo, and F. Per- ronnin. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2916-2929, 2013.","title":"Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval","authors":"Y Gong, S Lazebnik, A Gordo, F","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hadsell et al., 2006] Raia Hadsell, Sumit Chopra, and Yann Le- Cun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.","title":"Dimensionality reduction by learning an invariant mapping","authors":"Raia Hadsell, Sumit Chopra, Yann Le-Cun","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and van Gemert, 2021] Yunqiang Li and Jan van Gemert. Deep unsupervised image hashing by maximizing bit entropy. In AAAI, 2021.","title":"Deep unsupervised image hashing by maximizing bit entropy","authors":"Van Li, Yunqiang Gemert, Jan Li, Van Gemert","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in con- text. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2016] Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou. Learning compact binary descriptors with unsupervised deep neural networks. In CVPR, 2016.","title":"Learning compact binary descriptors with unsupervised deep neural networks","authors":"Kevin Lin, Jiwen Lu, Chu-Song Chen, Jie Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2011] Wei Liu, Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. Hashing with graphs. In ICML, 2011.","title":"Hashing with graphs","authors":"Wei Liu, Jun Wang, Sanjiv Kumar, Shih-Fu Chang","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2014] Wei Liu, Cun Mu, Sanjiv Kumar, and Shih-Fu Chang. Discrete graph hashing. In NeurIPS, 2014.","title":"Discrete graph hashing","authors":"Wei Liu, Cun Mu, Sanjiv Kumar, Shih-Fu Chang","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2021a] Xiao Luo, Daqing Wu, Zeyu Ma, Chong Chen, Minghua Deng, Jianqiang Huang, and Xian-Sheng Hua. A sta- tistical approach to mining semantic similarity for deep unsuper- vised hashing. In MM, 2021.","title":"A statistical approach to mining semantic similarity for deep unsupervised hashing","authors":"Luo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Luo et al., 2021b] Xiao Luo, Daqing Wu, Zeyu Ma, Chong Chen, Huasong Zhong, Minghua Deng, Jianqiang Huang, and Xian- sheng Hua. Cimon: Towards high-quality hash codes. In IJCAI, 2021.","title":"Cimon: Towards high-quality hash codes","authors":"Luo","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/125"},{"position":22,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learn- ing Research, 9(Nov):2579-2605, 2008.","title":"Visualizing data using t-sne","authors":"Hinton ; Laurens Maaten, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive cod- ing. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Prillo and Eisenschlos, 2020] Sebastian Prillo and Julian Eisen- schlos. Softsort: A continuous relaxation for the argsort operator. In ICML, 2020.","title":"Softsort: A continuous relaxation for the argsort operator","authors":"Sebastian Prillo, Julian Eisenschlos","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiu et al., 2021] Zexuan Qiu, Qinliang Su, Zijing Ou, Jianxing Yu, and Changyou Chen. Unsupervised hashing with contrastive in- formation bottleneck. In IJCAI, 2021.","title":"Unsupervised hashing with contrastive information bottleneck","authors":"Qiu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/133"},{"position":26,"raw":"[Shen et al., 2018] Fumin Shen, Yan Xu, Li Liu, Yang Yang, Zi Huang, and Heng Tao Shen. Unsupervised deep hashing with similarity-adaptive and discrete optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):3034- 3044, 2018.","title":"Unsupervised deep hashing with similarity-adaptive and discrete optimization","authors":"Fumin Shen, Yan Xu, Li Liu, Yang Yang, Zi Huang, Heng Tao Shen","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shen et al., 2019] Yuming Shen, Li Liu, and Ling Shao. Un- supervised binary representation learning with deep variational networks. International Journal of Computer Vision, 127(11- 12):1614-1628, 2019.","title":"Unsupervised binary representation learning with deep variational networks","authors":"Yuming Shen, Li Liu, Ling Shao","year":"2019","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shen et al., 2020] Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao. Auto-encoding twin-bottleneck hashing. In CVPR, 2020.","title":"Auto-encoding twin-bottleneck hashing","authors":"Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, Ling Shao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Song et al., 2018] Jingkuan Song, Tao He, Lianli Gao, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Binary generative adversarial networks for image retrieval. In AAAI, 2018.","title":"Binary generative adversarial networks for image retrieval","authors":"Song","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Su et al., 2018] Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cnn. In NeurIPS, 2018.","title":"Greedy hash: Towards fast optimization for accurate hash coding in cnn","authors":"Su","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tu et al., 2020] Rong-Cheng Tu, Xianling Mao, and Wei Wei. Mls3rduh: Deep unsupervised hashing via manifold based lo- cal semantic similarity structure reconstructing. In IJCAI, pages 3466-3472, 2020.","title":"Mls3rduh: Deep unsupervised hashing via manifold based local semantic similarity structure reconstructing","authors":"Rong-Cheng Tu, Xianling Mao, Wei Wei","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/479"},{"position":33,"raw":"[Wang et al., 2019] Xinshao Wang, Yang Hua, Elyor Kodirov, Gu- osheng Hu, Romain Garnier, and Neil M Robertson. Ranked list loss for deep metric learning. In CVPR, 2019.","title":"Ranked list loss for deep metric learning","authors":"Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, Neil M Robertson","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.","title":"Unsupervised feature learning via non-parametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2019] Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. Distillhash: Unsupervised deep hashing by distilling data pairs. In CVPR, 2019.","title":"Distillhash: Unsupervised deep hashing by distilling data pairs","authors":"Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zieba et al., 2018] Maciej Zieba, Piotr Semberecki, Tarek El- Gaaly, and Tomasz Trzcinski. Bingan: Learning compact binary descriptors with a regularized gan. In NeurIPS, 2018.","title":"Bingan: Learning compact binary descriptors with a regularized gan","authors":"Maciej Zieba, Piotr Semberecki, Tarek El-Gaaly, Tomasz Trzcinski","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}