{"type":"ScholarlyArticle","title":"Improved Deep Unsupervised Hashing with Fine-grained Semantic Similarity Mining for Multi-Label Image Retrieval","authors":[{"name":"Zeyu Ma","slug":"zeyu-ma","given":"Zeyu","family":"Ma","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"},{"name":"Xiao Luo","slug":"xiao-luo","given":"Xiao","family":"Luo","affiliation":"School of Mathematical Sciences , Peking University , Beijing , China"},{"name":"Yingjie Chen","slug":"yingjie-chen","given":"Yingjie","family":"Chen","affiliation":"School of Computer Science , Peking University , Beijing , China"},{"name":"Mixiao Hou","slug":"mixiao-hou","given":"Mixiao","family":"Hou","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"},{"name":"Jinxing Li","slug":"jinxing-li","given":"Jinxing","family":"Li","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China; Linklogis , Shenzhen , China"},{"name":"Minghua Deng","slug":"minghua-deng","given":"Minghua","family":"Deng","affiliation":"School of Mathematical Sciences , Peking University , Beijing , China"},{"name":"Guangming Lu","slug":"guangming-lu","given":"Guangming","family":"Lu","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"}],"abstract":"In this paper, we study deep unsupervised hashing, a critical problem for approximate nearest neighbor research. Most recent methods solve this problem by semantic similarity reconstruction for guiding hashing network learning or contrastive learning of hash codes. However, in multi-label scenarios, these methods usually either generate an inaccurate similarity matrix without reflection of similarity ranking or suffer from the violation of the underlying assumption in contrastive learning, resulting in limited retrieval performance. To tackle this issue, we propose a novel method termed HAMAN, which explores semantics from a fine-grained view to enhance the ability of multi-label image retrieval. In particular, we reconstruct the pairwise similarity structure by matching fine-grained patch features generated by the pre-trained neural network, serving as reliable guidance for similarity preserving of hash codes. Moreover, a novel conditional contrastive learning on hash codes is proposed to adopt self-supervised learning in multi-label scenarios. According to extensive experiments on three multi-label datasets, the proposed method outperforms a broad range of state-of-the-art methods.","doi":"10.24963/ijcai.2022/175","url":"https://ijcai.org/proceedings/2022/175","pdf":"https://ijcai.org/proceedings/2022/0175.pdf","markdown":"https://ijcai.org/proceedings/2022/175.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/175","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":175,"pages":{"first":1254,"last":1260},"keywords":["Data Mining: Information Retrieval","Computer Vision: Image and Video retrieval","Computer Vision: Representation Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986554","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984697","length_sec":343,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Pengguang Chen, Shu Liu, and Jiaya Jia. Jigsaw clustering for unsupervised visual representation learning. In CVPR, 2021.","title":"Jigsaw clustering for unsupervised visual representation learning","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus- wide: a real-world web image database from national uni- versity of singapore. In ACM CIVR, 2009.","title":"Nuswide: a real-world web image database from national university of singapore","authors":"Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng","year":"2009","venue":"ACM CIVR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. Stochastic generative hashing. In ICML, 2017.","title":"Stochastic generative hashing","authors":"Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, Le Song","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gionis et al., 1999] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In VLDB, 1999.","title":"Similarity search in high dimensions via hashing","authors":"Gionis","year":"1999","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2019] Yifan Gu, Shidong Wang, Haofeng Zhang, Yazhou Yao, and Li Liu. Clustering-driven unsuper- vised deep hashing for image retrieval. Neurocomputing, 368:114-123, 2019.","title":"Clustering-driven unsupervised deep hashing for image retrieval","authors":"Yifan Gu, Shidong Wang, Haofeng Zhang, Yazhou Yao, Li Liu","year":"2019","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hadsell et al., 2006] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.","title":"Dimensionality reduction by learning an invariant mapping","authors":"Raia Hadsell, Sumit Chopra, Yann Lecun","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huiskes and Lew, 2008] Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In ACMMM, 2008.","title":"The mir flickr retrieval evaluation","authors":"J Mark, Michael S Huiskes, Lew","year":"2008","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jang and Cho, 2021] Young Kyun Jang and Nam Ik Cho. Self-supervised product quantization for deep unsuper- vised image retrieval. In ICCV, 2021.","title":"Self-supervised product quantization for deep unsupervised image retrieval","authors":"Cho ; Jang, Kyun Young, Nam Ik Jang, Cho","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Shuyan Li, Xiu Li, Jiwen Lu, and Jie Zhou. Self-supervised video hashing via bidirectional transform- ers. In CVPR, 2021.","title":"Self-supervised video hashing via bidirectional transformers","authors":"Shuyan Li, Xiu Li, Jiwen Lu, Jie Zhou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2016] Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou. Learning compact binary descriptors with unsu- pervised deep neural networks. In CVPR, 2016.","title":"Learning compact binary descriptors with unsupervised deep neural networks","authors":"Kevin Lin, Jiwen Lu, Chu-Song Chen, Jie Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2021a] Xiao Luo, Daqing Wu, Chong Chen, Jin- wen Ma, and Minghua Deng. Deep unsupervised hashing by global and local consistency. In ICME, 2021.","title":"Deep unsupervised hashing by global and local consistency","authors":"Luo","year":"2021","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2021b] Xiao Luo, Daqing Wu, Zeyu Ma, Chong Chen, Minghua Deng, Jinwen Ma, Zhongming Jin, Jian- qiang Huang, and Xian-Sheng Hua. Cimon: Towards high-quality hash codes. In IJCAI, 2021.","title":"Cimon: Towards high-quality hash codes","authors":"Luo","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/125"},{"position":17,"raw":"[Luo et al., 2022] Xiao Luo, Haixin Wang, Chong Chen, Huasong Zhong, Hao Zhang, Minghua Deng, Jianqiang Huang, and Xian-Sheng Hua. A survey on deep hash- ing methods. ACM Transactions on Knowledge Discovery from Data, 2022.","title":"A survey on deep hashing methods","authors":"Xiao Luo, Haixin Wang, Chong Chen, Huasong Zhong, Hao Zhang, Minghua Deng, Jianqiang Huang, Xian-Sheng Hua","year":"2022","venue":"ACM Transactions on Knowledge Discovery from Data","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rodrigues et al., 2020] Josiane Rodrigues, Marco Cristo, and Juan G Colonna. Deep hashing for multi-label im- age retrieval: a survey. Artificial Intelligence Review, 53(7):5261-5307, 2020.","title":"Deep hashing for multi-label image retrieval: a survey","authors":"Josiane Rodrigues, Marco Cristo, Juan G Colonna","year":"2020","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2020] Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao. Auto-encoding twin-bottleneck hashing. In CVPR, 2020.","title":"Auto-encoding twin-bottleneck hashing","authors":"Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, Ling Shao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tu et al., 2019] Rong-Cheng Tu, Xian-Ling Mao, Bo-Si Feng, and Shu-Ying Yu. Object detection based deep un- supervised hashing. In IJCAI, 2019.","title":"Object detection based deep unsupervised hashing","authors":"Rong-Cheng Tu, Xian-Ling Mao, Bo-Si Feng, Shu-Ying Yu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/500"},{"position":21,"raw":"[Tu et al., 2020] Rong-Cheng Tu, Xian-Ling Mao, and Wei Wei. Mls3rduh: Deep unsupervised hashing via manifold based local semantic similarity structure reconstructing. In IJCAI, 2020.","title":"Mls3rduh: Deep unsupervised hashing via manifold based local semantic similarity structure reconstructing","authors":"Rong-Cheng Tu, Xian-Ling Mao, Wei Wei","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/479"},{"position":22,"raw":"[Tu et al., 2021a] Rong-Cheng Tu, Xian-Ling Mao, Jia-Nan Guo, Wei Wei, and Heyan Huang. Partial-softmax loss based deep hashing. In WWW, 2021.","title":"Partial-softmax loss based deep hashing","authors":"Tu","year":"2021","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tu et al., 2021b] Rong-Cheng Tu, Xian-Ling Mao, Cihang Kong, Zihang Shao, Ze-Lin Li, Wei Wei, and Heyan Huang. Weighted gaussian loss based hamming hashing. In ACMMM, 2021.","title":"Weighted gaussian loss based hamming hashing","authors":"Tu","year":"2021","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017] Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. A survey on learning to hash. IEEE transactions on pattern analysis and machine intelligence, 40(4):769-790, 2017.","title":"A survey on learning to hash","authors":"Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021] Yimu Wang, Bo Xue, Quan Cheng, Yuhui Chen, and Lijun Zhang. Deep unified cross- modality hashing by pairwise data alignment. In IJCAI, 2021.","title":"Deep unified crossmodality hashing by pairwise data alignment","authors":"Wang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Weiss et al., 2009] Yair Weiss, Antonio Torralba, and Robert Fergus. Spectral hashing. In NeurIPS, 2009.","title":"Spectral hashing","authors":"Yair Weiss, Antonio Torralba, Robert Fergus","year":"2009","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.","title":"Unsupervised feature learning via nonparametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2020] Yanzhao Xie, Yu Liu, Yangtao Wang, Lianli Gao, Peng Wang, and Ke Zhou. Label-attended hashing for multi-label image retrieval. In IJCAI, 2020.","title":"Label-attended hashing for multi-label image retrieval","authors":"Yanzhao Xie, Yu Liu, Yangtao Wang, Lianli Gao, Peng Wang, Ke Zhou","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/133"},{"position":29,"raw":"[Yang et al., 2018] Erkun Yang, Cheng Deng, Tongliang Liu, Wei Liu, and Dacheng Tao. Semantic structure-based unsupervised deep hashing. In IJCAI, 2018.","title":"Semantic structure-based unsupervised deep hashing","authors":"Erkun Yang, Cheng Deng, Tongliang Liu, Wei Liu, Dacheng Tao","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/148"},{"position":30,"raw":"[Yang et al., 2019] Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. Distillhash: Unsuper- vised deep hashing by distilling data pairs. In CVPR, 2019.","title":"Distillhash: Unsupervised deep hashing by distilling data pairs","authors":"Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}