{"type":"ScholarlyArticle","title":"Deep Multiple Instance Hashing for Object-based Image Retrieval","authors":[{"name":"Wanqing Zhao","slug":"wanqing-zhao","given":"Wanqing","family":"Zhao","affiliation":"School of information and technology Northwestern University , Shaanxi , China"},{"name":"Ziyu Guan","slug":"ziyu-guan","given":"Ziyu","family":"Guan","affiliation":"School of information and technology Northwestern University , Shaanxi , China"},{"name":"Hangzai Luo","slug":"hangzai-luo","given":"Hangzai","family":"Luo","affiliation":"School of information and technology Northwestern University , Shaanxi , China"},{"name":"Jinye Peng","slug":"jinye-peng","given":"Jinye","family":"Peng","affiliation":"Department of Computer Science UNC-Charlotte , NC28223 , USA"},{"name":"Jianping Fan","slug":"jianping-fan","given":"Jianping","family":"Fan","affiliation":"Department of Computer Science UNC-Charlotte , NC28223 , USA"}],"abstract":"Multi-keyword query is widely supported in text search engines. However, an analogue in image retrieval systems, multi-object query, is rarely studied. Meanwhile, traditional object-based image retrieval methods often involve multiple steps separately and need expensive location labeling for detecting objects. In this work, we propose a weakly-supervised Deep Multiple Instance Hashing (DMIH) framework for object-based image retrieval. DMIH integrates object detection and hashing learning on the basis of a popular CNN model to build the end-to-end relation between a raw image and the binary hashing codes of multiple objects in it. Specifically, we cast the object detection of each object class as a binary multiple instance learning problem where instances are object proposals extracted from multi-scale convolutional feature maps. For hashing training, we sample image pairs to learn their semantic relationships in terms of hash codes of the most probable proposals for owned labels as guided by object predictors. The two objectives benefit each other in learning. DMIH outperforms state-of-the-arts on public benchmarks for object-based image retrieval and achieves promising results for multi-object queries.","doi":"10.24963/ijcai.2017/490","url":"https://ijcai.org/proceedings/2017/490","pdf":"https://ijcai.org/proceedings/2017/0490.pdf","markdown":"https://ijcai.org/proceedings/2017/490.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/490","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":490,"pages":{"first":3504,"last":3510},"keywords":["Machine Learning: Deep Learning","Machine Learning: Multi-instance/Multi-label/Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Cheng et al., 2014] Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip Torr. Bing: Binarized normed gradients for objectness estimation at 300fps. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 3286-3293, 2014.","title":"Bing: Binarized normed gradients for objectness estimation at 300fps","authors":"Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, Philip Torr","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chum et al., 2008] Ondrej Chum, James Philbin, Andrew Zisserman, et al. Near duplicate image detection: min- hash and tf-idf weighting. In British Machine Vision Con- ference, 2008.","title":"Near duplicate image detection: minhash and tf-idf weighting","authors":"Chum","year":"2008","venue":"British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Everingham et al., 2007] Mark Everingham, Andrew Zis- serman, Christopher KI Williams, Luc Van Gool, Moray Allan, Christopher M Bishop, Olivier Chapelle, Navneet Dalal, Thomas Deselaers, Gyuri Dorkó, et al. The pas- cal visual object classes challenge 2007 (voc2007) results. 2007.","title":"The pascal visual object classes challenge","authors":"Mark Everingham, Andrew Zisserman, K I Christopher, Luc Williams, Moray Van Gool, Christopher M Allan, Olivier Bishop, Navneet Chapelle, Thomas Dalal, Gyuri Deselaers, Dorkó","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gionis et al., 2000] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In VLDB, volume 99, pages 518-529, 2000.","title":"Similarity search in high dimensions via hashing","authors":"Gionis","year":"2000","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hadsell et al., 2006] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 1735-1742, 2006.","title":"Dimensionality reduction by learning an invariant mapping","authors":"Raia Hadsell, Sumit Chopra, Yann Lecun","year":"2006","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kraus et al., 2016] Oren Z Kraus, Jimmy Lei Ba, and Bren- dan J Frey. Classifying and segmenting microscopy im- ages with deep multiple instance learning. Bioinformatics, 32(12):i52-i59, 2016.","title":"Classifying and segmenting microscopy images with deep multiple instance learning","authors":"Oren Z Kraus, Jimmy Lei Ba, Brendan J Frey","year":"2016","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kulis and Darrell, 2009] Brian Kulis and Trevor Darrell. Learning to hash with binary reconstructive embeddings. In Advances in Neural Information Processing Systems, pages 1042-1050, 2009.","title":"Learning to hash with binary reconstructive embeddings","authors":"Brian Kulis, Trevor Darrell","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee et al., 2010] David C Lee, Qifa Ke, and Michael Isard. Partition min-hash for partial duplicate image discovery. In European Conference on Computer Vision, pages 648- 662, 2010.","title":"Partition min-hash for partial duplicate image discovery","authors":"Qifa David C Lee, Michael Ke, Isard","year":"2010","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li and Liu, 2015] Fei Li and Rujie Liu. Multi-graph multi- instance learning with soft label consistency for object- based image retrieval. In 2015 IEEE International Con- ference on Multimedia and Expo, pages 1-6, 2015.","title":"Multi-graph multiinstance learning with soft label consistency for objectbased image retrieval","authors":"Li, Liu ; ] Fei Li, Rujie Liu","year":"2015","venue":"2015 IEEE International Conference on Multimedia and Expo","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2015] Kevin Lin, Huei-Fang Yang, Jen-Hao Hsiao, and Chu-Song Chen. Deep learning of binary hash codes for fast image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 27-35, 2015.","title":"Deep learning of binary hash codes for fast image retrieval","authors":"Kevin Lin, Huei-Fang Yang, Jen-Hao Hsiao, Chu-Song Chen","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2012] Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and Shih-Fu Chang. Supervised hashing with ker- nels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2074-2081, 2012.","title":"Supervised hashing with kernels","authors":"Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, Shih-Fu Chang","year":"2012","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision, pages 21-37, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rahmani et al., 2008] Rouhollah Rahmani, Sally A Gold- man, Hui Zhang, John Krettek, and Jason E Fritts. Local- ized content-based image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1902, 2008.","title":"Localized content-based image retrieval","authors":"Rahmani","year":"2008","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2016] Weiqiang Ren, Kaiqi Huang, Dacheng Tao, and Tieniu Tan. Weakly supervised large scale object localization with multiple instance learning and bag split- ting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):405-416, 2016.","title":"Weakly supervised large scale object localization with multiple instance learning and bag splitting","authors":"Weiqiang Ren, Kaiqi Huang, Dacheng Tao, Tieniu Tan","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Russell et al., 2006] Bryan C Russell, William T Freeman, Alexei A Efros, Josef Sivic, and Andrew Zisserman. Us- ing multiple segmentations to discover objects and their extent in image collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 1605-1614, 2006.","title":"Using multiple segmentations to discover objects and their extent in image collections","authors":"William T Bryan C Russell, Alexei A Freeman, Josef Efros, Andrew Sivic, Zisserman","year":"2006","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Advances in Neural In- formation Processing Systems, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Smeulders et al., 2000] Arnold WM Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 22(12):1349-1380, 2000.","title":"Content-based image retrieval at the end of the early years","authors":"W M Arnold, Marcel Smeulders, Simone Worring, Amarnath Santini, Ramesh Gupta, Jain","year":"2000","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Weiss et al., 2008] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In Advances in Neural Informa- tion Processing Systems, pages 1753-1760, 2008.","title":"Spectral hashing","authors":"Yair Weiss, Antonio Torralba, Rob Fergus","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2015] Jiajun Wu, Yinan Yu, Chang Huang, and Kai Yu. Deep multiple instance learning for image classi- fication and auto-annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3460-3469, 2015.","title":"Deep multiple instance learning for image classification and auto-annotation","authors":"Jiajun Wu, Yinan Yu, Chang Huang, Kai Yu","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2009] Dan Zhang, Fei Wang, Zhenwei Shi, and Changshui Zhang. Interactive localized content based image retrieval with multiple-instance active learning. Pat- tern Recognition, 43(2):478-484, 2009.","title":"Interactive localized content based image retrieval with multiple-instance active learning","authors":"Dan Zhang, Fei Wang, Zhenwei Shi, Changshui Zhang","year":"2009","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2015] Fang Zhao, Yongzhen Huang, Liang Wang, and Tieniu Tan. Deep semantic ranking based hash- ing for multi-label image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1556-1564, 2015.","title":"Deep semantic ranking based hashing for multi-label image retrieval","authors":"Fang Zhao, Yongzhen Huang, Liang Wang, Tieniu Tan","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zheng et al., 2006] Qing-Fang Zheng, Wei-Qiang Wang, and Wen Gao. Effective and efficient object-based image retrieval using visual phrases. In Proceedings of the 14th ACM International Conference on Multimedia, pages 77- 80, 2006.","title":"Effective and efficient object-based image retrieval using visual phrases","authors":"Qing-Fang Zheng, Wei-Qiang Wang, Wen Gao","year":"2006","venue":"Proceedings of the 14th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}