{"type":"ScholarlyArticle","title":"Supervised Set-to-Set Hashing in Visual Recognition","authors":[{"name":"I-Hong Jhuo","slug":"i-hong-jhuo","given":"I-Hong","family":"Jhuo","affiliation":""}],"abstract":"Visual data, such as an image or a sequence of video frames, is often naturally represented as a point set. In this paper, we consider the fundamental problem of finding a nearest set from a collection of sets, to a query set. This problem has obvious applications in large-scale visual retrieval and recognition, and also in applied fields beyond computer vision. One challenge stands out in solving the problem---set representation and measure of similarity. Particularly, the query set and the sets in dataset collection can have varying cardinalities. The training collection is large enough such that linear scan is impractical. We propose a simple representation scheme that encodes both statistical and structural information of the sets. The derived representations are integrated in a kernel framework for flexible similarity measurement. For the query set process, we adopt a learning-to-hash pipeline that turns the kernel representations into hash bits based on simple learners, using multiple kernel learning. Experiments on two visual retrieval datasets show unambiguously that our set-to-set hashing framework outperforms prior methods that do not take the set-to-set search setting.","doi":"10.24963/ijcai.2019/113","url":"https://ijcai.org/proceedings/2019/113","pdf":"https://ijcai.org/proceedings/2019/0113.pdf","markdown":"https://ijcai.org/proceedings/2019/113.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/113","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":113,"pages":{"first":803,"last":810},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Multidisciplinary Topics and Applications: Information Retrieval"],"videos":[],"references":[{"position":1,"raw":"[Arandjelovic et al., 2005] O. Arandjelovic, G. Shakhnarovich, J. Fisher, R. Cipolla, and T. Dar- rell. Face recognition with image sets using manifold density divergence. In CVPR, 2005.","title":"Face recognition with image sets using manifold density divergence","authors":"O Arandjelovic, G Shakhnarovich, J Fisher, R Cipolla, T Darrell","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Basri et al., 2011] R. Basri, T. Hassner, and L. Zelnik- Manor. Approximate nearest subspace search. In IEEE Trans PAMI, 2011.","title":"Approximate nearest subspace search","authors":"R Basri, T Hassner, L Zelnik-Manor","year":"2011","venue":"IEEE Trans PAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bauml et al., 2013] M. Bauml, M. Tapaswi, and R. Stiefel- hagen. Semi-supervised learning with constraints for per- son identification in multimedia data. In CVPR, 2013.","title":"Semi-supervised learning with constraints for person identification in multimedia data","authors":"M Bauml, M Tapaswi, R Stiefelhagen","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berretti et al., 2010] S. Berretti, A. D. Bimbo, and P. Pala. 3d face recognition using isogeodesic stripes. In IEEE Trans. PAMI, 2010.","title":"3d face recognition using isogeodesic stripes","authors":"S Berretti, A D Bimbo, P Pala","year":"2010","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cevikalp and Triggs, 2010] H. Cevikalp and B. Triggs. Face recognition based on image sets. In CVPR, 2010.","title":"Face recognition based on image sets","authors":"H Cevikalp, B Triggs","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Charikar, 2002] M. S. Charikar. Similarity estimation tech- niques from rounding algorithms. In STOC, 2002.","title":"Similarity estimation techniques from rounding algorithms","authors":"M S Charikar","year":"2002","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Datar et al., 2004] M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni. Locality sensitive hashing scheme based on p-stable distributions. In SCG, 2004.","title":"Locality sensitive hashing scheme based on p-stable distributions","authors":"M Datar, N Immorlica, P Indyk, V S Mirrokni","year":"2004","venue":"SCG","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Freund and Schapire, 1995] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In Computational Learning Theory, 1995.","title":"A decision-theoretic generalization of on-line learning and an application to boosting","authors":"Y Freund, R Schapire","year":"1995","venue":"Computational Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gartner, 2003] T. Gartner. A survey of kernels for structured data. In SIGKDD, 2003.","title":"A survey of kernels for structured data","authors":"T Gartner","year":"2003","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gong and Lazebnik, 2011] Y. Gong and S. Lazebnik. Iter- ative quantization: a procrustean approach to learning bi- nary codes. In CVPR, 2011.","title":"Iterative quantization: a procrustean approach to learning binary codes","authors":"Y Gong, S Lazebnik","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hamm and Lee, 2009] J. Hamm and D. Lee. Grassmann discriminant analysis: a unifying view on subspace-based learning. In ICML, 2009.","title":"Grassmann discriminant analysis: a unifying view on subspace-based learning","authors":"J Hamm, D Lee","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2011] Y. Hu, A. S. Mian, and R. Owens. Sparse approximated nearest points for image set classification. In CVPR, 2011.","title":"Sparse approximated nearest points for image set classification","authors":"Y Hu, A S Mian, R Owens","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Indyk and Motwani, 1998] P. Indyk and R. Motwani. Ap- proximate nearest neighbors: towards removing the curse of dimensionality. In STOC, 1998.","title":"Approximate nearest neighbors: towards removing the curse of dimensionality","authors":"P Indyk, R Motwani","year":"1998","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jayasumana et al., 2013] S. Jayasumana, R. Hartley, M. Salzmann, H. Li, and M. Harandi. Kernel methods on the riemannian manifold of symmetric positive definite matrices. In CVPR, 2013.","title":"Kernel methods on the riemannian manifold of symmetric positive definite matrices","authors":"S Jayasumana, R Hartley, M Salzmann, H Li, M Harandi","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2007] T.-K. Kim, J. Kittler, and R. Cipolla. Dis- criminative learning and recognition of image set classes using canonical correlations. In IEEE Trans. PAMI, 2007.","title":"Discriminative learning and recognition of image set classes using canonical correlations","authors":"T.-K Kim, J Kittler, R Cipolla","year":"2007","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulis and Darrell, 2009] B. Kulis and T. Darrell. Learning to hash with binary reconstructive embeddings. In NIPS, 2009.","title":"Learning to hash with binary reconstructive embeddings","authors":"B Kulis, T Darrell","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kulis et al., 2009] B. Kulis, P. Jain, and K. Grauman. Fast similarity search for learned metrics. In IEEE Trans. PAMI, 2009.","title":"Fast similarity search for learned metrics","authors":"B Kulis, P Jain, K Grauman","year":"2009","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2015] Y. Li, R. Wang, Z. Huang, S. Shan, and X. Chen. Face video retrieval with image query via hash- ing across euclidean space and riemannian manifold. In CVPR, 2015.","title":"Face video retrieval with image query via hashing across euclidean space and riemannian manifold","authors":"Y Li, R Wang, Z Huang, S Shan, X Chen","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2016] W.-J. Li, S. Wang, and W.-C. Kang. Fea- ture learning based deep supervised hashing with pairwise labels. In IJCAI, 2016.","title":"Feature learning based deep supervised hashing with pairwise labels","authors":"W.-J Li, S Wang, W.-C Kang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/245"},{"position":20,"raw":"[Lin et al., 2013] G. Lin, C. Shen, D. Suter, and A. V. D. Hengel. A general two-step approach to learning-based hashing. In ICCV, 2013.","title":"A general two-step approach to learning-based hashing","authors":"G Lin, C Shen, D Suter, A V D Hengel","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liong et al., 2015] V. E. Liong, J. Lu, G. Wang, P. Moulin, and J. Zhou. Deep hashing for compact binary codes learn- ing. In CVPR, 2015.","title":"Deep hashing for compact binary codes learning","authors":"V E Liong, J Lu, G Wang, P Moulin, J Zhou","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2012a] W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang. Supervised hashing with kernels. In CVPR, 2012.","title":"Supervised hashing with kernels","authors":"Liu","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2012b] W. Liu, J. Wang, Y. Mu, S. Kumar, and S.-F. Chang. Compact hyperplane hashing with bilinear functions. In ICML, 2012.","title":"Compact hyperplane hashing with bilinear functions","authors":"Liu","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2014] M. Liu, S. Shan, R. Wang, and X. Chen. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In CVPR, 2014.","title":"Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition","authors":"M Liu, S Shan, R Wang, X Chen","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lu et al., 2013] J. Lu, G. Wang, and P. Moulin. Image set classification using holistic multiple order statistics fea- tures and localized multi-kernel metric learning. In ICCV, 2013.","title":"Image set classification using holistic multiple order statistics features and localized multi-kernel metric learning","authors":"J Lu, G Wang, P Moulin","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[M. Rastegari J. Choi and Davis, 2013] D. Hal M. Rastegari J. Choi, S. Fakhraei and L. Davis. Predictable dual-view hashing. In ICML, 2013.","title":"Predictable dual-view hashing","authors":"M Rastegari, J Choi, ; ] D Davis, M Hal, J Rastegari, S Choi, L Fakhraei, Davis","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Masci et al., 2013] J. Masci, M. Bronstein, A. Bronstein, and J. Schmidhuber. Multimodal similarity-preserving hashing. In ICLR, 2013.","title":"Multimodal similarity-preserving hashing","authors":"J Masci, M Bronstein, A Bronstein, J Schmidhuber","year":"2013","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Moghaddam and Shakhnarovich, 2002] B. Moghaddam and G. Shakhnarovich. Boosted dyadic kernel discrimi- nants. In NIPS, 2002.","title":"Boosted dyadic kernel discriminants","authors":"B Moghaddam, G Shakhnarovich","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mu et al., 2010] Y. Mu, J. Shen, and S. Yan. Weakly- supervised hashing in kernel space. In CVPR, 2010.","title":"Weaklysupervised hashing in kernel space","authors":"Y Mu, J Shen, S Yan","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Norouzi and Fleet, 2011] M. Norouzi and D. J. Fleet. Min- imal loss hashing for compact binary codes. In ICML, 2011.","title":"Minimal loss hashing for compact binary codes","authors":"M Norouzi, D J Fleet","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Oliva and Torralba, 2001] A. Oliva and A. Torralba. Model- ing the shape of the scene: a holistic representation of the spatial envelope. In IJCV, 2001.","title":"Modeling the shape of the scene: a holistic representation of the spatial envelope","authors":"A Oliva, A Torralba","year":"2001","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rastegari et al., 2012] M. Rastegari, A. Farhadi, and D. Forsyth. Attribute discovery via predictable discrimi- native binary codes. In ECCV, 2012.","title":"Attribute discovery via predictable discriminative binary codes","authors":"M Rastegari, A Farhadi, D Forsyth","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Relja and Zisserman, 2014] A. Relja and A. Zisserman. Ex- tremely low bit-rate nearest neighbor search using a set compression tree. In IEEE Trans. PAMI, 2014.","title":"Extremely low bit-rate nearest neighbor search using a set compression tree","authors":"A Relja, A Zisserman","year":"2014","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sablayrolles et al., 2017] A. Sablayrolles, M. Douze, H. Ju, and N. Usunier. How should we evaluate supervised hash- ing? In ICASSP, 2017.","title":"How should we evaluate supervised hashing?","authors":"A Sablayrolles, M Douze, H Ju, N Usunier","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Salakhutdinov and Hinton, 2007] R. Salakhutdinov and G. E. Hinton. Learning a nonlinear embedding by preserving class neighborhood structure. In ICAIS, 2007.","title":"Learning a nonlinear embedding by preserving class neighborhood structure","authors":"R Salakhutdinov, G E Hinton","year":"2007","venue":"ICAIS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Salakhutdinov and Hinton, 2009] R. Salakhutdinov and G. Hinton. Semantic hashing. In Inter'l. J. Approximate Reasoning, 2009.","title":"Semantic hashing","authors":"R Salakhutdinov, G Hinton","year":"2009","venue":"Inter'l. J. Approximate Reasoning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Scholkopf et al., 1997] B. Scholkopf, A. Smola, and K.-R. Muller. Kernel principle component analysis. In ICANN, 1997.","title":"Kernel principle component analysis","authors":"B Scholkopf, A Smola, K.-R Muller","year":"1997","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Shen et al., 2015] F. Shen, C. Shen, W. Liu, and H.T. Shen. Supervised discrete hashing. In CVPR, 2015.","title":"Supervised discrete hashing","authors":"F Shen, C Shen, W Liu, H T Shen","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Sivic et al., 2005] J. Sivic, M. Everingham, and A. Zisser- man. Person spotting: video shot retrieval for face sets. In Image and Video Retrieval, 2005.","title":"Person spotting: video shot retrieval for face sets","authors":"J Sivic, M Everingham, A Zisserman","year":"2005","venue":"Image and Video Retrieval","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sun et al., 2014] X. Sun, Q. Qu, N. M. Nasrabadi, and T. D. Tran. Structured priors for sparse-representation-based hy- perspectral image classification. In IEEE GRSL, 2014.","title":"Structured priors for sparse-representation-based hyperspectral image classification","authors":"X Sun, Q Qu, N M Nasrabadi, T D Tran","year":"2014","venue":"IEEE GRSL","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sun et al., 2015] X. Sun, N. M. Nasrabadi, and T. D. Tran. Task-driven dictionary learning for hyperspectral image classification with structured sparsity constraints. In IEEE Trans. GRSL, 2015.","title":"Task-driven dictionary learning for hyperspectral image classification with structured sparsity constraints","authors":"Sun","year":"2015","venue":"IEEE Trans. GRSL","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Sun et al., 2017] X. Sun, N. M. Nasrabadi, and T. D. Tran. Supervised multilayer sparse coding networks for image classification. In CoRR, 2017.","title":"Supervised multilayer sparse coding networks for image classification","authors":"X Sun, N M Nasrabadi, T D Tran","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Tenenbaum et al., 2000] J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global geometric framework for nonlin- ear dimensionality reduction. In Science, 2000.","title":"A global geometric framework for nonlinear dimensionality reduction","authors":"J B Tenenbaum, V Silva, J C Langford","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Tseng, 2001] P. Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. In J. of optimization theory and applications, 2001.","title":"Convergence of a block coordinate descent method for nondifferentiable minimization","authors":"P Tseng","year":"2001","venue":"J. of optimization theory and applications","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Tuzel et al., 2007] O. Tuzel, F. Porikli, and P. Meer. Human detection via classification on riemannian manifolds. In CVPR, 2007.","title":"Human detection via classification on riemannian manifolds","authors":"O Tuzel, F Porikli, P Meer","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Vemulapalli et al., 2013] R. Vemulapalli, J. K. Pillai, and R. Chellappa. Kernel learning for extrinsic classification of manifold features. In CVPR, 2013.","title":"Kernel learning for extrinsic classification of manifold features","authors":"R Vemulapalli, J K Pillai, R Chellappa","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2010] J. Wang, S. Kumar, and S.-F. Chang. Sequential projection learning for hashing with compact codes. In ICML, 2010.","title":"Sequential projection learning for hashing with compact codes","authors":"J Wang, S Kumar, S.-F Chang","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"Wang et al., 20114]","title":null,"authors":"Wang","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"J. Wang, H. T. Shen, J. Song, and J. Ji. Hashing for similarity search: a survey. In arXiv:1408.2927, 20114.","title":"Hashing for similarity search: a survey","authors":"J Wang, H T Shen, J Song, J Ji","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Wang et al., 2012a] J. Wang, S. Kumar, and S.-F. Chang. Semi-supervised hashing for large scale search. In IEEE Trans. PAMI, 2012.","title":"Semi-supervised hashing for large scale search","authors":"Wang","year":"2012","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Wang et al., 2012b] R. Wang, H. Guo, L. S. Davis, and Q. Dai. Covariance discriminative learning: a natural and efficient approach to image set classification. In CVPR, 2012.","title":"Covariance discriminative learning: a natural and efficient approach to image set classification","authors":"Wang","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Wang et al., 2013] X. Wang, S. Atev, J. Wright, and G. Ler- man. Fast subspace search via grassmannian based hash- ing. In ICCV, 2013.","title":"Fast subspace search via grassmannian based hashing","authors":"Wang","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Wang et al., 2016] J. Wang, W. Liu, S. Kumar, and S.-F. Chang. Learning to hash for indexing big data -a survey. In Proceedings of the IEEE. 104(1):34-57, 2016.","title":"Learning to hash for indexing big data -a survey","authors":"J Wang, W Liu, S Kumar, S.-F Chang","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Weiss et al., 2008] Y. Weiss, A. Torralba, and R. Fergus. Spectral hashing. In NIPS, 2008.","title":"Spectral hashing","authors":"Y Weiss, A Torralba, R Fergus","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Weng et al., 2019] L. Weng, I.-H. Jhuo, and W.-H. Cheng. Perceptual hashing for large-scale multimedia search. In John Wiley Sons, 2019.","title":"Perceptual hashing for large-scale multimedia search","authors":"L Weng, I.-H Jhuo, W.-H Cheng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Zhang and Li, 2014] D. Zhang and W.-J. Li. Large-scale supervised multimodal hashing with semantic correlation maximization. In AAAI, 2014.","title":"Large-scale supervised multimodal hashing with semantic correlation maximization","authors":"D Zhang, W.-J Li","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Zhou et al., 2009] Z.H. Zhou, Y.-Y. Sun, and Y.-F. Li. Multi-instance learning by treating instances as non-i.i.d. samples. In ICML, 2009.","title":"Multi-instance learning by treating instances as non-i.i.d. samples","authors":"Z H Zhou, Y.-Y Sun, Y.-F Li","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}