{"type":"ScholarlyArticle","title":"Is My Object in This Video? Reconstruction-based Object Search in Videos","authors":[{"name":"Tan Yu","slug":"tan-yu","given":"Tan","family":"Yu","affiliation":"ROSE Lab , Interdisciplinary Graduate School , Nanyang Technological Univeristy , Singapore"},{"name":"Jingjing Meng","slug":"jingjing-meng","given":"Jingjing","family":"Meng","affiliation":"ROSE Lab , Interdisciplinary Graduate School , Nanyang Technological Univeristy , Singapore"},{"name":"Junsong Yuan","slug":"junsong-yuan","given":"Junsong","family":"Yuan","affiliation":"ROSE Lab , Interdisciplinary Graduate School , Nanyang Technological Univeristy , Singapore"}],"abstract":"This paper addresses the problem of video-level object instance search, which aims to retrieve the videos in the database that contain a given query object instance. Without prior knowledge about \"when\" and \"where\" an object of interest may appear in a video, determining \"whether\" a video contains the target object is computationally prohibitive, as it requires exhaustively matching the query against all possible spatial-temporal locations in each video that an object may appear. To alleviate the computational and memory cost, we propose the Reconstruction-based Object SEarch (ROSE) method.It characterizes a huge corpus of features of possible spatial-temporal locations in the video into the parameters of the reconstruction model. Since the memory cost of storing reconstruction model is much less than that of storing features of possible spatial-temporal locations in the video, the efficiency of the search is significantly boosted. Comprehensive experiments on three benchmark datasets demonstrate the promising performance of the proposed ROSE method.","doi":"10.24963/ijcai.2017/635","url":"https://ijcai.org/proceedings/2017/635","pdf":"https://ijcai.org/proceedings/2017/0635.pdf","markdown":"https://ijcai.org/proceedings/2017/635.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/635","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Robotics and Vision","track":"Main track","number":635,"pages":{"first":4551,"last":4557},"keywords":["Robotics and Vision: Robotics and Vision","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Araujo et al., 2014] A Araujo, Mina Makar, Vijay Chan- drasekhar, D Chen, S Tsai, Huizhong Chen, Roland Angst, and Bernd Girod. Efficient video search using image queries. In ICIP, pages 3082-3086. IEEE, 2014.","title":"Efficient video search using image queries","authors":"Mina Araujo, Vijay Makar, D Chandrasekhar, S Chen, Huizhong Tsai, Roland Chen, Bernd Angst, Girod","year":"2014","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhattacharjee et al., 2016] Sreyasee Das Bhattacharjee, Junsong Yuan, Yap-Peng Tan, and Ling-Yu Duan. Query- adaptive small object search using object proposals and shape-aware descriptors. IEEE Transactions on Multimedia, 18(4):726-737, 2016.","title":"Queryadaptive small object search using object proposals and shape-aware descriptors","authors":"Bhattacharjee","year":"2016","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chandrasekhar et al., 2014] Vijay Chandrasekhar, Wu Min, Xiao Li, Cheston Tan, Bappaditya Mandal, Liyuan Li, and Joo Hwee Lim. Efficient retrieval from large-scale ego- centric visual data using a sparse graph representation. In CVPR Workshops, pages 527-534, 2014.","title":"Efficient retrieval from large-scale egocentric visual data using a sparse graph representation","authors":"Vijay Chandrasekhar, Xiao Wu Min, Cheston Li, Bappaditya Tan, Liyuan Mandal, Joo Hwee Li, Lim","year":"2014","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cong et al., 2011] Yang Cong, Junsong Yuan, and Ji Liu. Sparse reconstruction cost for abnormal event detection. In CVPR, pages 3449-3456. IEEE, 2011.","title":"Sparse reconstruction cost for abnormal event detection","authors":"Yang Cong, Junsong Yuan, Ji Liu","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elhamifar et al., 2012] Ehsan Elhamifar, Guillermo Sapiro, and Rene Vidal. See all by looking at a few: Sparse mod- eling for finding representative objects. In CVPR, pages 1600-1607. IEEE, 2012.","title":"See all by looking at a few: Sparse modeling for finding representative objects","authors":"Ehsan Elhamifar, Guillermo Sapiro, Rene Vidal","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hayat et al., 2015] Munawar Hayat, Mohammed Ben- namoun, and Senjian An. Deep reconstruction models for image set classification. IEEE transactions on pattern analysis and machine intelligence, 37(4):713-727, 2015.","title":"Deep reconstruction models for image set classification","authors":"Munawar Hayat, Mohammed Bennamoun, Senjian An","year":"2015","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2015] Yuning Jiang, Jingjing Meng, Junsong Yuan, and Jiebo Luo. Randomized spatial context for ob- ject search. IEEE Trans. Image Processing, 24(6):1748- 1762, 2015.","title":"Randomized spatial context for object search","authors":"Jiang","year":"2015","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mairal et al., 2009] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th annual inter- national conference on machine learning, pages 689-696. ACM, 2009.","title":"Online dictionary learning for sparse coding","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Makhzani and Frey, 2013] Alireza Makhzani and Bren- dan J. Frey. k-sparse autoencoders. CoRR, abs/1312.5663, 2013.","title":"k-sparse autoencoders","authors":"Alireza Makhzani, Brendan J Frey","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Meng et al., 2016a] Jingjing Meng, Hongxing Wang, Jun- song Yuan, and Yap-Peng Tan. From keyframes to key ob- jects: Video summarization by representative object pro- posal selection. In CVPR, pages 1039-1048. IEEE, 2016.","title":"From keyframes to key objects: Video summarization by representative object proposal selection","authors":"Meng","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Meng et al., 2016b] Jingjing Meng, Junsong Yuan, Jiong Yang, Gang Wang, and Yap-Peng Tan. Object instance search in videos via spatio-temporal trajectory discovery. IEEE Transactions on Multimedia, 18(1):116-127, 2016.","title":"Object instance search in videos via spatio-temporal trajectory discovery","authors":"Meng","year":"2016","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ng, 2011] Andrew Ng. Sparse autoencoder. CS294A Lec- ture notes, 72:1-19, 2011.","title":"Sparse autoencoder","authors":"Andrew Ng","year":"2011","venue":"CS294A Lecture notes","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Hae-Sang Park and Chi-Hyuck Jun. A simple and fast algorithm for k-medoids clustering. Expert Systems with Applications, 36(2):3336-3341, 2009.","title":"A simple and fast algorithm for k-medoids clustering","authors":"Hae-Sang Park, Chi-Hyuck Jun","year":"2009","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sivic and Zisserman, 2009] Josef Sivic and Andrew Zisser- man. Efficient visual search of videos cast as text retrieval. IEEE transactions on pattern analysis and machine intel- ligence, 31(4):591-606, 2009.","title":"Efficient visual search of videos cast as text retrieval","authors":"Zisserman Sivic, Josef Sivic, Andrew Zisserman","year":"2009","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tao et al., 2015] Ran Tao, Arnold WM Smeulders, and Shih-Fu Chang. Attributes and categories for generic in- stance search from one example. In CVPR, pages 177- 186. IEEE, 2015.","title":"Attributes and categories for generic instance search from one example","authors":"Ran Tao, W M Arnold, Shih-Fu Smeulders, Chang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tolias et al., 2016] Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- pooling of cnn activations. In Proc. International Con- ference on Learning Representations (ICLR), 2016.","title":"Particular object retrieval with integral maxpooling of cnn activations","authors":"Giorgos Tolias, Ronan Sicre, Hervé Jégou","year":"2016","venue":"Proc. International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tu et al., 2014] Zhigang Tu, Nico Van Der Aa, Coert Van Gemeren, and Remco C Veltkamp. A combined post- filtering method to improve accuracy of variational optical flow estimation. Pattern Recognition, 47(5):1926-1940, 2014.","title":"A combined postfiltering method to improve accuracy of variational optical flow estimation","authors":"Zhigang Tu, Nico Van Der, Coert Aa, Remco C Van Gemeren, Veltkamp","year":"2014","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wright et al., 2009] John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recog- nition via sparse representation. IEEE transactions on pattern analysis and machine intelligence, 31(2):210-227, 2009.","title":"Robust face recognition via sparse representation","authors":"John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, Yi Ma","year":"2009","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xia et al., 2015] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstruc- tions for unsupervised outlier removal. In Proceedings of the IEEE International Conference on Computer Vision, pages 1511-1519, 2015.","title":"Learning discriminative reconstructions for unsupervised outlier removal","authors":"Yan Xia, Xudong Cao, Fang Wen, Gang Hua, Jian Sun","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang and Satoh, 2013] Yan Yang and Shin'ichi Satoh. Effi- cient instance search from large video database via sparse filters in subspaces. In 2013 IEEE International Confer- ence on Image Processing, pages 3972-3976. IEEE, 2013.","title":"Efficient instance search from large video database via sparse filters in subspaces","authors":"Yang, Satoh, Yan Yang, Shin'ichi Satoh","year":"2013","venue":"IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2015] Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, and Baining Guo. Unsu- pervised extraction of video highlights via robust recurrent auto-encoders. In Proceedings of the IEEE International Conference on Computer Vision, pages 4633-4641, 2015.","title":"Unsupervised extraction of video highlights via robust recurrent auto-encoders","authors":"Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, Baining Guo","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2017a] Tan Yu, Yuwei Wu, Sreyasee Das Bhat- tacharjee, and Junsong Yuan. Efficient object instance search using fuzzy objects matching. In AAAI, pages 4320-4326, 2017.","title":"Efficient object instance search using fuzzy objects matching","authors":"Yu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017b] Tan Yu, Yuwei Wu, and Junsong Yuan. Hope: Hierarchical object prototype encoding for efficient object instance search in videos. In CVPR. IEEE, 2017.","title":"Hope: Hierarchical object prototype encoding for efficient object instance search in videos","authors":"Yu","year":"2017","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu and Satoh, 2012] Cai-Zhi Zhu and Shin'ichi Satoh. Large vocabulary quantization for searching instances from videos. In ICMR, page 52, 2012.","title":"Large vocabulary quantization for searching instances from videos","authors":"Cai-Zhi Zhu, Shin'ichi Satoh","year":"2012","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zitnick and Dollár, 2014] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, pages 391-405. Springer, 2014.","title":"Edge boxes: Locating object proposals from edges","authors":"Lawrence Zitnick, Piotr Dollár","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}