{"type":"ScholarlyArticle","title":"Dynamically Visual Disambiguation of Keyword-based Image Search","authors":[{"name":"Yazhou Yao","slug":"yazhou-yao","given":"Yazhou","family":"Yao","affiliation":"Nanjing University of Science and Technology , Nanjing , China; Inception Institute of Artificial Intelligence , Abu Dhabi , UAE"},{"name":"Zeren Sun","slug":"zeren-sun","given":"Zeren","family":"Sun","affiliation":"Nanjing University of Science and Technology , Nanjing , China"},{"name":"Fumin Shen","slug":"fumin-shen","given":"Fumin","family":"Shen","affiliation":"University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Li Liu","slug":"li-liu","given":"Li","family":"Liu","affiliation":"Inception Institute of Artificial Intelligence , Abu Dhabi , UAE"},{"name":"Limin Wang","slug":"limin-wang","given":"Limin","family":"Wang","affiliation":"Nanjing University , Nanjing , China"},{"name":"Fan Zhu","slug":"fan-zhu","given":"Fan","family":"Zhu","affiliation":"Inception Institute of Artificial Intelligence , Abu Dhabi , UAE"},{"name":"Lizhong Ding","slug":"lizhong-ding","given":"Lizhong","family":"Ding","affiliation":"Inception Institute of Artificial Intelligence , Abu Dhabi , UAE"},{"name":"Gangshan Wu","slug":"gangshan-wu","given":"Gangshan","family":"Wu","affiliation":"Nanjing University , Nanjing , China"},{"name":"Ling Shao","slug":"ling-shao","given":"Ling","family":"Shao","affiliation":"Inception Institute of Artificial Intelligence , Abu Dhabi , UAE"}],"abstract":"Due to the high cost of manual annotation, learning directly from the web has attracted broad attention. One issue that limits their performance is the problem of visual polysemy. To address this issue, we present an adaptive multi-model framework that resolves polysemy by visual disambiguation. Compared to existing methods, the primary advantage of our approach lies in that our approach can adapt to the dynamic changes in the search results. Our proposed framework consists of two major steps: we first discover and dynamically select the text queries according to the image search results, then we employ the proposed saliency-guided deep multi-instance learning network to remove outliers and learn classification models for visual disambiguation. Extensive experiments demonstrate the superiority of our proposed approach.","doi":"10.24963/ijcai.2019/140","url":"https://ijcai.org/proceedings/2019/140","pdf":"https://ijcai.org/proceedings/2019/0140.pdf","markdown":"https://ijcai.org/proceedings/2019/140.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/140","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":140,"pages":{"first":996,"last":1002},"keywords":["Computer Vision: Language and Vision","Computer Vision: Computer Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Deng, 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei, Imagenet: A large-scale hierarchi- cal image database. CVPR, 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Yao, 2018] Yazhou Yao, Jian Zhang, Fumin Shen, Wankou Yang, Pu Huang and Zhenmin Tang, Discovering and Distinguishing Multiple Visual Senses for Polysemous Words. AAAI, 523-530, 2018.","title":"Discovering and Distinguishing Multiple Visual Senses for Polysemous Words","authors":"Yazhou Yao, Jian Zhang, Fumin Shen, Wankou Yang, Pu Huang, Zhenmin Tang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Shen, 2019] Yazhou Yao, Fumin Shen, Jian Zhang, Li Liu, Zhenmin Tang and Ling Shao, Extracting Multiple Visual Senses for Web Learning. TMM, 21(1): 184-196, 2019.","title":"Extracting Multiple Visual Senses for Web Learning","authors":"Yazhou Yao, Fumin Shen, Jian Zhang, Li Liu, Zhenmin Tang, Ling Shao","year":"2019","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen, 2013] Xinlei Chen, Abhinav Gupta, \"Neil: Extract- ing visual knowledge from web data.\" ICCV, 1409-1416, 2013.","title":"Neil: Extracting visual knowledge from web data","authors":"Xinlei Chen, Abhinav Gupta","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen, 2015] Xinlei Chen, Alan Ritter, Abhinav Gupta, Tom Mitchell, Sense discovery via co-clustering on images and text. CVPR, 5298-5306, 2015.","title":"Sense discovery via co-clustering on images and text","authors":"Xinlei Chen, Alan Ritter, Abhinav Gupta, Tom Mitchell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Shu, 2015] Xiangbo Shu, Guojun Qi, Jinhui Tang, Jing- dong Wang, Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation. ACM MM, 35-44, 2015.","title":"Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation","authors":"Xiangbo Shu, Guojun Qi, Jinhui Tang, Jingdong Wang","year":"2015","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Wan, 2009] Kong-Wah Wan, Ah-Hwee Tan, Joo-Hwee Lim, Liangtien Chia, Sujoy Roy, A latent model for visual disambiguation of keyword-based image search. BMVC, 163-170, 2009.","title":"A latent model for visual disambiguation of keyword-based image search","authors":"Wan ; Kong-Wah Wan, Ah-Hwee Tan, Joo-Hwee Lim, Liangtien Chia, Sujoy Roy","year":"2009","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Min, 2016] Weiqing Min, Shuqiang Jiang, Jitao Sang, Huayang Wang, Xinda Liu, \"Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration\" TMM, 19(5): 1100- 1113, 2017.","title":"Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration","authors":"Weiqing Min, Shuqiang Jiang, Jitao Sang, Huayang Wang, Xinda Liu","year":"2017","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Zhang, 2017] Guo-Sen Xie, Xu-Yao Zhang, Shuicheng Yan, Cheng-Lin Liu, \"SDE: A novel selective, discrim- inative and equalizing feature representation for visual recognition\" IJCV, 124(2): 145-168, 2017.","title":"SDE: A novel selective, discriminative and equalizing feature representation for visual recognition","authors":"Guo-Sen Xie, Xu-Yao Zhang, Shuicheng Yan, Cheng-Lin Liu","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Xie, 2019] Guo-Sen Xie, Li Liu, Xiao-Bo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao, \"At- tentive Region Embedding Network for Zero-shot Learn- ing\" CVPR, 1245-1253, 2019.","title":"Attentive Region Embedding Network for Zero-shot Learning","authors":"Guo-Sen Xie, Li Liu, Xiao-Bo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, Ling Shao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Shu, 2018] Xiangbo Shu, Jinhui Tang, Guo-Jun Qi, Wei Liu, Jian Yang, \"Hierarchical Long Short-Term Con- current Memory for Human Interaction Recognition\" ArXiv:1811.00270, 2018.","title":"Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition","authors":"Xiangbo Shu, Jinhui Tang, Guo-Jun Qi, Wei Liu, Jian Yang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu, 2017] Guosheng Hu, Xiaojiang Peng, Yongxin Yang, Timothy Hospedales, Jakob Verbeek, \"Frankenstein: Learning deep face representations using small data\" TIP, 27(1): 293-303, 2017.","title":"Frankenstein: Learning deep face representations using small data","authors":"Guosheng Hu, Xiaojiang Peng, Yongxin Yang, Timothy Hospedales, Jakob Verbeek","year":"2017","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hua, 2017] Guosheng Hu, Yang Hua, Yang Yuan, Zhihong Zhang, \"Attribute-enhanced face recognition with neural tensor fusion networks\" ICCV, 3744-3753, 2017.","title":"Attribute-enhanced face recognition with neural tensor fusion networks","authors":"Guosheng Hu, Yang Hua, Yang Yuan, Zhihong Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu, 2019] Yazhou Yao, Fumin Shen, Jian Zhang, Li Liu, Zhenmin Tang and Ling Shao, Extracting Privileged In- formation for Enhancing Classifier Learning. TIP, 28(1): 436-450, 2019.","title":"Extracting Privileged Information for Enhancing Classifier Learning","authors":"Yazhou Yao, Fumin Shen, Jian Zhang, Li Liu, Zhenmin Tang, Ling Shao","year":"2019","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tang, 2018] Yazhou Yao, Jian Zhang, Fumin Shen, Wankou Yang, Xiansheng Hua and Zhenmin Tang, Extracting Privileged Information from Untagged Corpora for Clas- sifier Learning. IJCAI, 1085-1091, 2018.","title":"Extracting Privileged Information from Untagged Corpora for Classifier Learning","authors":"Yazhou Yao, Jian Zhang, Fumin Shen, Wankou Yang, Xiansheng Hua, Zhenmin Tang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/151"},{"position":16,"raw":"[Loeff, 2006] Nicolas Loeff, David Forsyth, Discriminat- ing image senses by clustering with multimodal features. ACL, 547-554, 2006.","title":"Discriminating image senses by clustering with multimodal features","authors":"Nicolas Loeff, David Loeff, Forsyth","year":"2006","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Saenko, 2009] Kate Saenko, Trevor Darrell, Unsupervised learning of visual sense models for polysemous words. NIPS, 1393-1400, 2009.","title":"Unsupervised learning of visual sense models for polysemous words","authors":"Kate Saenko, Trevor Darrell","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Barnard, 2005] Kobus Barnard, Matthw Johnson, David Forsyth, \"Word sense disambiguation with pictures,\" AI, 167(2): 13-30, 2005.","title":"Word sense disambiguation with pictures","authors":"Kobus Barnard, Matthw Johnson, David Forsyth","year":"2005","venue":"AI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hua, 2017] Yazhou Yao, Jian Zhang, Fumin Shen, Xian- Sheng Hua, Jingsong Xu and Zhenmin Tang, Exploiting Web Images for Dataset Construction: A Domain Robust Approach. TMM, 19(8): 1771-1784, 2017.","title":"Exploiting Web Images for Dataset Construction: A Domain Robust Approach","authors":"Yazhou Yao, Jian Zhang, Fumin Shen, Xian-Sheng Hua, Jingsong Xu, Zhenmin Tang","year":"2017","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin, 2012] Yuri Lin, Jean Michel, Erez Aiden, Jon Orwant, Will Brockman, Syntactic annotations for the google books ngram corpus. ACL, 169-174, 2012.","title":"Syntactic annotations for the google books ngram corpus","authors":"Yuri Lin, Jean Michel, Erez Aiden, Jon Orwant, Will Brockman","year":"2012","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang, 2014] Xiaogang Wang, Shi Qiu, Ke Liu, Xiaoou Tang, Web image re-ranking usingquery-specific seman- tic signatures. TPAMI, 36(4): 810-823, 2014.","title":"Web image re-ranking usingquery-specific semantic signatures","authors":"Xiaogang Wang, Shi Qiu, Ke Liu, Xiaoou Tang","year":"2014","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou, 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba, Learning deep features for discriminative localization CVPR, 2921-2929, 2016.","title":"Learning deep features for discriminative localization CVPR","authors":"Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rumelhart, 1986] David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, Learning representations by back- propagating errors. Nature, 5(3): 1-12, 1986.","title":"Learning representations by backpropagating errors","authors":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams","year":"1986","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Divvala, 2014] Santosh Kumar Divvala, Ali Farhadi, Carlos Guestrin, \"Learning everything about anything: Webly- supervised visual concept learning,\" CVPR, 3270-3277, 2014.","title":"Learning everything about anything: Weblysupervised visual concept learning","authors":"Santosh Kumar Divvala, Ali Farhadi, Carlos Guestrin","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang, 2016] Yazhou Yao, Xiansheng Hua, Fumin Shen, Jian Zhang and Zhenmin Tang, A domain robust ap- proach for image dataset construction. ACM MM, 212- 216, 2016.","title":"A domain robust approach for image dataset construction","authors":"Yazhou Yao, Xiansheng Hua, Fumin Shen, Jian Zhang, Zhenmin Tang","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao, 2019] Yazhou Yao, Jian Zhang, Fumin Shen, Li Liu, Fan Zhu, and Heng-Tao Shen, Towards Automatic Construction of Diverse, High-quality Image Datasets. TKDE, 2019.","title":"Towards Automatic Construction of Diverse, High-quality Image Datasets","authors":"Yazhou Yao, Jian Zhang, Fumin Shen, Li Liu, Fan Zhu, Heng-Tao Shen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Gella, 2016] Spandana Gella, Mirella Lapata, Frank Keller, \"Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings\" ACL, 1600-1613, 2016.","title":"Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings","authors":"Spandana Gella, Mirella Lapata, Frank Keller","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lucchi, 2012] Aurelien Lucchi, Jason Weston, \"Joint im- age and word sense discrimination for image retrieval\" ECCV, 130-143, 2012.","title":"Joint image and word sense discrimination for image retrieval","authors":"Aurelien Lucchi, Jason Weston","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Golge, 2014] Eren Golge, Pinar Duygulu, \"Concept map: Mining noisy web data for concept learning\" ECCV, 439-455, 2014.","title":"Concept map: Mining noisy web data for concept learning","authors":"Eren Golge, Pinar Duygulu","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Qiu, 2013] Shi Qiu, Xiangang Wang, Xiaoou Tang, \"Vi- sual semantic complex network for web images\" ICCV, 3623-3630, 2013.","title":"Visual semantic complex network for web images","authors":"Xiangang Shi Qiu, Xiaoou Wang, Tang","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wah, 2011] Catherine Wah, Steve Branson, Peter Welin- der, Pietro Perona, Serge Belongie, \"The Caltech-UCSD birds-200-2011 dataset\", Tech Report, 2011.","title":"The Caltech-UCSD birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lin, 2015] Tsung-Yu Lin, Aruni Roychowdhury, Subhransu Maji, \"Bilinear CNN models for fine-grained visual recognition\" ICCV, 1449-1457, 2015.","title":"Bilinear CNN models for fine-grained visual recognition","authors":"Tsung-Yu Lin, Aruni Roychowdhury, Subhransu Maji","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}