{"type":"ScholarlyArticle","title":"Multi-Modal Word Synset Induction","authors":[{"name":"Jesse Thomason","slug":"jesse-thomason","given":"Jesse","family":"Thomason","affiliation":"Department of Computer Science , University of Texas at Austin Austin , TX 78712 , USA"},{"name":"Raymond J. Mooney","slug":"raymond-j-mooney","given":"Raymond J.","family":"Mooney","affiliation":"Department of Computer Science , University of Texas at Austin Austin , TX 78712 , USA"}],"abstract":"A word in natural language can be polysemous, having multiple meanings, as well as synonymous, meaning the same thing as other words. Word sense induction attempts to find the senses of polysemous words. Synonymy detection attempts to find when two words are interchangeable. We combine these tasks, first inducing word senses and then detecting similar senses to form word-sense synonym sets (synsets) in an unsupervised fashion. Given pairs of images and text with noun phrase labels, we perform synset induction to produce collections of underlying concepts described by one or more noun phrases. We find that considering multi-modal features from both visual and textual context yields better induced synsets than using either context alone. Human evaluations show that our unsupervised, multi-modally induced synsets are comparable in quality to annotation-assisted ImageNet synsets, achieving about 84% of ImageNet synsets' approval.","doi":"10.24963/ijcai.2017/575","url":"https://ijcai.org/proceedings/2017/575","pdf":"https://ijcai.org/proceedings/2017/0575.pdf","markdown":"https://ijcai.org/proceedings/2017/575.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/575","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":575,"pages":{"first":4116,"last":4122},"keywords":["Natural Language Processing: NLP Applications and Tools","Robotics and Vision: Vision and Perception","Natural Language Processing: Natural Language Semantics"],"videos":[],"references":[{"position":1,"raw":"[Barnard and Johnson, 2005] Kobus Barnard and Matthew Johnson. Word sense disambiguation with pictures. Jour- nal of Artificial Intelligence, 167:13-30, 2005.","title":"Word sense disambiguation with pictures","authors":"Johnson ; Barnard, Kobus Barnard, Matthew Johnson","year":"2005","venue":"Journal of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordag, 2006] Stefan Bordag. Word sense induction: Triplet-based clusering and automatic evaluation. In Pro- ceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 137-144, 2006.","title":"Word sense induction: Triplet-based clusering and automatic evaluation","authors":"Stefan Bordag, Bordag","year":"2006","venue":"Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruni et al., 2014] Elia Bruni, Nam Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1-47, 2014.","title":"Multimodal distributional semantics","authors":"Elia Bruni, Nam Khanh Tran, Marco Baroni","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Xinlei Chen, Alan Ritter, Abhinav Gupta, and Tom Mitchell. Sense Discovery via Co- Clustering on Images and Text. In Computer Vision and Pattern Recognition, 2015.","title":"Sense Discovery via Co-Clustering on Images and Text","authors":"Chen","year":"2015","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deerwester et al., 1990] Scott Deerwester, Susan T. Du- mais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391-407, 1990.","title":"Indexing by latent semantic analysis","authors":"Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, Richard Harshman","year":"1990","venue":"Journal of the American Society for Information Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pat- tern Recognition, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deselaers and Ferrari, 2011] Thomas Deselaers and Vitto- rio Ferrari. Visual and semantic similarity in imagenet. In Computer Vision and Pattern Recognition, pages 1777- 178, June 2011.","title":"Thomas Deselaers and Vittorio Ferrari. Visual and semantic similarity in imagenet","authors":"Ferrari Deselaers","year":"2011","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Di Marco and Navigli, 2013] Antonio Di Marco and Roberto Navigli. Clustering and diversifying web search results with graph-based word sense induction. Computational Linguistics, 39(3):709-754, 2013.","title":"Antonio Di Marco and Roberto Navigli. Clustering and diversifying web search results with graph-based word sense induction","authors":"Di Marco, Navigli","year":"2013","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fellbaum, 1998] Christiane D. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, 1998.","title":"WordNet: An Electronic Lexical Database","authors":"Christiane D Fellbaum","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kanishcheva and Angelova, 2016] Olga Kanishcheva and Galia Angelova. About Sense Disambiguation of Image Tags in Large Annotated Image Collections, pages 133- 149. Springer International Publishing, 2016.","title":"About Sense Disambiguation of Image Tags in Large Annotated Image Collections","authors":"Angelova Kanishcheva, Olga Kanishcheva, Galia Angelova","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiela et al., 2015] Douwe Kiela, Laura Rimell, Ivan Vulić, and Stephen Clark. Exploiting image generality for lexical entailment detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguis- tics and the 7th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 119-124, Beijing, China, July 2015.","title":"Exploiting image generality for lexical entailment detection","authors":"Douwe Kiela, Laura Rimell, Ivan Vulić, Stephen Clark","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loeff et al., 2006] Nicolas Loeff, Cecilia Ovesdotter Alm, and David A. Forsyth. Discriminating image senses by clustering with multimodal features. In Proceedings of the Conference on Computational Linguistics, pages 547-554, Stroudsburg, PA, USA, 2006.","title":"Discriminating image senses by clustering with multimodal features","authors":"Loeff","year":"2006","venue":"Proceedings of the Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lucchi and Weston, 2012] Aurelien Lucchi and Jason We- ston. Joint image and word sense discrimination for image retrieval. In European Conference on Computer Vision, pages 130-143. Springer, 2012.","title":"Joint image and word sense discrimination for image retrieval","authors":"Weston ; Lucchi, Aurelien Lucchi, Jason Weston","year":"2012","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Manandhar et al., 2010] Suresh Manandhar, Ioannis P. Kla- paftis, Dmitriy Dligach, and Sameer S. Pradhan. Semeval- 2010 task 14: Word sense induction & disambiguation. In Proceedings of the 5th International Workshop on Seman- tic Evaluation, SemEval '10, pages 63-68, Stroudsburg, PA, USA, 2010.","title":"Semeval-2010 task 14: Word sense induction & disambiguation","authors":"Suresh Manandhar, P Ioannis, Dmitriy Klapaftis, Sameer S Dligach, Pradhan","year":"2010","venue":"Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval '10","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McCarthy and Navigli, 2007] Diana McCarthy and Roberto Navigli. Semeval-2007 task 10: English lexical substitu- tion task. In Proceedings of the 4th International Work- shop on Semantic Evaluations, pages 48-53. Association for Computational Linguistics, 2007.","title":"Semeval-2007 task 10: English lexical substitution task","authors":"Diana Mccarthy, Roberto Navigli","year":"2007","venue":"Proceedings of the 4th International Workshop on Semantic Evaluations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed rep- resentations of words and phrases and their composition- ality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, pages 3111- 3119, Lake Tahoe, Nevada, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proceedings of the 26th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Navigli and Vannella, 2013] Roberto Navigli and Daniele Vannella. Semeval-2013 task 11: Word sense induction and disambiguation within an end-user application. In In Second Joint Conference on Lexical and Computational Semantics, 2013.","title":"Semeval-2013 task 11: Word sense induction and disambiguation within an end-user application","authors":"Roberto Navigli, Daniele Vannella","year":"2013","venue":"Second Joint Conference on Lexical and Computational Semantics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parde et al., 2015] Natalie Parde, Adam Hair, Michalis Pa- pakostas, Konstantinos Tsiakas, Maria Dagioglou, Vange- lis Karkaletsis, and Rodney D. Nielsen. Grounding the meaning of words through vision and interactive game- play. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, pages 1895-1901, Buenos Aires, Argentina, 2015.","title":"Grounding the meaning of words through vision and interactive gameplay","authors":"Natalie Parde, Adam Hair, Michalis Papakostas, Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, Rodney D Nielsen","year":"2015","venue":"Proceedings of the 24th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/269"},{"position":19,"raw":"[Pedersen and Bruce, 1997] Ted Pedersen and Rebecca Bruce. Distinguishing word senses in untagged text. In Proceedings of the Second Conference on Empirical Methods in Natural Language, pages 197-207, 1997.","title":"Distinguishing word senses in untagged text","authors":"Bruce ; Pedersen, Ted Pedersen, Rebecca Bruce","year":"1997","venue":"Proceedings of the Second Conference on Empirical Methods in Natural Language","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Perera and Allen, 2013] Ian Perera and James F. Allen. Sall- e: Situated agent for language learning. In Proceedings of the 27th AAAI Conference on Artificial Intelligence, pages 1241-1247, Bellevue, Washington, USA, 2013.","title":"Salle: Situated agent for language learning","authors":"Ian Perera, James F Allen","year":"2013","venue":"Proceedings of the 27th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reisinger and Mooney, 2010] Joseph Reisinger and Ray- mond J. Mooney. Multi-prototype vector-space models of word meaning. In Proceedings of Human Language Tech- nologies: The Conference of the North American Chap- ter of the Association for Computational Linguistics, pages 109-117, 2010.","title":"Multi-prototype vector-space models of word meaning","authors":"Joseph Reisinger, Raymond J Mooney","year":"2010","venue":"Proceedings of Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rosenberg and Hirschberg, 2007] Andrew Rosenberg and Julia Hirschberg. V-Measure: A Conditional Entropy- Based External Cluster Evaluation Measure. In Proceed- ings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 410-420, 2007.","title":"V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure","authors":"Andrew Rosenberg, Julia Hirschberg","year":"2007","venue":"Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saenko and Darrell, 2008] Kate Saenko and Trevor Darrell. Unsupervised learning of visual sense models for polyse- mous words. In Advances in Neural Information Process- ing Systems 21, pages 1393-1400. 2008.","title":"Unsupervised learning of visual sense models for polysemous words","authors":"Darrell ; Saenko, Kate Saenko, Trevor Darrell","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schutze, 1998] Hinrich Schutze. Automatic word sense dis- crimination. Computational Linguistics, 24(1):97-123, 1998.","title":"Automatic word sense discrimination","authors":"Hinrich Schutze, Schutze","year":"1998","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. Computing Research Repository, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"Computing Research Repository","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thomason et al., 2016] Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond Mooney. Learning multi-modal grounded linguistic semantics by playing \"I spy\". In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 3477- 3483, July 2016.","title":"Learning multi-modal grounded linguistic semantics by playing \"I spy","authors":"Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, Raymond Mooney","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/491"},{"position":27,"raw":"[Tibshirani et al., 2001] Robert Tibshirani, Guenther Walther, and Trevor Hastie. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2):411-423, 2001.","title":"Estimating the number of clusters in a data set via the gap statistic","authors":"Robert Tibshirani, Walther Guenther, Trevor Hastie","year":"2001","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}