{"type":"ScholarlyArticle","title":"Predicting Human Similarity Judgments with Distributional Models: The Value of Word Associations","authors":[{"name":"Simon De Deyne","slug":"simon-de-deyne","given":"Simon","family":"De Deyne","affiliation":"Computational Cognitive Science Lab , University of Adelaide"},{"name":"Amy Perfors","slug":"amy-perfors","given":"Amy","family":"Perfors","affiliation":"Computational Cognitive Science Lab , University of Adelaide"},{"name":"Daniel J. Navarro","slug":"daniel-j-navarro","given":"Daniel J.","family":"Navarro","affiliation":"School of Psychology , University of New South Wales"}],"abstract":"To represent the meaning of a word, most models use external language resources, such as text corpora, to derive the distributional properties of word usage. In this study, we propose that internal language models, that are more closely aligned to the mental representations of words, can be used to derive new theoretical questions regarding the structure of the mental lexicon. A comparison with internal models also puts into perspective a number of assumptions underlying recently proposed distributional text-based models could provide important insights into cognitive science, including linguistics and artificial intelligence. We focus on word-embedding models which have been proposed to learn aspects of word meaning in a manner similar to humans and contrast them with internal language models derived from a new extensive data set of word associations. An evaluation using relatedness judgments shows that internal language models consistently outperform current state-of-the art text-based external language models. This suggests alternative approaches to represent word meaning using properties that aren't encoded in text.","doi":"10.24963/ijcai.2017/671","url":"https://ijcai.org/proceedings/2017/671","pdf":"https://ijcai.org/proceedings/2017/0671.pdf","markdown":"https://ijcai.org/proceedings/2017/671.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/671","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":671,"pages":{"first":4806,"last":4810},"keywords":["Artificial Intelligence: knowledge representation and reasoning","Artificial Intelligence: cognitive science","Artificial Intelligence: natural language processing"],"videos":[],"references":[{"position":1,"raw":"[Agirre et al., 2009] Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pas ¸ca, and Aitor Soroa. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Lan- guage Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics, pages 19-27. ACL, 2009.","title":"A study on similarity and relatedness using distributional and wordnet-based approaches","authors":"Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pas, Aitor Soroa","year":"2009","venue":"Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baroni et al., 2014] Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't count, predict! A systematic comparison of context-counting vs. context-predicting se- mantic vectors. In The 52nd Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages 238- 247, 2014.","title":"Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors","authors":"Marco Baroni, Georgiana Dinu, Germán Kruszewski","year":"2014","venue":"The 52nd Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruni et al., 2012] Elia Bruni, Gemma Boleda, Marco Ba- roni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Com- putational Linguistics, 2012.","title":"Distributional semantics in technicolor","authors":"Elia Bruni, Gemma Boleda, Marco Baroni, Nam-Khanh Tran","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Deyne et al., 2013] Simon De Deyne, Daniel J Navarro, and Gert Storms. Better explanations of lexical and se- mantic cognition using networks derived from continued rather than single word associations. Behavior Research Methods, 45:480-498, 2013.","title":"Better explanations of lexical and semantic cognition using networks derived from continued rather than single word associations","authors":"Simon De Deyne, Daniel J Navarro, Gert Storms","year":"2013","venue":"Behavior Research Methods","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Deyne et al., 2016a] Simon De Deyne, Daniel J Navarro, Amy Perfors, and Gert Storms. Structure at every scale: A semantic network account of the similarities between unrelated concepts. Journal of Experimental Psychology: General, 145:1228-1254, 2016.","title":"Structure at every scale: A semantic network account of the similarities between unrelated concepts","authors":"De Deyne","year":"2016","venue":"Journal of Experimental Psychology: General","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Deyne et al., 2016b] Simon De Deyne, Amy Perfors, and Daniel J. Navarro. Predicting human similarity judg- ments with distributional models: The value of word asso- ciations. In Proceedings of the 26th International Confer- ence on Computational Linguistics. ACL, 2016.","title":"Predicting human similarity judgments with distributional models: The value of word associations","authors":"De Deyne","year":"2016","venue":"Proceedings of the 26th International Conference on Computational Linguistics. ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deese, 1965] James Deese. The structure of associations in language and thought. Johns Hopkins Press, 1965.","title":"The structure of associations in language and thought","authors":"James Deese, Deese","year":"1965","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Griffiths et al., 2007] Thomas L Griffiths, Mark Steyvers, and Joshua B Tenenbaum. Topics in semantic represen- tation. Psychological review, 114(2):211, 2007.","title":"Topics in semantic representation","authors":"Mark Thomas L Griffiths, Joshua B Steyvers, Tenenbaum","year":"2007","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Halawi et al., 2012] Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 1406-1414. ACM, 2012.","title":"Large-scale learning of word relatedness with constraints","authors":"Halawi","year":"2012","venue":"Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hill et al., 2016] Felix Hill, Roi Reichart, and Anna Korho- nen. Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41:665-695, 2016.","title":"Simlex-999: Evaluating semantic models with (genuine) similarity estimation","authors":"Felix Hill, Roi Reichart, Anna Korhonen","year":"2016","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Landauer and Dumais, 1997] Tom K. Landauer and Su- san T. Dumais. A solution to Plato's Problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 104:211-240, 1997.","title":"A solution to Plato's Problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge","authors":"Tom K Landauer, Susan T Dumais","year":"1997","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lazaridou et al., 2015] Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. Combining language and vi- sion with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598, 2015.","title":"Combining language and vision with a multimodal skip-gram model","authors":"Lazaridou","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems, pages 2177-2185, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levy et al., 2015] Omer Levy, Yoav Goldberg, and Ido Da- gan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Asso- ciation for Computational Linguistics, 3:211-225, 2015.","title":"Improving distributional similarity with lessons learned from word embeddings","authors":"Omer Levy, Yoav Goldberg, Ido Dagan","year":"2015","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mandera et al., 2017] Paweł Mandera, Emmanuel Keuleers, and Marc Brysbaert. Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation. Journal of Memory and Language, 92:57-78, 2017.","title":"Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation","authors":"Paweł Mandera, Emmanuel Keuleers, Marc Brysbaert","year":"2017","venue":"Journal of Memory and Language","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Michel et al., 2011] Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. Quantitative analysis of culture using millions of digitized books. Science, 331:176-182, 2011.","title":"Quantitative analysis of culture using millions of digitized books","authors":"Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, K Matthew, Joseph P Gray, Dale Pickett, Dan Hoiberg, Peter Clancy, Jon Norvig, Orwant","year":"2011","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Newman, 2010] Mark E. J. Newman. Networks: An Intro- duction. Oxford University Press, 2010.","title":"Networks: An Introduction","authors":"E J Mark, Newman","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radinsky et al., 2011] Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal seman- tic analysis. In Proceedings of the 20th international con- ference on World wide web, pages 337-346. ACM, 2011.","title":"A word at a time: computing word relatedness using temporal semantic analysis","authors":"Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, Shaul Markovitch","year":"2011","venue":"Proceedings of the 20th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rubenstein and Goodenough, 1965] Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8:627-633, 1965.","title":"Contextual correlates of synonymy","authors":"Herbert Rubenstein, John B Goodenough","year":"1965","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simmons et al., 2008] William K. Simmons, Stephan B. Hamann, Carla N. Harenski, Xiaoping P. Hu, and Lawrence W. Barsalou. fMRI evidence for word asso- ciation and situated simulation in conceptual processing. Journal of Physiology -Paris, 102:106-119, 2008.","title":"fMRI evidence for word association and situated simulation in conceptual processing","authors":"K William, Stephan B Simmons, Carla N Hamann, Harenski, P Xiaoping, Lawrence W Hu, Barsalou","year":"2008","venue":"Journal of Physiology -Paris","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Taylor, 2012] John R. Taylor. The mental corpus: How lan- guage is represented in the mind. Oxford University Press, 2012.","title":"The mental corpus: How language is represented in the mind","authors":"John R Taylor","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}