{"type":"ScholarlyArticle","title":"Leveraging the Wikipedia Graph for Evaluating Word Embeddings","authors":[{"name":"Joachim Giesen","slug":"joachim-giesen","given":"Joachim","family":"Giesen","affiliation":"Friedrich Schiller University Jena"},{"name":"Paul Kahlmeyer","slug":"paul-kahlmeyer","given":"Paul","family":"Kahlmeyer","affiliation":"DLR Institute of Data Science"},{"name":"Frank Nussbaum","slug":"frank-nussbaum","given":"Frank","family":"Nussbaum","affiliation":"Friedrich Schiller University Jena; DLR Institute of Data Science"},{"name":"Sina Zarrieß","slug":"sina-zarrie","given":"Sina","family":"Zarrieß","affiliation":"Bielefeld University"}],"abstract":"Deep learning models for different NLP tasks often rely on pre-trained word embeddings, that is, vector representations of words. Therefore, it is crucial to evaluate pre-trained word embeddings independently of downstream tasks. Such evaluations try to assess whether the geometry induced by a word embedding captures connections made in natural language, such as, analogies, clustering of words, or word similarities. Here, traditionally, similarity is measured by comparison to human judgment. However, explicitly annotating word pairs with similarity scores by surveying humans is expensive. We tackle this problem by formulating a similarity measure that is based on an agent for routing the Wikipedia hyperlink graph. In this graph, word similarities are implicitly encoded by edges between articles. We show on the English Wikipedia that our measure correlates well with a large group of traditional similarity measures, while covering a much larger proportion of words and avoiding explicit human labeling. Moreover, since Wikipedia is available in more than 300 languages, our measure can easily be adapted to other languages, in contrast to traditional similarity measures.","doi":"10.24963/ijcai.2022/574","url":"https://ijcai.org/proceedings/2022/574","pdf":"https://ijcai.org/proceedings/2022/0574.pdf","markdown":"https://ijcai.org/proceedings/2022/574.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/574","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":574,"pages":{"first":4136,"last":4142},"keywords":["Search: Local search","Natural Language Processing: Embeddings","Natural Language Processing: Resources and Evaluation","Agent-based and Multi-agent Systems: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986837","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984980","length_sec":353,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agirre et al., 2009] Eneko Agirre, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. A study on similarity and relatedness using distri- butional and wordnet-based approaches. In Proceedings of the Conference on Human Language Technologies (NAACL-HLT), pages 19-27, 2009.","title":"A study on similarity and relatedness using distributional and wordnet-based approaches","authors":"Eneko Agirre, Enrique Alfonseca, Keith B Hall, Jana Kravalova, Marius Pasca, Aitor Soroa","year":"2009","venue":"Proceedings of the Conference on Human Language Technologies (NAACL-HLT)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bakarov, 2018] Amir Bakarov. A Survey of Word Embed- dings Evaluation Methods. CoRR, abs/1801.09536, 2018.","title":"A Survey of Word Embeddings Evaluation Methods","authors":"Amir Bakarov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baroni et al., 2014] Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't count, predict! A system- atic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 238-247, 2014.","title":"Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors","authors":"Marco Baroni, Georgiana Dinu, Germán Kruszewski","year":"2014","venue":"Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boldi et al., 2011] Paolo Boldi, Marco Rosa, Massimo San- tini, and Sebastiano Vigna. Layered label propagation: a multiresolution coordinate-free ordering for compressing social networks. In Proceedings of the International Con- ference on World Wide Web (WWW), pages 587-596, 2011.","title":"Layered label propagation: a multiresolution coordinate-free ordering for compressing social networks","authors":"Paolo Boldi, Marco Rosa, Massimo Santini, Sebastiano Vigna","year":"2011","venue":"Proceedings of the International Conference on World Wide Web (WWW)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chi et al., 2001] Ed Huai-hsin Chi, Peter Pirolli, Kim Chen, and James E. Pitkow. Using information scent to model user information needs and actions and the web. In Pro- ceedings of the Conference on Human Factors in Comput- ing Systems (CHI), pages 490-497, 2001.","title":"Using information scent to model user information needs and actions and the web","authors":"Chi","year":"2001","venue":"Proceedings of the Conference on Human Factors in Computing Systems (CHI)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clemesha, 2018] Alex Clemesha. The Wiki Game. https: //www.thewikigame.com, 2018. Accessed: 2022-01-13.","title":"The Wiki Game","authors":"Alex Clemesha","year":"2018","venue":null,"doi":null,"url":"https://www.thewikigame.com","internal_url":null},{"position":7,"raw":"[Faruqui et al., 2016] Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. Problems With Evaluation of Word Embeddings Using Word Similarity Tasks. In Proceedings of the Workshop on Evaluating Vector-Space Representation for NLP (RepEval), pages 30-35, 2016.","title":"Problems With Evaluation of Word Embeddings Using Word Similarity Tasks","authors":"Faruqui","year":"2016","venue":"Proceedings of the Workshop on Evaluating Vector-Space Representation for NLP (RepEval)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hoenen, 2016] Armin Hoenen. Wikipedia Titles As Noun Tag Predictors. In Proceedings of the International Con- ference on Language Resources and Evaluation (LREC), 2016.","title":"Wikipedia Titles As Noun Tag Predictors","authors":"Armin Hoenen","year":"2016","venue":"Proceedings of the International Conference on Language Resources and Evaluation (LREC)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim, 2014] Yoon Kim. Convolutional Neural Networks for Sentence Classification. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1746-1751, 2014.","title":"Convolutional Neural Networks for Sentence Classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lastra-Díaz et al., 2019] Juan J. Lastra-Díaz, Josu Goikoetxea, Mohamed Ali Hadj Taieb, Ana García- Serrano, Mohamed Ben Aouicha, and Eneko Agirre. A reproducible survey on word embeddings and ontology- based methods for word similarity: Linear combinations outperform the state of the art. Engineering Applications of Artificial Intelligence, 85:645-665, 2019.","title":"A reproducible survey on word embeddings and ontologybased methods for word similarity: Linear combinations outperform the state of the art","authors":"J Juan, Josu Lastra-Díaz, Mohamed Goikoetxea, Ali Hadj, Ana Taieb, Mohamed García-Serrano, Eneko Ben Aouicha, Agirre","year":"2019","venue":"Engineering Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liza and Grzes, 2016] Farhana Ferdousi Liza and Marek Grzes. An Improved Crowdsourcing Based Evaluation Technique for Word Embedding Methods. In Proceedings of the 1st Workshop on Evaluating Vector-Space Represen- tations for NLP (RepEval@ACL), pages 55-61, 2016.","title":"An Improved Crowdsourcing Based Evaluation Technique for Word Embedding Methods","authors":"Farhana Ferdousi, Liza, Marek Grzes","year":"2016","venue":"Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP (RepEval@ACL)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mandelbaum and Shalev, 2016] Amit Mandelbaum and Adi Shalev. Word embeddings and their use in sentence clas- sification tasks. CoRR, abs/1610.08229, 2016.","title":"Word embeddings and their use in sentence classification tasks","authors":"Shalev Mandelbaum, Amit Mandelbaum, Adi Shalev","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomás Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient Estimation of Word Rep- resentations in Vector Space. In Proceedings of the Inter- national Conference on Learning Representations (ICLR) Workshop Track, 2013.","title":"Efficient Estimation of Word Representations in Vector Space","authors":"Tomás Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"Proceedings of the International Conference on Learning Representations (ICLR) Workshop Track","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2018] Tomás Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Ad- vances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Lan- guage Resources and Evaluation (LREC), 2018.","title":"Advances in Pre-Training Distributed Word Representations","authors":"Tomás Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin","year":"2018","venue":"Proceedings of the International Conference on Language Resources and Evaluation (LREC)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Murphy, 1986] M. Lynne Murphy. Semantic Relations and the Lexicon. Cambridge University Press, 1986.","title":"Semantic Relations and the Lexicon","authors":"M, Lynne Murphy","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of the Conference on Hu- man Language Technologies (NAACL-HLT), pages 2227- 2237, 2018.","title":"Deep Contextualized Word Representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"Proceedings of the Conference on Human Language Technologies (NAACL-HLT)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2018] Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? In Proceedings of the Con- ference on Human Language Technologies (NAACL-HLT), pages 529-535, 2018.","title":"When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation","authors":"Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, Graham Neubig","year":"2018","venue":"Proceedings of the Conference on Human Language Technologies (NAACL-HLT)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rada et al., 1989] Roy Rada, Hafedh Mili, Ellen Bicknell, and Maria Blettner. Development and application of a met- ric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics, 19(1):17-30, 1989.","title":"Development and application of a metric on semantic nets","authors":"Roy Rada, Hafedh Mili, Ellen Bicknell, Maria Blettner","year":"1989","venue":"IEEE Transactions on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schnabel et al., 2015] Tobias Schnabel, Igor Labutov, David M. Mimno, and Thorsten Joachims. Evaluation methods for unsupervised word embeddings. In Proceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 298-307, 2015.","title":"Evaluation methods for unsupervised word embeddings","authors":"Tobias Schnabel, Igor Labutov, David M Mimno, Thorsten Joachims","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sezerer and Tekir, 2021] Erhan Sezerer and Selma Tekir. A Survey On Neural Word Embeddings. CoRR, abs/2110.01804, 2021.","title":"A Survey On Neural Word Embeddings","authors":"Erhan Sezerer, Selma Tekir","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Torregrossa et al., 2020] Franc ¸ois Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Allesiardo. On the Correlation of Word Embedding Evaluation Metrics. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), pages 4789-4797, 2020.","title":"On the Correlation of Word Embedding Evaluation Metrics","authors":"Torregrossa","year":"2020","venue":"Proceedings of the International Conference on Language Resources and Evaluation (LREC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wenger, 2018] Jacob Wenger. Six degrees of wikipedia. https://www.sixdegreesofwikipedia.com/, 2018. Ac- cessed: 2022-01-13.","title":"Jacob Wenger. Six degrees of wikipedia","authors":"Wenger","year":"2018","venue":null,"doi":null,"url":"https://www.sixdegreesofwikipedia.com/,2018.Ac-cessed:2022-01-13","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}