{"type":"ScholarlyArticle","title":"A Unified Framework for Intrinsic Evaluation of Word-Embedding Algorithms","authors":[{"name":"Anna Giabelli","slug":"anna-giabelli","given":"Anna","family":"Giabelli","affiliation":"Università degli studi di Milano -Bicocca , Italy"}],"abstract":"Word embeddings are widely used in copious Natural Language Processing tasks, including semantic analysis, information retrieval, dependency parsing, question answering, and machine translation. This extensive use implies that the evaluation of the performance of such representations is crucial for choosing the best model to perform those tasks.\r\n\r\nThough there are well-established procedures and benchmarks for intrinsic evaluation, as far as we know, a unified method of evaluation that can merge the results of those tasks to provide a comprehensive evaluation is missing. The main goal of this work is to create a pipeline to blend all major intrinsic evaluation tasks to compute such overall evaluation - the PCE - of word embeddings.","doi":"10.24963/ijcai.2022/829","url":"https://ijcai.org/proceedings/2022/829","pdf":"https://ijcai.org/proceedings/2022/0829.pdf","markdown":"https://ijcai.org/proceedings/2022/829.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/829","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":829,"pages":{"first":5853,"last":5854},"keywords":["Speech & Natural Language Processing (SNLP): General"],"videos":[],"references":[{"position":1,"raw":"[Bakarov, 2018] A. Bakarov. A survey of word embeddings evaluation methods. arXiv:1801.09536, 2018.","title":"A survey of word embeddings evaluation methods","authors":"A Bakarov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baroni et al., 2014] M. Baroni, G. Dinu, and G. Kruszewski. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, 2014.","title":"Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors","authors":"M Baroni, G Dinu, G Kruszewski","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Giabelli et al., 2022] A. Giabelli, L. Malandri, F. Mercorio, M. Mezzanzanica, and N. Nobani. Embeddings evaluation using a novel measure of semantic similarity. Cognitive Computation, 2022.","title":"Embeddings evaluation using a novel measure of semantic similarity","authors":"A Giabelli, L Malandri, F Mercorio, M Mezzanzanica, N Nobani","year":"2022","venue":"Cognitive Computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Levy et al., 2015] O. Levy, Y. Goldberg, and I. Dagan. Im- proving distributional similarity with lessons learned from word embeddings. TACL, 3, 2015.","title":"Improving distributional similarity with lessons learned from word embeddings","authors":"O Levy, Y Goldberg, I Dagan","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Malandri et al., 2020] L. Malandri, F. Mercorio, M. Mez- zanzanica, and N. Nobani. Meet: A method for embed- dings evaluation for taxonomic data. In 2020 ICDMW. IEEE, 2020.","title":"Meet: A method for embeddings evaluation for taxonomic data","authors":"L Malandri, F Mercorio, M Mezzanzanica, N Nobani","year":"2020","venue":"ICDMW. IEEE","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Mikolov et al., 2013] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"T Mikolov, I Sutskever, K Chen, G S Corrado, J Dean","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Roy and Pan, 2021] A. Roy and S. Pan. Incorporating extra knowledge to enhance word embedding. In IJCAI, 2021.","title":"Incorporating extra knowledge to enhance word embedding","authors":"A Roy, S Pan","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/686"},{"position":8,"raw":"[Schnabel et al., 2015] T. Schnabel, I. Labutov, D. Mimno, and T. Joachims. Evaluation methods for unsupervised word embeddings. In EMNLP, 2015.","title":"Evaluation methods for unsupervised word embeddings","authors":"T Schnabel, I Labutov, D Mimno, T Joachims","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Wang et al., 2019] B. Wang, A. Wang, F. Chen, Y. Wang, and C. C. J. Kuo. Evaluating word embedding models: Methods and experimental results. APSIPA, 8, 2019.","title":"Evaluating word embedding models: Methods and experimental results","authors":"B Wang, A Wang, F Chen, Y Wang, C C J Kuo","year":"2019","venue":"APSIPA","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}