{"type":"ScholarlyArticle","title":"Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings","authors":[{"name":"Danushka Bollegala","slug":"danushka-bollegala","given":"Danushka","family":"Bollegala","affiliation":"University of Liverpool , Amazon"}],"abstract":"Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. \r\n\tPrior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. \r\n\tHowever, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. \r\n\tWe show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss.\r\n\tFollowing this theoretical analysis, we propose two \\emph{unsupervised} methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings.\r\n\tExperimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.","doi":"10.24963/ijcai.2022/563","url":"https://ijcai.org/proceedings/2022/563","pdf":"https://ijcai.org/proceedings/2022/0563.pdf","markdown":"https://ijcai.org/proceedings/2022/563.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/563","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":563,"pages":{"first":4058,"last":4064},"keywords":["Machine Learning: Representation learning","Natural Language Processing: Embeddings","Machine Learning: Theory of Deep Learning","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986743","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984886","length_sec":296,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Artetxe et al., 2016] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In EMNLP, pages 2289-2294, 2016.","title":"Learning principled bilingual mappings of word embeddings while preserving monolingual invariance","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao and Bollegala, 2018] Cong Bao and Danushka Bolle- gala. Learning word meta-embeddings by autoencoding. In COLING, pages 1650-1661, 2018.","title":"Learning word meta-embeddings by autoencoding","authors":"Cong Bao, Danushka Bollegala","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bollegala and O'Neill, 2022] Danushka Bollegala and James O'Neill. A survey on word meta-embedding learning. In Proc. of IJCAI, 2022.","title":"Danushka Bollegala and James O'Neill. A survey on word meta-embedding learning","authors":"O' Bollegala, Neill","year":"2022","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bollegala et al., 2017] Danushka Bollegala, Kohei Hayashi, and Ken-ichi Kawarabayashi. Learning linear transforma- tions between counting-based and prediction-based word embeddings. PLoS ONE, 12(9):1-21, September 2017.","title":"Learning linear transformations between counting-based and prediction-based word embeddings","authors":"Danushka Bollegala, Kohei Hayashi, Ken-Ichi Kawarabayashi","year":"2017","venue":"PLoS ONE","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bollegala et al., 2018] Danushka Bollegala, Kohei Hayashi, and Ken-ichi Kawarabayashi. Think globally, embed lo- cally -locally linear meta-embedding of words. In Proc. of IJCAI-EACI, pages 3970-3976, 2018.","title":"Think globally, embed locally -locally linear meta-embedding of words","authors":"Danushka Bollegala, Kohei Hayashi, Ken-Ichi Kawarabayashi","year":"2018","venue":"Proc. of IJCAI-EACI","doi":null,"url":null,"internal_url":"/proceedings/2018/552"},{"position":6,"raw":"[Bollegala, 2022] Danushka Bollegala. Learning meta word embeddings by unsupervised weighted concatenation of source embeddings. 10.48550/ARXIV.2204.12386, 2022.","title":"Learning meta word embeddings by unsupervised weighted concatenation of source embeddings","authors":"Danushka Bollegala","year":"2022","venue":null,"doi":"10.48550/arxiv.2204.12386","url":null,"internal_url":null},{"position":7,"raw":"[Chatterjee, 2012] Sourav Chatterjee. Matrix estimation by Universal Singular Value Thresholding. Annals of Statis- tics, 2012.","title":"Matrix estimation by Universal Singular Value Thresholding","authors":"Sourav Chatterjee","year":"2012","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Coates and Bollegala, 2018] Joshua Coates and Danushka Bollegala. Frustratingly easy meta-embedding -computing meta-embeddings by averaging source word embeddings. In Proc. of NAACL-HLT, pages 194-198, 2018.","title":"Frustratingly easy meta-embedding -computing meta-embeddings by averaging source word embeddings","authors":"Bollegala Coates, Joshua Coates, Danushka Bollegala","year":"2018","venue":"Proc. of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cotterell et al., 2016] Ryan Cotterell, Hinrich Schütze, and Jason Eisner. Morphological smoothing and extrapolation of word embeddings. In Proc. of ACL, 2016.","title":"Morphological smoothing and extrapolation of word embeddings","authors":"Ryan Cotterell, Hinrich Schütze, Jason Eisner","year":"2016","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deerwester et al., 1990] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391- 407, 1990.","title":"Indexing by latent semantic analysis","authors":"Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, Richard Harshman","year":"1990","venue":"Journal of the American Society for Information Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dhillon et al., 2015] Paramveer S. Dhillon, Dean P. Foster, and Lyle H. Ungar. Eigenwords: Spectral word embeddings. JMLR, 16:3035-3078, 2015.","title":"Eigenwords: Spectral word embeddings","authors":"S Paramveer, Dean P Dhillon, Lyle H Foster, Ungar","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Faruqui et al., 2016] Mannal Faruqui, Yulia Tsvetkov, Push- pendre Rastogi, and Chris Dyer. Problems with evaluation of word embeddings using word similarity tasks. In Eval Vec, pages 30-35, 2016.","title":"Problems with evaluation of word embeddings using word similarity tasks","authors":"Mannal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, Chris Dyer","year":"2016","venue":"Eval Vec","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goikoetxea et al., 2016] Josu Goikoetxea, Eneko Agirre, and Aitor Soroa. Single or multiple? combining word representations independently learned from text and word- net. In Proc. of AAAI, pages 2608-2614, 2016.","title":"Single or multiple? combining word representations independently learned from text and wordnet","authors":"Josu Goikoetxea, Eneko Agirre, Aitor Soroa","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hamilton et al., 2016] William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statistical laws of semantic change. In ACL, pages 1489- 1501, 2016.","title":"Diachronic word embeddings reveal statistical laws of semantic change","authors":"William L Hamilton, Jure Leskovec, Dan Jurafsky","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2020] Jingyi He, KC Tsiolis, Kian Kenyon-Dean, and Jackie Chi Kit Cheung. Learning Efficient Task- Specific Meta-Embeddings with Word Prisms. In Proc. of COLING, 2020.","title":"Learning Efficient Task-Specific Meta-Embeddings with Word Prisms","authors":"Jingyi He, Kian Tsiolis, Jackie Kenyon-Dean, Kit Chi, Cheung","year":"2020","venue":"Proc. of COLING","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2012] Eric H. Huang, Richard Socher, Christo- pher D. Manning, and Andrew Y. Ng. Improving word representations via global context and multiple word proto- types. In Proc. of ACL, pages 873-882, 2012.","title":"Improving word representations via global context and multiple word prototypes","authors":"Eric H Huang, Richard Socher, Christopher D Manning, Andrew Y Ng","year":"2012","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jawanpuria et al., 2020] Pratik Jawanpuria, Satya Dev N T V, Anoop Kunchukuttan, and Bamdev Mishra. Learn- ing geometric word meta-embeddings. In Reps4NLP, pages 39-44, Online, 2020.","title":"Learning geometric word meta-embeddings","authors":"Pratik Jawanpuria, Satya Dev, N T V, Anoop Kunchukuttan, Bamdev Mishra","year":"2020","venue":"Reps4NLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NuerIPS, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Levy et al., 2015] Omer Levy, Yoav Goldberg, and Ido Da- gan. Improving distributional similarity with lessons learned from word embeddings. TACL, 3:211-225, 2015.","title":"Improving distributional similarity with lessons learned from word embeddings","authors":"Omer Levy, Yoav Goldberg, Ido Dagan","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2015] Shaohua Li, Jun Zhu, and Chunyan Miao. A generative word embedding model and its low rank positive semidefinite solution. In EMNLP, pages 1599-1609, 2015.","title":"A generative word embedding model and its low rank positive semidefinite solution","authors":"Shaohua Li, Jun Zhu, Chunyan Miao","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, and Jeffrey Dean. Efficient estimation of word representation in vector space. In Proc. of ICLR, 2013.","title":"Efficient estimation of word representation in vector space","authors":"Mikolov","year":"2013","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Neelakantan et al., 2014] Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. Effi- cient non-parametric estimation of multiple embeddings per word in vector space. In Proc. of EMNLP, pages 1059-1069, 2014.","title":"Efficient non-parametric estimation of multiple embeddings per word in vector space","authors":"Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, Andrew Mccallum","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Neill and Bollegala, 2018] James O' Neill and Danushka Bollegala. Angular-Based Word Meta-Embedding Learn- ing. 10.48550/arXiv.1808.04334, 2018.","title":"Angular-Based Word Meta-Embedding Learn","authors":"O' James, Danushka Neill, Bollegala","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ni et al., 2019] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled re- views and fine-grained aspects. In EMNLP, pages 188-197, 2019.","title":"Justifying recommendations using distantly-labeled reviews and fine-grained aspects","authors":"Ni","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pennington et al., 2014] Jeffery Pennington, Richard Socher, and Christopher D. Manning. Glove: global vectors for word representation. In Proc. of EMNLP, pages 1532-1543, 2014.","title":"Glove: global vectors for word representation","authors":"Jeffery Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Reisinger and Mooney, 2010] Joseph Reisinger and Ray- mond J. Mooney. Multi-prototype vector-space models of word meaning. In Proc. of HLT-NAACL, pages 109-117, 2010.","title":"Multi-prototype vector-space models of word meaning","authors":"Joseph Reisinger, Raymond J Mooney","year":"2010","venue":"Proc. of HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Smith et al., 2017] Samuel L. Smith, David H.P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformation and the inverted softmax. In Proc. of ICLR, 2017.","title":"Offline bilingual word vectors, orthogonal transformation and the inverted softmax","authors":"L Samuel, Smith, H P David, Steven Turban, Nils Y Hamblin, Hammerla","year":"2017","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Takahashi and Bollegala, 2022] Keigo Takahashi and Danushka Bollegala. Unsupervised attention-based sentence-level meta-embeddings from contextualised language models. In Proc. of LREC, 2022.","title":"Unsupervised attention-based sentence-level meta-embeddings from contextualised language models","authors":"Keigo Takahashi, Danushka Bollegala","year":"2022","venue":"Proc. of LREC","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vu, 2011] Van Vu. Singular vectors under random pertur- bation. Random Structures & Algorithms, 39(4):526-538, May 2011.","title":"Singular vectors under random perturbation","authors":"Van Vu","year":"2011","venue":"Random Structures & Algorithms","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yin and Schütze, 2016] Wenpeng Yin and Hinrich Schütze. Learning meta-embeddings by using ensembles of embed- ding sets. In Proc. of ACL, pages 1351-1360, 2016.","title":"Learning meta-embeddings by using ensembles of embedding sets","authors":"Schütze Yin, Wenpeng Yin, Hinrich Schütze","year":"2016","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yin and Shen, 2018] Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. In Proc. of NeurIPS, pages 887-898, 2018.","title":"On the dimensionality of word embedding","authors":"Yin, Shen ; ] Zi Yin, Yuanyuan Shen","year":"2018","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}