{"type":"ScholarlyArticle","title":"Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words","authors":[{"name":"Danushka Bollegala","slug":"danushka-bollegala","given":"Danushka","family":"Bollegala","affiliation":"* University of Liverpool , Liverpool , United Kingdom"},{"name":"Kohei Hayashi","slug":"kohei-hayashi","given":"Kohei","family":"Hayashi","affiliation":"† National Institute of Advanced Industrial Science and Technology , Tokyo , Japan"},{"name":"Ken-ichi Kawarabayashi","slug":"ken-ichi-kawarabayashi","given":"Ken-ichi","family":"Kawarabayashi","affiliation":"‡ National Institute of Informatics , Tokyo , Japan Japan Science and Technology Agency ,; ERATO , Kawarabayashi Large Graph"}],"abstract":"Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks.\r\nHowever, their performances vary significantly across different tasks,\r\nimplying that the word embeddings learnt by those methods capture complementary aspects of lexical semantics.\r\nTherefore, we believe that it is important to combine the existing word embeddings to produce more accurate and complete\r\nmeta-embeddings of words.\r\nFor this purpose, we propose an unsupervised locally linear meta-embedding learning method that takes \r\npre-trained word embeddings as the input, and produces more accurate meta embeddings.\r\nUnlike previously proposed meta-embedding learning methods that learn a global projection over all words in a vocabulary,\r\nour proposed method is sensitive to the differences in local neighbourhoods of the individual source word embeddings. \r\nMoreover, we show that vector concatenation, a previously proposed highly competitive baseline\r\napproach for integrating word embeddings, can be derived as a special case of the proposed method.\r\nExperimental results on semantic similarity, word analogy, relation classification, and short-text classification tasks show that our\r\nmeta-embeddings to significantly outperform prior methods in several benchmark datasets,\r\nestablishing a new state of the art for meta-embeddings.","doi":"10.24963/ijcai.2018/552","url":"https://ijcai.org/proceedings/2018/552","pdf":"https://ijcai.org/proceedings/2018/0552.pdf","markdown":"https://ijcai.org/proceedings/2018/552.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/552","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":552,"pages":{"first":3970,"last":3976},"keywords":["Natural Language Processing: Natural Language Semantics","Machine Learning: Deep Learning","Machine Learning Applications: Applications of Unsupervised Learning","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bansal et al., 2014] Mohit Bansal, Kevin Gimpel, and Karen Livescu. Tailoring continouos word representations for dependency parsing. In Proc. of ACL, 2014.","title":"Tailoring continouos word representations for dependency parsing","authors":"Mohit Bansal, Kevin Gimpel, Karen Livescu","year":"2014","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bruni et al., 2012] Elia Bruni, Gemma Boleda, Marco Ba- roni, and Nam Khanh Tran. Distributional semantics in technicolor. In Proc. of ACL, pages 136-145, 2012.","title":"Distributional semantics in technicolor","authors":"Elia Bruni, Gemma Boleda, Marco Baroni, Nam Khanh, Tran","year":"2012","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2013] Yanqing Chen, Bryan Perozzi, Rami Al- Rfou, and Steven Skiena. The expressive power of word embeddings. In Proc. of ICML Workshop, 2013.","title":"The expressive power of word embeddings","authors":"Yanqing Chen, Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2013","venue":"Proc. of ICML Workshop","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In Proc. of ICML, pages 160 -167, 2008.","title":"A unified architecture for natural language processing: Deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Firth, 1957] John R. Firth. A synopsis of linguistic theory 1930-55. Studies in Linguistic Analysis, pages 1 -32, 1957.","title":"A synopsis of linguistic theory 1930-55","authors":"John R Firth","year":"1957","venue":"Studies in Linguistic Analysis","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goikoetxea et al., 2016] Josu Goikoetxea, Eneko Agirre, and Aitor Soroa. Single or multiple? combining word rep- resentations independently learned from text and wordnet. In Proc. of AAAI, pages 2608-2614, 2016.","title":"Single or multiple? combining word representations independently learned from text and wordnet","authors":"Josu Goikoetxea, Eneko Agirre, Aitor Soroa","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hill et al., 2014] Felix Hill, Kyunghyun Cho, Sébastian Jean, Coline Devin, and Yoshua Bengio. Not all neural embeddings are born equal. In NIPS workshop, 2014.","title":"Not all neural embeddings are born equal","authors":"Felix Hill, Kyunghyun Cho, Sébastian Jean, Coline Devin, Yoshua Bengio","year":"2014","venue":"NIPS workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hill et al., 2015a] Felix Hill, Kyunghyun Cho, Sébastian Jean, Coline Devin, and Yoshua Bengio. Embedding word similarity with neural machine translation. In ICLR Work- shop, 2015.","title":"Embedding word similarity with neural machine translation","authors":"Hill","year":"2015","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hill et al., 2015b] Felix Hill, Roi Reichart, and Anna Ko- rhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguis- tics, 41(4):665-695, 2015.","title":"Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation","authors":"Hill","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2012] Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. Improving word representations via global context and multiple word prototypes. In Proc. of ACL, pages 873-882, 2012.","title":"Improving word representations via global context and multiple word prototypes","authors":"Eric H Huang, Richard Socher, Christopher D Manning, Andrew Y Ng","year":"2012","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jurgens et al., 2012] David A. Jurgens, Saif Mohammad, Peter D. Turney, and Keith J. Holyoak. Measuring degrees of relational similarity. In Proc. of SemEval, 2012.","title":"Measuring degrees of relational similarity","authors":"David A Jurgens, Saif Mohammad, Peter D Turney, Keith J Holyoak","year":"2012","venue":"Proc. of SemEval","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kibriya and Frank, 2007] Ashraf M. Kibriya and Eibe Frank. An empirical comparison of exact nearest neigh- bout algorithms. In Proc. of PKDD, pages 140-151, 2007.","title":"An empirical comparison of exact nearest neighbout algorithms","authors":"M Ashraf, Eibe Kibriya, Frank","year":"2007","venue":"Proc. of PKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2014] Yong Luo, Jian Tang, Jun Yan, Chao Xu, and Zheng Chen. Pre-trained multi-view word embedding using two-side neural network. In Proc. of AAAI, pages 1982-1988, 2014.","title":"Pre-trained multi-view word embedding using two-side neural network","authors":"Yong Luo, Jian Tang, Jun Yan, Chao Xu, Zheng Chen","year":"2014","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luong et al., 2013] Minh-Thang Luong, Richard Socher, and Christopher D. Manning. Better word representations with recursive neural networks for morphology. In CoNLL, 2013.","title":"Better word representations with recursive neural networks for morphology","authors":"Minh-Thang Luong, Richard Socher, Christopher D Manning","year":"2013","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, and Jef- frey Dean. Efficient estimation of word representation in vector space. In Proc. of ICLR, 2013. [Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their composi- tionality. In Proc. of NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miller and Charles, 1998] G. Miller and W. Charles. Con- textual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1-28, 1998.","title":"Contextual correlates of semantic similarity","authors":"G Miller, W Charles","year":"1998","venue":"Language and Cognitive Processes","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih and Hinton, 2009] Andriy Mnih and Geoffrey E. Hin- ton. A scalable hierarchical distributed language model. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Proc. of NIPS, pages 1081-1088. 2009.","title":"A scalable hierarchical distributed language model","authors":"Andriy Mnih, Geoffrey E Hinton","year":"2009","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffery Pennington, Richard Socher, and Christopher D. Manning. Glove: global vectors for word representation. In Proc. of EMNLP, pages 1532-1543, 2014.","title":"Glove: global vectors for word representation","authors":"Jeffery Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rubenstein and Goodenough, 1965] H. Rubenstein and J.B. Goodenough. Contextual correlates of synonymy. Com- munications of the ACM, 8:627-633, 1965.","title":"Contextual correlates of synonymy","authors":"H Rubenstein, J B Goodenough","year":"1965","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saul and Roweis, 2003] Lawrence K. Saul and Sam T. Roweis. Think globally, fit locally: Unsupervised learn- ing of low dimensional manifolds. Journal of Machine Learning Research, 4:119-115, 2003.","title":"Think globally, fit locally: Unsupervised learning of low dimensional manifolds","authors":"Lawrence K Saul, Sam T Roweis","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for seman- tic compositionality over a sentiment treebank. In Proc. of EMNLP, pages 1631-1642, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, Christopher Potts","year":"2013","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tian et al., 2016] Yingtao Tian, Vivek Kullkarni, Bryan Per- ozzi, and Steven Skiena. On the convergent properties of word embedding methods. arXiv, 2016.","title":"On the convergent properties of word embedding methods","authors":"Tian","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tsuboi, 2014] Yuta Tsuboi. Neural networks leverage corpus-wide information for part-of-speech tagging. In Proc. of EMNLP, pages 938-950, 2014.","title":"Neural networks leverage corpus-wide information for part-of-speech tagging","authors":"Yuta Tsuboi, Tsuboi","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Turian et al., 2010] Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: A simple and general method for semi-supervised learning. In Proc. of ACL, pages 384 -394, 2010.","title":"Word representations: A simple and general method for semi-supervised learning","authors":"Turian","year":"2010","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Turney and Pantel, 2010] Peter D. Turney and Patrick Pan- tel. From frequency to meaning: Vector space models of semantics. Journal of Aritificial Intelligence Research, 37:141 -188, 2010.","title":"From frequency to meaning: Vector space models of semantics","authors":"D Peter, Patrick Turney, Pantel","year":"2010","venue":"Journal of Aritificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vylomova et al., 2016] Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relational learning. In ACL, pages 1671-1682, 2016.","title":"Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relational learning","authors":"Ekaterina Vylomova, Laura Rimell, Trevor Cohn, Timothy Baldwin","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin and Schütze, 2016] Wenpeng Yin and Hinrich Schütze. Learning meta-embeddings by using ensembles of embed- ding sets. In Proc. of ACL, pages 1351-1360, 2016.","title":"Learning meta-embeddings by using ensembles of embedding sets","authors":"Schütze Yin, Wenpeng Yin, Hinrich Schütze","year":"2016","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zou et al., 2013] Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. Bilingual word embeddings for phrase-based machine translation. In Proc. of EMNLP, pages 1393-1398, 2013.","title":"Bilingual word embeddings for phrase-based machine translation","authors":"Will Y Zou, Richard Socher, Daniel Cer, Christopher D Manning","year":"2013","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}