{"type":"ScholarlyArticle","title":"Refining Word Representations by Manifold Learning","authors":[{"name":"Chu Yonghe","slug":"chu-yonghe","given":"Chu","family":"Yonghe","affiliation":""},{"name":"Hongfei Lin","slug":"hongfei-lin","given":"Hongfei","family":"Lin","affiliation":""},{"name":"Liang Yang","slug":"liang-yang","given":"Liang","family":"Yang","affiliation":""},{"name":"Yufeng Diao","slug":"yufeng-diao","given":"Yufeng","family":"Diao","affiliation":""},{"name":"Shaowu Zhang","slug":"shaowu-zhang","given":"Shaowu","family":"Zhang","affiliation":""},{"name":"Fan Xiaochao","slug":"fan-xiaochao","given":"Fan","family":"Xiaochao","affiliation":""}],"abstract":"Pre-trained distributed word representations have been proven useful in various natural language processing (NLP) tasks. However, the effect of words’ geometric structure on word representations has not been carefully studied yet. The existing word representations methods underestimate the words whose distances are close in the Euclidean space, while overestimating words with a much greater distance. In this paper, we propose a word vector refinement model to correct the pre-trained word embedding, which brings the similarity of words in Euclidean space closer to word semantics by using manifold learning. This approach is theoretically founded in the metric recovery paradigm. Our word representations have been evaluated on a variety of lexical-level intrinsic tasks (semantic relatedness, semantic similarity) and the experimental results show that the proposed model outperforms several popular word representations approaches.","doi":"10.24963/ijcai.2019/749","url":"https://ijcai.org/proceedings/2019/749","pdf":"https://ijcai.org/proceedings/2019/0749.pdf","markdown":"https://ijcai.org/proceedings/2019/749.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/749","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":749,"pages":{"first":5394,"last":5400},"keywords":["Natural Language Processing: NLP Applications and Tools","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Agirre et al., 2009] Eneko Agirre, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, Aitor Soroa. A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. In Proc. of HLT-NAACL, pages 19-27,2009.","title":"A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches","authors":"Eneko Agirre, Enrique Alfonseca, Keith B Hall, Jana Kravalova, Marius Pasca, Aitor Soroa","year":"2009","venue":"Proc. of HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bruni et al., 2014] Elia Bruni, Nam Khanh Tran, Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49(1):1-47,2014.","title":"Multimodal distributional semantics","authors":"Elia Bruni, Nam Khanh Tran, Marco Baroni","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel P. Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537,2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel P Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collobert and Weston, 2008] Ronan Collobert, Jason Wes- ton. A unified architecture for natural language pro- cessing: deep neural networks with multitask learning. In Proc. of ICML, pages 160-167, 2008.","title":"A unified architecture for natural language processing: deep neural networks with multitask learning","authors":"Ronan Collobert, Jason Weston","year":"2008","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Collell et al., 2017] Guillem Collell, Ted Zhang, Ma- rie-Francine Moens. Imagined Visual Representations as Multimodal Embeddings. In Proc. of AAAI, pages 4378-4384, 2017.","title":"Imagined Visual Representations as Multimodal Embeddings","authors":"Guillem Collell, Ted Zhang, Marie-Francine Moens","year":"2017","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Finkelstein et al., 2001] Lev Finkelstein, Evgeniy Gabrilo- vich, Yossi Matias. Placing search in context: The con- cept revisited. In Proc. of WWW, pages 406-414, 2001.","title":"Placing search in context: The concept revisited","authors":"Finkelstein","year":"2001","venue":"Proc. of WWW","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gerz et al., 2016] Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, Anna Korhonen. Simverb-3500: A large-scale evaluation set of verb similarity. arXiv preprint arXiv:1608.00869,2016.","title":"Simverb-3500: A large-scale evaluation set of verb similarity","authors":"Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, Anna Korhonen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hasan and Curry, 2017] Souleiman Hasan, Edward Curry. Word Re-Embedding via Manifold Dimensionality Re- tention, In Proc. of EMNLP, pages 321-326, 2017.","title":"Word Re-Embedding via Manifold Dimensionality Retention","authors":"Souleiman Hasan, Edward Curry","year":"2017","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hashimoto et al., 2016] Tatsunori B Hashimoto, David Alvarez-Melis, Tommi S Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics, 4:273-286, 2016.","title":"Word embeddings as metric recovery in semantic spaces","authors":"B Tatsunori, David Hashimoto, Tommi S Alvarez-Melis, Jaakkola","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hill and Korhonen, 2015] Felix Hill, Anna Korho- nen.Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41(4): 665-695, 2015.","title":"Felix Hill, Anna Korhonen.Simlex-999: Evaluating semantic models with (genuine) similarity estimation","authors":"Korhonen Hill","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kira et al., 2011] Radinsky Kira, Eugene Agichtein, Evgeniy Gabrilovich. A word at a time: computing word relatedness using temporal semantic analysis. In Proc. of WWW, pages 337-346, 2011.","title":"A word at a time: computing word relatedness using temporal semantic analysis","authors":"Radinsky Kira, Eugene Agichtein, Evgeniy Gabrilovich","year":"2011","venue":"Proc. of WWW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Labutov and Lipson, 2016] lgor Labutov, Hod Lipson. Re-embedding words. In Proc. of ACL, pages 489-493, 2013.","title":"lgor Labutov, Hod Lipson. Re-embedding words","authors":"Lipson Labutov","year":"2013","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2016] Yang-Yin Lee, Hao Ke, Hen-Hsen Huang, and Hsin-Hsi Chen. Less is More: Filtering Abnormal Dimensions in GloVe. In Proc. of WWW, pages 71-72, 2016.","title":"Less is More: Filtering Abnormal Dimensions in GloVe","authors":"Yang-Yin Lee, Hao Ke, Hen-Hsen, Hsin-Hsi Huang, Chen","year":"2016","venue":"Proc. of WWW","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lazaridou et al., 2015] Angeliki Lazaridou, Nghia The Pham, Marco Baroni. Combining language and vision with a multimodal skip-gram model. In Proc. of NAACL, pages 153-163,2015.","title":"Combining language and vision with a multimodal skip-gram model","authors":"Lazaridou","year":"2015","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Proc. of NIPS, pages 3111-3119, 2013b.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. In Proc. of ICLR, 2013a.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mu and Viswanath, 2018] Jiaqi Mu, Suma Bhat, Pramod Viswanath. All-but-the-top: Simple and effective post- processing for word representations. In Proc. of ICLR, 2018.","title":"All-but-the-top: Simple and effective postprocessing for word representations","authors":"Jiaqi Mu, Suma Bhat, Pramod Viswanath","year":"2018","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014a] Jeffrey Pennington, Richard So- cher, Christopher D. Manning. Glove: Global vectors for word representation. In Proc. of EMNLP, pages 1532-1543,2014a.","title":"Glove: Global vectors for word representation","authors":"Pennington","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014b] Jeffrey Pennington, Richard So- cher, and Christopher D Manning. Glove resources. Available :http://nlp.stanford.edu/projects/glove/,2014b.","title":"Jeffrey Pennington, Richard Socher, and Christopher D Manning","authors":"Pennington","year":"2014","venue":null,"doi":null,"url":"http://nlp.stanford.edu/projects/glove/,2014b","internal_url":null},{"position":20,"raw":"[Qiu et al., 2014] Lin Qiu, Yong Cao, Zaiqing Nie, Yong Yu, Yong Rui. Learning word representation considering proximity and ambiguity. In Proc. of AAAI, pages 1572-1578, 2014.","title":"Learning word representation considering proximity and ambiguity","authors":"Lin Qiu, Yong Cao, Zaiqing Nie, Yong Yu, Yong Rui","year":"2014","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roweis and Saul, 2000] Sam T. Roweis, Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290 (5500): 2323-2326,2000.","title":"Nonlinear dimensionality reduction by locally linear embedding","authors":"T Sam, Lawrence K Roweis, Saul","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rubenstein et al., 2000] Herbert Rubenstein, John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):627-633,1965.","title":"Contextual correlates of synonymy","authors":"Herbert Rubenstein, John B Goodenough","year":"1965","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silberer and Lapata, 2014] Carina Silberer, Mirella Lapata. Learning grounded meaning representations with auto- encoders. In Proc. of ACL, pages 721-732,2014.","title":"Learning grounded meaning representations with autoencoders","authors":"Carina Silberer, Mirella Lapata","year":"2014","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Shaonan Wang, Jiajun Zhang, Chengqing Zong. Learning Multimodal Word Represen- tation via Dynamic Fusion Methods. In Proc. of AAAI, 2018.","title":"Learning Multimodal Word Representation via Dynamic Fusion Methods","authors":"Wang","year":"2018","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2018] Liang-Chih Yu, Jin Wang, K. Robert Lai, Xuejie Zhang . Refining Word Embeddings Using Intensity Scores for Sentiment Analysis. IEEE/ACM Trans. Audio, Speech & Language Processing, 26(3): 671-681,2018.","title":"Refining Word Embeddings Using Intensity Scores for Sentiment Analysis","authors":"Liang-Chih Yu, Jin Wang, K Robert Lai, Xuejie Zhang","year":"2018","venue":"IEEE/ACM Trans. Audio, Speech & Language Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Wang, 2006] Zhenyue Zhang, Jing Wang. MLLE: Modified Locally Linear Embedding Using Multiple Weights. In Proc. of NIPS, pages 1593-1600,2006.","title":"MLLE: Modified Locally Linear Embedding Using Multiple Weights","authors":"Wang ; ] Zhenyue Zhang, Jing Zhang, Wang","year":"2006","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}