{"type":"ScholarlyArticle","title":"Learning Word Vectors with Linear Constraints: A Matrix Factorization Approach","authors":[{"name":"Wenye Li","slug":"wenye-li","given":"Wenye","family":"Li","affiliation":"The Chinese University of Hong Kong , Shenzhen , China; Shenzhen Research Institute of Big Data , Shenzhen , China"},{"name":"Jiawei Zhang","slug":"jiawei-zhang","given":"Jiawei","family":"Zhang","affiliation":"The Chinese University of Hong Kong , Shenzhen , China"},{"name":"Jianjun Zhou","slug":"jianjun-zhou","given":"Jianjun","family":"Zhou","affiliation":"Shenzhen Research Institute of Big Data , Shenzhen , China"},{"name":"Laizhong Cui","slug":"laizhong-cui","given":"Laizhong","family":"Cui","affiliation":"Shenzhen University , Shenzhen , China"}],"abstract":"Learning vector space representation of words, or word embedding, has attracted much recent research attention. With the objective of better capturing the semantic and syntactic information inherent in words, we propose two new embedding models based on the singular value decomposition of lexical co-occurrences of words. Different from previous work, our proposed models allow for injecting linear constraints when performing the decomposition, with which the desired semantic and syntactic information will be maintained in word vectors. Conceptually the models are flexible and convenient to encode prior knowledge about words. Computationally they can be easily solved by direct matrix factorization. Surprisingly simple yet effective, the proposed models have reported significantly improved performance in empirical word analogy and sentence classification evaluations, and demonstrated high potentials in practical applications.","doi":"10.24963/ijcai.2018/582","url":"https://ijcai.org/proceedings/2018/582","pdf":"https://ijcai.org/proceedings/2018/0582.pdf","markdown":"https://ijcai.org/proceedings/2018/582.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/582","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":582,"pages":{"first":4187,"last":4193},"keywords":["Natural Language Processing: Natural Language Processing","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pas- cal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio, 2009] Yushua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2:1- 127, 2009.","title":"Learning deep architectures for AI","authors":"Yushua Bengio, Bengio","year":"2009","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beymer and Poggio, 1996] David Beymer and Tomaso Pog- gio. Image representations for visual learning. Science, 272:1905-1909, 1996.","title":"David Beymer and Tomaso Poggio. Image representations for visual learning","authors":"Poggio Beymer","year":"1996","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bullinaria and Levy, 2007] John Bullinaria and Joseph Levy. Extracting semantic representations from word co- occurrence statistics: A computational study. Behavior Research Methods, 39:510-526, 2007.","title":"Extracting semantic representations from word cooccurrence statistics: A computational study","authors":"Levy ; Bullinaria, John Bullinaria, Joseph Levy","year":"2007","venue":"Behavior Research Methods","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Faruqui et al., 2014] Manaal Faruqui, Jesse Dodge, Su- jay Jauhar, Chris Dyer, Eduard Hovy, and Noah Smith. Retrofitting word vectors to semantic lexicons. arXiv:1411.4166, 2014.","title":"Retrofitting word vectors to semantic lexicons","authors":"Faruqui","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganitkevitch et al., 2013] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 758-764, 2013.","title":"PPDB: The paraphrase database","authors":"Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch","year":"2013","venue":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Golub and Van Loan, 2012] Gene Golub and Charles Van Loan. Matrix Computations. John Hopkins University Press, 2012.","title":"Gene Golub and Charles Van Loan","authors":"Van Golub, Loan","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hofmann, 1999] Thomas Hofmann. Probabilistic latent se- mantic analysis. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, pages 289-296.","title":"Probabilistic latent semantic analysis","authors":"Thomas Hofmann","year":null,"venue":"Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiela et al., 2015] Douwe Kiela, Felix Hill, and Stephen Clark. Specializing word embeddings for similarity or relatedness. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2044-2048, 2015.","title":"Specializing word embeddings for similarity or relatedness","authors":"Kiela","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. arXiv:1408.5882, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Landauer and Dumais, 1997] Thomas Landauer and Susan Dumais. A solution to Plato's problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 1997.","title":"A solution to Plato's problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge","authors":"Dumais Landauer, Thomas Landauer, Susan Dumais","year":"1997","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lang, 1995] Ken Lang. Newsweeder: Learning to filter net- news. In Machine Learning Proceedings 1995, pages 331- 339. Elsevier, 1995.","title":"Newsweeder: Learning to filter netnews","authors":"Lang ;, Ken Lang","year":"1995","venue":"Machine Learning Proceedings 1995","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lebret and Collobert, 2013] Rémi Lebret and Ronan Col- lobert. Word emdeddings through Hellinger PCA. arXiv:1312.5542, 2013.","title":"Word emdeddings through Hellinger PCA","authors":"Collobert Lebret","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems 27, pages 2177-2185.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2015] Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. Learning semantic word embeddings based on ordinal knowledge constraints. In Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics, pages 1501-1511, 2015.","title":"Learning semantic word embeddings based on ordinal knowledge constraints","authors":"Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, Yu Hu","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lund and Burgess, 1996] Kevin Lund and Curt Burgess. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28:203-208, 1996.","title":"Producing high-dimensional semantic spaces from lexical co-occurrence","authors":"Burgess ; Lund, Kevin Lund, Curt Burgess","year":"1996","venue":"Behavior Research Methods, Instruments, & Computers","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Miller, 1995] George Miller. Wordnet: a lexical database for English. Communications of the ACM, 38:39-41, 1995.","title":"Wordnet: a lexical database for English","authors":"George Miller, Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mrkšić et al., 2016] Nikola Mrkšić, Diarmuid Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei- Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Counter-fitting word vectors to linguistic con- straints. arXiv:1603.00892, 2016.","title":"Counter-fitting word vectors to linguistic constraints","authors":"Nikola Mrkšić, Diarmuid Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, Steve Young","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Osborne et al., 2015] Dominique Osborne, Shashi Narayan, and Shay Cohen. Encoding prior knowledge with eigen- word embeddings. arXiv:1509.01007, 2015.","title":"Encoding prior knowledge with eigenword embeddings","authors":"Dominique Osborne, Shashi Narayan, Shay Cohen","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd An- nual Meeting on Association for Computational Linguistics, pages 115-124, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rothe and Schütze, 2016] Sascha Rothe and Hinrich Schütze. Word embedding calculus in meaningful ultradense subspaces. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 512-517, 2016.","title":"Word embedding calculus in meaningful ultradense subspaces","authors":"Schütze Rothe, Sascha Rothe, Hinrich Schütze","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Salton et al., 1975] Gerard Salton, Anita Wong, and Chung- Shu Yang. A vector space model for automatic indexing. Communications of the ACM, 18:613-620, 1975.","title":"A vector space model for automatic indexing","authors":"Gerard Salton, Anita Wong, Chung-Shu Yang","year":"1975","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tversky, 1977] Amos Tversky. Features of similarity. Psy- chological Review, 84:327, 1977.","title":"Amos Tversky. Features of similarity","authors":"Tversky","year":"1977","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wieting et al., 2015] John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. From paraphrase database to compositional paraphrase model and back. arXiv:1506.03487, 2015.","title":"From paraphrase database to compositional paraphrase model and back","authors":"John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, Dan Roth","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2009] Linli Xu, Wenye Li, and Dale Schuurmans. Fast normalized cut with linear constraints. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2866-2873, 2009.","title":"Fast normalized cut with linear constraints","authors":"Xu","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu and Dredze, 2014] Mo Yu and Mark Dredze. Improving lexical embeddings with semantic knowledge. In Proceed- ings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 545-550, 2014.","title":"Improving lexical embeddings with semantic knowledge","authors":"Mo Yu, Mark Dredze","year":"2014","venue":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zeiler, 2012] Matthew Zeiler. Adadelta: an adaptive learning rate method. arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"Matthew Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}