{"type":"ScholarlyArticle","title":"Getting in Shape: Word Embedding SubSpaces","authors":[{"name":"Tianyuan Zhou","slug":"tianyuan-zhou","given":"Tianyuan","family":"Zhou","affiliation":"Department of Statistics , University of Virginia"},{"name":"João Sedoc","slug":"joao-sedoc","given":"João","family":"Sedoc","affiliation":"Department of Computer and Information Science , University of Pennsylvania"},{"name":"Jordan Rodu","slug":"jordan-rodu","given":"Jordan","family":"Rodu","affiliation":"Department of Statistics , University of Virginia"}],"abstract":"Many tasks in natural language processing require the alignment of word embeddings. Embedding alignment relies on the geometric properties of the manifold of word vectors. This paper focuses on supervised linear alignment and studies the relationship between the shape of the target embedding. We assess the performance of aligned word vectors on semantic similarity tasks and find that the isotropy of the target embedding is critical to the alignment. Furthermore, aligning with an isotropic noise can deliver satisfactory results. We provide a theoretical framework and guarantees which aid in the understanding of empirical results.","doi":"10.24963/ijcai.2019/761","url":"https://ijcai.org/proceedings/2019/761","pdf":"https://ijcai.org/proceedings/2019/0761.pdf","markdown":"https://ijcai.org/proceedings/2019/761.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/761","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":761,"pages":{"first":5478,"last":5484},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Andreas and Klein, 2015] Jacob Andreas and Dan Klein. When and why are log-linear models self-normalizing? In Proceedings of NAACL HLT, pages 244-249, 2015.","title":"When and why are log-linear models self-normalizing","authors":"Andreas, Klein, Jacob Andreas, Dan Klein","year":"2015","venue":"Proceedings of NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2015] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Rand-walk: A latent variable model approach to word embeddings. arXiv preprint arXiv:1502.03520, 2015.","title":"Rand-walk: A latent variable model approach to word embeddings","authors":"Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Artetxe et al., 2016] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of EMNLP, pages 2289-2294, 2016.","title":"Learning principled bilingual mappings of word embeddings while preserving monolingual invariance","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of represen- tations for domain adaptation. In NIPS, pages 137-144, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blitzer et al., 2011] John Blitzer, Sham Kakade, and Dean Foster. Domain adaptation with coupled subspaces. In Proceedings of AISTAT, pages 173-181, 2011.","title":"Domain adaptation with coupled subspaces","authors":"John Blitzer, Sham Kakade, Dean Foster","year":"2011","venue":"Proceedings of AISTAT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bojanowski et al., 2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vec- tors with subword information. TACL, 5:135-146, 2017.","title":"Enriching word vectors with subword information","authors":"Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Consortium, 2007] BNC Consortium. British national cor- pus, version 3 BNC XML edition, 2007.","title":"BNC Consortium. British national corpus, version 3 BNC XML edition","authors":null,"year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dhillon et al., 2015] Paramveer S. Dhillon, Dean P. Foster, and Lyle H. Ungar. Eigenwords: Spectral word embeddings. JMLR, 16(1):3035-3078, 2015.","title":"Eigenwords: Spectral word embeddings","authors":"S Paramveer, Dean P Dhillon, Lyle H Foster, Ungar","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Faruqui and Dyer, 2014] Manaal Faruqui and Chris Dyer. Community evaluation and exchange of word vectors at wordvectors. org. In Proceedings of ACL: System Demon- strations, pages 19-24, 2014.","title":"Community evaluation and exchange of word vectors at wordvectors","authors":"Dyer Faruqui, Manaal Faruqui, Chris Dyer","year":"2014","venue":"Proceedings of ACL: System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Faruqui and Dyer, 2015] Manaal Faruqui and Chris Dyer. Non-distributional word vector representations. In Proceed- ings of ACL, pages 464-469, Beijing, China, July 2015. Association for Computational Linguistics.","title":"Manaal Faruqui and Chris Dyer. Non-distributional word vector representations","authors":"Dyer Faruqui","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fernando et al., 2013] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceed- ings of the IEEE international conference on computer vision, pages 2960-2967, 2013.","title":"Unsupervised visual domain adaptation using subspace alignment","authors":"Basura Fernando, Amaury Habrard, Marc Sebban, Tinne Tuytelaars","year":"2013","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hartmann et al., 2018] Mareike Hartmann, Yova Ke- mentchedjhieva, and Anders Søgaard. Why is unsupervised alignment of English embeddings from different algorithms so hard? In Proceedings of EMNLP, pages 582-586, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.","title":"Why is unsupervised alignment of English embeddings from different algorithms so hard?","authors":"Mareike Hartmann, Yova Kementchedjhieva, Anders Søgaard","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lample et al., 2018] Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In ICLR, 2018.","title":"Word translation without parallel data","authors":"Guillaume Lample, Alexis Conneau, Marc'aurelio Ranzato, Ludovic Denoyer, Hervé Jégou","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Landauer et al., 1998] Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. An introduction to latent semantic analysis. Discourse processes, 25(2-3):259-284, 1998.","title":"An introduction to latent semantic analysis","authors":"Thomas K Landauer, Peter W Foltz, Darrell Laham","year":"1998","venue":"Discourse processes","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Levy and Goldberg, 2014a] Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In Proceedings of ACL, volume 2, pages 302-308, 2014.","title":"Dependency-based word embeddings","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levy and Goldberg, 2014b] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NIPS, pages 2177-2185, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019a] Tianlin Liu, Lyle Ungar, and João Sedoc. Continual learning for sentence representations using con- ceptors. In Proceedings of NAACL HLT, 2019.","title":"Continual learning for sentence representations using conceptors","authors":"Liu","year":"2019","venue":"Proceedings of NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019b] Tianlin Liu, Lyle Ungar, and João Sedoc. Unsupervised post-processing of word vectors via concep- tor negation. In Proceedings of AAAI, 2019.","title":"Unsupervised post-processing of word vectors via conceptor negation","authors":"Liu","year":"2019","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013.","title":"Exploiting similarities among languages for machine translation","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mimno and Thompson, 2017a] David Mimno and Laure Thompson. The strange geometry of skip-gram with neg- ative sampling. In Proceedings of EMNLP, pages 2873- 2878, 2017.","title":"The strange geometry of skip-gram with negative sampling","authors":"Thompson Mimno, David Mimno, Laure Thompson","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mimno and Thompson, 2017b] David Mimno and Laure Thompson. The strange geometry of skip-gram with neg- ative sampling. In Proceedings of EMNLP, pages 2873- 2878, 2017.","title":"David Mimno and Laure Thompson. The strange geometry of skip-gram with negative sampling","authors":"Thompson Mimno","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mu et al., 2017] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective post- processing for word representations. arXiv preprint arXiv:1702.01417, 2017.","title":"All-but-the-top: Simple and effective postprocessing for word representations","authors":"Jiaqi Mu, Suma Bhat, Pramod Viswanath","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532- 1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shin et al., 2018] Jamin Shin, Andrea Madotto, and Pascale Fung. Interpreting word embeddings with eigenvector anal- ysis. https:// openreview.net/ pdf?id=rJfJiR5ooX, 2018.","title":"Interpreting word embeddings with eigenvector analysis","authors":"Jamin Shin, Andrea Madotto, Pascale Fung","year":null,"venue":null,"doi":null,"url":"https://openreview.net/pdf?id=rJfJiR5ooX,2018","internal_url":null},{"position":26,"raw":"[Tsvetkov et al., 2015] Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. Evalu- ation of word vector representations by subspace alignment. In Proceedings of EMNLP, pages 2049-2054, 2015.","title":"Evaluation of word vector representations by subspace alignment","authors":"Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, Chris Dyer","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang and Mahadevan, 2013] Chang Wang and Sridhar Ma- hadevan. Manifold alignment preserving global geometry. In IJCAI, pages 1743-1749, 2013.","title":"Manifold alignment preserving global geometry","authors":"Chang Wang, Sridhar Mahadevan","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/258"},{"position":28,"raw":"[Xing et al., 2015] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the NAACL HLT, pages 1006-1011, 2015.","title":"Normalized word embedding and orthogonal transform for bilingual word translation","authors":"Chao Xing, Dong Wang, Chao Liu, Yiye Lin","year":"2015","venue":"Proceedings of the NAACL HLT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}