Getting in Shape: Word Embedding SubSpaces

1 Introduction

Mono-lingual and multi-lingual alignment of word embeddings is important for domain adaptation, word embedding assessment, and machine translation [Ben-David et al., 2007; Blitzer et al., 2011; Tsvetkov et al., 2015; Lample et al., 2018]. Fundamentally, this is a subspace alignment problem which has seen much interest in machine learning communities, including computer vision and nature language processing (NLP) [Fernando et al., 2013; Xing et al., 2015; Wang and Mahadevan, 2013; Lample et al., 2018; Mikolov et al., 2013a] and can be either supervised or unsupervised, depending on the setting.

Simultaneously, some work has focused on uncovering the structure of word embedding manifolds [Mimno and Thompson, 2017a; Hartmann et al., 2018; Shin et al., 2018; Mu et al., 2017], where in this paper we focus their isotropy/anisotropy. Word embeddings are known to not be isotropic [Andreas and Klein, 2015], but Arora [2015] argue that isotropic word embeddings mitigate the effect of approximation error. Indeed, recent work in post-processing word embeddings has shown that increasing isotropy increases semantic task performance [Mu et al., 2017; Liu et al., 2019b; Liu et al., 2019a].

∗Corresponding Author

Link of Supplementary Materials and Source codes: https:// github.com/NoahZhouTianyuan/ConceptorOnNondistEmbedding

Despite a large body of work in each of the two areas, the link between subspace alignment and manifold isotropy has not been fully explored. Given that word embeddings contain some form of "meaning", presumably common to all word embeddings, different representations should align to some degree. But alignment necessarily prioritizes resolving disparity in larger singular values, which could be problematic for two word embeddings that encode information differently across their spectrums. When two word embeddings represent information similarly, they can be successfully aligned. For instance, Artetxe [2016] shows orthogonal transformations for alignment are superior for retaining performance on analogy tasks. However, orthogonal transformations do not allow for the alignment of more disparate methods such as distributional and non-distributional word embeddings.

In this paper, we present a theoretical framework for understanding the alignment between word embeddings- when they can work, and when they might fail. Our theoretical results show that the singular value structure of the source embeddings is completely discarded, and when information is encoded differently in the source and target embeddings, the distortion of the spectrum from source to aligned embeddings could drastically effect downstream results.

2 Theoretical Results

In this section we provide some theoretical underpinnings for the phenomena observed in our experiments. Lemma 1 shows that when a source representation is aligned to a target representation using a linear transformation, the column space of the aligned representation is determined by the column space of the source, but the singular value structure of the source is entirely discarded. Theorem 1 guarantees the existence of a lower-bounded singular value. Since the summation of the singular values is upper-bounded by the target embedding, if the lower bound of the largest singular value is relatively large in comparison, then the singular value structure will have high eccentricity.

Finally, proposition 1 shows that the correlation of measurements between words (Euclidean distance between vectors or the cosine of their angles) in a vector cloud, and those measurements after the vector cloud has been stretched by unequal amounts in different directions, is small. Combined with the results of theorem 1, this implies that alignment can greatly impact the performance of aligned word representations on semantic similarity tasks. We expand on this below.

We include the proof of the theorem here, but defer the proof of the lemma and the proposition to the supplementary material. We also provide additional theorems in the supplementary material that, while not critical for the main focus of this work, provide justification for some minor experimental results and contribute more broadly to our fundamental theoretical understanding of word representation alignment.

We begin by stating our word representation alignment model.

Alignment Model: Let Y ∈ Rn×p be the target word representations, and X ∈ Rn×p the source representations. To align X to Y , we seek a linear mapping W such that Y = XW + ϵ. The least squares solution for W yields ̂ Y =XW ∗ =X(XT X)-1XT Y = UXU TX Y .

Lemma 1. Let UX and UY be the left singular vectors of X and Y respectively, and ΣY a diagonal matrix containing the singular values of Y , then σ(̂ Y ) = σ(U TX UY ΣY ).

For the following theorem, define σi(M) to be the ith singular value of the matrix M, and UMi to be the singular vector associated with the ith singular value of M.

Theorem 1. Suppose X, Y ∈ Rn×p , and let ̂ Y = XW ∗ be the least squares solution of Y = XW + ϵ. For singular value σi(Y ) and corresponding left singular vector Uyi, let ci = ||U ⊤ X Uyi||2, then there exists a singular value in ̂ Y at least as large as ciσi(Y ).

Proof. By Lemma 1, σ(ˆ Y ) = σ(U TX UY ΣY ). Recalling the definition of singular values, σ1(U TX UY ΣY ) = max||u||2=1,||v||2=1 uT U TX UY ΣY v where u, v are unit vectors. Define ei = (0, 0, · · · , 0, 1, 0, · · · , 0)T where ei is a unit vector whose elements are 0 except the ith element which is 1. Then for any 1 ≤ i ≤ p, σ1(U ⊤ X UY ΣY ) ≥ max||u||2=1 uT U TX UY ΣY ei = max||u||2=1 u⊤U ⊤ X σi(Y )Uyi = σi(Y )||U ⊤ X Uyi||2 = ciσi(Y ). Therefore, the largest singular value must be greater than or equal to ciσi(Y ), for all i.

Proposition 1. Suppose X ∈ Rp is a random vector with all entries distributed i.i.d with mean zero and a bounded fourth moment. Let S be a matrix with √s1, √s2, · · · , √sp) along the diagonal and 0 everywhere else. Then for realizations Xi and Xj of X, we have the following two results:

and

Further, for a subset I ⊂ {1, . . . , p} define Xi,I to be the vector Xi with indices not in I set to 0. Then

To see the implications of this proposition, consider the alignment process loosely as 1) establishing the directions of the aligned singular vectors (which could be by an orthogonal rotation of the source singular vectors, or through linear combinations of the source embeddings if the information content is not encoded similarly in the source and target embeddings), 2) stretching the singular vectors according to the singular values of the source embeddings (or functions of the singular values that distribute their information appropriately if the transformation is not orthogonal), and 3) adjusting the stretched singular vectors through the S matrix of proposition 1 according to the spectrum of the target embeddings (see lemma 1). For two word embeddings that encode information similarly (say two distributional word embeddings) the entries of the S matrix will all be roughly equal. However, for two word embeddings that do not, it is likely that significant adjustment will be required, with some entries in the S large to stretch the singular vectors, and some small to shrink them. The results of our experiments support this idea.

3 Empirical Results

In this section, we show some empirical results of word representation alignment. Our key finding suggests that isotropy is important to successful alignment. In fact, aligning with isotropic noise can even yield satisfactory intrinsic evaluation results.

Conceptor Negation (CN). It is worth noting that the geometry of the distributional word embedding has been studied carefully [Mimno and Thompson, 2017b]. Mimno et al. [2017b] note that for word2vec word embedding point clouds are concentrated within a narrow cone, which may lead to bias. ABTT [Mu et al., 2017] and conceptor negation [Liu et al., 2019b] are two methods used to correct this bias by damping the larger singular values. Hence we suggest that conceptor negation should be used post alignment in order to control the eccentricity of the resulting aligned representation.

3.1 Experimental Setup

We perform multiple experiments using distributional word representations (each 300-dimensional) including word2vec [Mikolov et al., 2013b] (Google News), GloVe [Pennington et al., 2014] (840 billion Common Crawl) and FastText [Bojanowski et al., 2017] (Common Crawl without subword), as our source embeddings, and align them through linear regression to various target representations. We then test the aligned word vectors on seven similarity tasks [Faruqui and Dyer, 2014], and in some cases an additional three concept categorization tasks as supplement. Our target representations are (1) other distributional word representations; (2) word representations such as low-rank non-distributional word vectors with singular values [Faruqui and Dyer, 2015], LSA [Landauer et al., 1998] (on British National Corpus [Consortium, 2007]), Eigenwords [Dhillon et al., 2015] and dependency word embedding [Levy and Goldberg, 2014a]; (3) The same word representations in (2) but with the labels permuted; and (4) designed noise, including isotropic Gaussian noise and rank-1 noise. We also ran experiments in which we performed conceptor negation on the target, source, or aligned word representation, and compared these results to the non-conceptored versions.

Note that the shapes of the manifolds of the different distributional word representations are quite similar and the alignment of distributional source representations to distributional target representations has a high R2 . In contrast, the shapes of the manifolds of the word representations in (2) are quite different than in distributional word representations, and as a consequence the alignment has a relatively low R2 . The permuted embeddings disassociate the labels with the representations, thus breaking the supervised nature of alignment. Finally, the designed noise simulates two extreme cases, one purely isotropic target representation and one highly non-isotropic.

Alignment Order for Display of Semantic Similarity Results

To better display and compare the results of our alignment experiments on semantic similarity tasks, we establish a sequence of alignments. For aligning two distributional word representations (call them representation 1 and representation 2), the alignment sequence is (in parentheses are example labels of taking word2vec as representation 1 and GloVe as representation 2): (1) representation 1 (e.g. W2V), (2) conceptor negated representation 1 (e.g. W2V+CN), (3) representation 1 aligned to conceptor negated representation 2 (e.g. W2V by GV+CN), (4) representation 1 aligned to representation 2 (e.g. W2V by GV), (5) representation 2 aligned to representation 1 (e.g. GV by W2V), (6) representation 2 aligned to conceptor negated representation 1 (e.g. GV by W2V+CN), (7) conceptor negated representation 2 (e.g. GV+CN), (8) representation 2 (e.g. GV). Results for distributional sources and targets are shown in Fig 1.

For a distributional source aligned with an 'other' target, the sequence is (in parentheses are the labels shown in figures): (1) Source representation (Source Embedding), (2) source aligned with permuted and conceptor negated target (Aligned by permuted+CN Target), (3) source aligned with permuted target (Aligned by permuted Target), (4) source aligned with target (Aligned by Target), (5) target representation (Target Embedding). For example, the results for the FastText source and non-distributional target representations are shown in Fig 4.

3.2 Alignment Results

Distributional Source and Target Representations

Distributional word vector representations model similar information [Levy and Goldberg, 2014b; Pennington et al., 2014] implying that the manifold shapes are similar across representations. We show that aligning with conceptor negated target vectors leads to improved semantic similarity task scores. From Corollary 1, we have that, when the subspaces of the source and target embeddings are nearly identical, the singular value information of the target representations dictate the information of the aligned word representations, which predicts that conceptor negating the target representation yields good results. The results are shown in Fig 1.

Distributional Source with Other Word Representations Distributional source with non-distributional targets.

Non-distributional word representations typically encode lexical information, and thus the shape of the representation manifold is quite different from that of distributional representations, confirmed by their differences in performance as seen in Fig 3. Alignment of distributional word representations to non-distributional word representations highlights another important consequence of Lemma 1. Namely, both the singular values of the target representations, and the degree of overlap in the subspaces spanned by the source and target representations, matter in determining the singular value structure of the aligned representations. Importantly, the distributional and non-distributional subspaces seem to overlap only in directions with high target singular values- Fig 2 shows that singular vectors corresponding to large singular values have much higher R2 . From Theorem 1 then, the largest singular value of the aligned representation is lower-bounded, particularly by ||U ⊤ X Uy1||2σ1(Y ) as both σ1(Y ) and ||U ⊤ X Uy1||2 are largest among σi(Y )s and ||U ⊤ X Uyi||2s respectively, resulting in highly eccentric aligned word representations. Also, the overall R2 when the target embedding is not conceptornegated is higher because these singular vectors contribute more than when the target embedding is conceptor-negated. This suggests that the best way to combine the information from distributional and non-distributional word vectors should be through concatenation instead of alignment. The sequential comparisons (Fig 4) generally show that alignment with the 'permuted target' ('Aligned by permuted Target' in figures) or the unchanged target ('Aligned by Target' in figures) tend to give the worst results, while aligning with 'permuted + conceptor negated target' ('Aligned by permuted+CN Target' in figures) typically performs much better. A major difference between the source representation and the aligned representation is the singular value structure, as the singular vector structure of the source is lost. This explains why aligning with the 'permuted + conceptor negated target' does relatively well. The information from the target singular values is tempered, making the aligned vectors more isotropic. In contrast, aligning with 'permuted target' or the unchanged target transmit full information of the target singular values. Target singular value information seems most detrimental when the source and target representations cannot be aligned well. Note that although aligning with 'permute + conceptor negated target' performs well on semantic similarity tasks, the R2 is nearly 0.

Distributional source with other targets (except nondistributional). For distributional source representations aligned with other target representations (Figs 5 - 10), the sequential comparison trends are similar: aligning the distributional word vectors with permuted target representations yields the worst results in most cases, while aligning with the unchanged target performs worse than aligning with permuted and conceptor negated target. In some cases, aligning with a permuted and conceptor negated target yields better results than the source embedding, especially GloVe, though the R2 is very low. Results of alignments with dependency CBOW and GloVe (500 dimensions) are included in the supplementary material.

Distributional Source with Designed Noise Target

To further understand the scope of and issues with alignment, we align the distributional source representations with two types of designed noise targets, isotropic Gaussian noise and rank-1 noise. The results (Fig [11]), show that aligning with Gaussian noise only minimally decreases the performance from the performance of the source embedding, while aligning with rank-1 noise destroys information across the board. This result is a consequence of Proposition 1.

4 Conclusion

Despite the success of word representations in NLP tasks, relatively little is known about their theoretical properties. In this paper we try to aid understanding through studying the properties of word representation alignment. We show through theory and experiment that the target representations drive the information content of the singular values of the aligned representations, while the source representations preserve the subspace. In theory, a carefully designed experiment with carefully constructed tasks could help tease apart what aspects of the word representations encode which characteristics of words. We lay this groundwork here.

Further, one must take care when performing word representation alignment. While there are demonstrated benefits to alignment, they do not uniformly apply to semantic similarity tasks. Our theoretical framework provides guidance as to why alignment might work in some cases but not in others. For instance, our theory and results provide justification for why concatenating distributional and non-distributional word representations is preferable to alignment.

Establishing a theoretical foundation for understanding word representations will provide impetus for improved performance of word representations in NLP tasks. In this paper, we lay the groundwork for understanding alignment which, in addition to allowing for the integration of information across word representations, can provide a unique lens into how word representations encode information.