{"type":"ScholarlyArticle","title":"Context Vectors Are Reflections of Word Vectors in Half the Dimensions (Extended Abstract)","authors":[{"name":"Zhenisbek Assylbekov","slug":"zhenisbek-assylbekov","given":"Zhenisbek","family":"Assylbekov","affiliation":"School of Sciences and Humanities , Nazarbayev University"},{"name":"Rustem Takhanov","slug":"rustem-takhanov","given":"Rustem","family":"Takhanov","affiliation":"School of Sciences and Humanities , Nazarbayev University"}],"abstract":"This paper takes a step towards the theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.","doi":"10.24963/ijcai.2020/718","url":"https://ijcai.org/proceedings/2020/718","pdf":"https://ijcai.org/proceedings/2020/0718.pdf","markdown":"https://ijcai.org/proceedings/2020/718.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/718","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Journal track","track":"Journal track","number":718,"pages":{"first":5115,"last":5119},"keywords":["Natural Language Processing: Embeddings","Machine Learning: Probabilistic Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Tensor and Matrix Methods"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24978/file_24978.m3u8","length_sec":230,"provider":"hls"}],"references":[{"position":1,"raw":"[Allen et al., 2019] Carl Allen, Ivana Balazevic, and Timo- thy Hospedales. What the vec? towards probabilistically grounded embeddings. In Advances in Neural Information Processing Systems, pages 7465-7475, 2019.","title":"What the vec? towards probabilistically grounded embeddings","authors":"Carl Allen, Ivana Balazevic, Timothy Hospedales","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2016] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Trans- actions of the Association for Computational Linguistics, 4:385-399, 2016.","title":"A latent variable model approach to pmi-based word embeddings","authors":"Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Assylbekov and Takhanov, 2019] Zhenisbek Assylbekov and Rustem Takhanov. Context vectors are reflections of word vectors in half the dimensions. Journal of Artificial Intelligence Research, 66:225-242, 2019.","title":"Zhenisbek Assylbekov and Rustem Takhanov. Context vectors are reflections of word vectors in half the dimensions","authors":"Takhanov Assylbekov","year":"2019","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2001] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. 2001.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bruni et al., 2012] Elia Bruni, Gemma Boleda, Marco Ba- roni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of ACL, pages 136-145. As- sociation for Computational Linguistics, 2012.","title":"Distributional semantics in technicolor","authors":"Elia Bruni, Gemma Boleda, Marco Baroni, Nam-Khanh Tran","year":"2012","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cancho and Solé, 2001] R Ferrer I Cancho and Ricard V Solé. The small world of human language. Proceedings of the Royal Society B: Biological Sciences, 268(1482):2261, 2001.","title":"The small world of human language","authors":"I Ferrer, Ricard V Cancho, Solé","year":"1482","venue":"Proceedings of the Royal Society B: Biological Sciences","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Farkas et al., 2001] Illés J Farkas, Imre Derényi, Albert- László Barabási, and Tamas Vicsek. Spectra of \"real- world\" graphs: Beyond the semicircle law. Physical Re- view E, 64(2):026704, 2001.","title":"Spectra of \"realworld\" graphs: Beyond the semicircle law","authors":"J Illés, Imre Farkas, Albert-László Derényi, Tamas Barabási, Vicsek","year":"2001","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finkelstein et al., 2002] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. ACM Transactions on information systems, 20(1):116-131, 2002.","title":"Placing search in context: The concept revisited","authors":"Finkelstein","year":"2002","venue":"ACM Transactions on information systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gittens et al., 2017] Alex Gittens, Dimitris Achlioptas, and Michael W Mahoney. Skip-gram-zipf+ uniform= vector additivity. In Proceedings of ACL, volume 1, pages 69-76, 2017.","title":"Skip-gram-zipf+ uniform= vector additivity","authors":"Alex Gittens, Dimitris Achlioptas, Michael W Mahoney","year":"2017","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goldberg and Levy, 2014] Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.'s negative- sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.","title":"word2vec explained: deriving mikolov et al.'s negativesampling word-embedding method","authors":"Levy ; Goldberg, Yoav Goldberg, Omer Levy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gulordava et al., 2018] Kristina Gulordava, Laura Aina, and Gemma Boleda. How to represent a word and pre- dict it, too: Improving tied architectures for language modelling. In Proceedings of EMNLP, pages 2936-2941, 2018.","title":"How to represent a word and predict it, too: Improving tied architectures for language modelling","authors":"Kristina Gulordava, Laura Aina, Gemma Boleda","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hashimoto et al., 2016] Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics, 4:273-286, 2016.","title":"Word embeddings as metric recovery in semantic spaces","authors":"B Tatsunori, David Hashimoto, Tommi S Alvarez-Melis, Jaakkola","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Inan et al., 2017] Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Proceedings of ICLR, 2017.","title":"Tying word vectors and word classifiers: A loss framework for language modeling","authors":"Hakan Inan, Khashayar Khosravi, Richard Socher","year":"2017","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levy and Goldberg, 2014a] Omer Levy and Yoav Gold- berg. Linguistic regularities in sparse and explicit word representations. In Proceedings of CoNLL, pages 171- 180, 2014.","title":"Linguistic regularities in sparse and explicit word representations","authors":"Goldberg Levy","year":"2014","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Levy and Goldberg, 2014b] Omer Levy and Yoav Gold- berg. Neural word embedding as implicit matrix factoriza- tion. In Proceedings of NeurIPS, pages 2177-2185, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy","year":"2014","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levy et al., 2015] Omer Levy, Yoav Goldberg, and Ido Da- gan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Asso- ciation for Computational Linguistics, 3:211-225, 2015.","title":"Improving distributional similarity with lessons learned from word embeddings","authors":"Omer Levy, Yoav Goldberg, Ido Dagan","year":"2015","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luong et al., 2013] Thang Luong, Richard Socher, and Christopher Manning. Better word representations with recursive neural networks for morphology. In Proceedings of CoNLL, pages 104-113, 2013.","title":"Better word representations with recursive neural networks for morphology","authors":"Thang Luong, Richard Socher, Christopher Manning","year":"2013","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Melamud et al., 2017] Oren Melamud, Ido Dagan, and Ja- cob Goldberger. A simple language model based on pmi matrix approximations. In Proceedings of EMNLP, pages 1860-1865, 2017.","title":"A simple language model based on pmi matrix approximations","authors":"Oren Melamud, Ido Dagan, Jacob Goldberger","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In Proceedings of NeurIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mimno and Thompson, 2017] David Mimno and Laure Thompson. The strange geometry of skip-gram with neg- ative sampling. In Proceedings of EMNLP, pages 2873- 2878, 2017.","title":"David Mimno and Laure Thompson. The strange geometry of skip-gram with negative sampling","authors":"Thompson Mimno","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Press and Wolf, 2017] Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Pro- ceedings of EACL, volume 2, pages 157-163, 2017.","title":"Using the output embedding to improve language models","authors":"Wolf ; Press, Ofir Press, Lior Wolf","year":"2017","venue":"Proceedings of EACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radinsky et al., 2011] Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal seman- tic analysis. In Proceedings of the 20th international con- ference on World wide web, pages 337-346. ACM, 2011.","title":"A word at a time: computing word relatedness using temporal semantic analysis","authors":"Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, Shaul Markovitch","year":"2011","venue":"Proceedings of the 20th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wigner, 1955] Eugene P Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics, pages 548-564, 1955.","title":"Characteristic vectors of bordered matrices with infinite dimensions","authors":"Eugene P Wigner","year":"1955","venue":"Annals of Mathematics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wigner, 1958] Eugene P Wigner. On the distribution of the roots of certain symmetric matrices. Annals of Mathemat- ics, pages 325-327, 1958.","title":"On the distribution of the roots of certain symmetric matrices","authors":"Eugene P Wigner","year":"1958","venue":"Annals of Mathematics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zipf, 1935] GK Zipf. The psycho-biology of language: an introduction to dynamic philology. Houghton Mifflin, 1935.","title":"The psycho-biology of language: an introduction to dynamic philology","authors":"Zipf","year":"1935","venue":"GK Zipf","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}