{"type":"ScholarlyArticle","title":"Graph-based Dynamic Word Embeddings","authors":[{"name":"Yuyin Lu","slug":"yuyin-lu","given":"Yuyin","family":"Lu","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China"},{"name":"Xin Cheng","slug":"xin-cheng","given":"Xin","family":"Cheng","affiliation":"School of Mathematical Sciences , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Ziran Liang","slug":"ziran-liang","given":"Ziran","family":"Liang","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China"},{"name":"Yanghui Rao","slug":"yanghui-rao","given":"Yanghui","family":"Rao","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China"}],"abstract":"As time goes by, language evolves with word semantics changing. Unfortunately, traditional word embedding methods neglect the evolution of language and assume that word representations are static. Although contextualized word embedding models can capture the diverse representations of polysemous words, they ignore temporal information as well. To tackle the aforementioned challenges, we propose a graph-based dynamic word embedding (GDWE) model, which focuses on capturing the semantic drift of words continually. We introduce word-level knowledge graphs (WKGs) to store short-term and long-term knowledge. WKGs can provide rich structural information as supplement of lexical information, which help enhance the word embedding quality and capture semantic drift quickly. Theoretical analysis and extensive experiments validate the effectiveness of our GDWE on dynamic word embedding learning.","doi":"10.24963/ijcai.2022/594","url":"https://ijcai.org/proceedings/2022/594","pdf":"https://ijcai.org/proceedings/2022/0594.pdf","markdown":"https://ijcai.org/proceedings/2022/594.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/594","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":594,"pages":{"first":4280,"last":4288},"keywords":["Machine Learning: Time-series; Data Streams","Natural Language Processing: Embeddings","Machine Learning: Online Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986697","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984840","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bamler and Mandt, 2017] Robert Bamler and Stephan Mandt. Dynamic word embeddings. In ICML, pages 380-389, 2017.","title":"Dynamic word embeddings","authors":"Mandt Bamler, Robert Bamler, Stephan Mandt","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Basile et al., 2020] Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Var- vara. Diacr-ita @ EVALITA2020: overview of the EVALITA2020 diachronic lexical semantics (diacr-ita) task. In EVALITA, 2020.","title":"Diacr-ita @ EVALITA2020: overview of the EVALITA2020 diachronic lexical semantics (diacr-ita) task","authors":"Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, Rossella Varvara","year":"2020","venue":"EVALITA","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dhingra et al., 2022] Bhuwan Dhingra, Jeremy R Cole, Ju- lian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-aware language models as temporal knowledge bases. TACL, 10:257-273, 2022.","title":"Time-aware language models as temporal knowledge bases","authors":"Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W Cohen","year":"2022","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Giulianelli et al., 2020] Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. Analysing lexical se- mantic change with contextualised word representations. In ACL, pages 3960-3973, 2020.","title":"Analysing lexical semantic change with contextualised word representations","authors":"Mario Giulianelli, Marco Del Tredici, Raquel Fernández","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gong et al., 2020] Hongyu Gong, Suma Bhat, and Pramod Viswanath. Enriching word embeddings with temporal and spatial information. In CoNLL, pages 1-11, 2020.","title":"Enriching word embeddings with temporal and spatial information","authors":"Hongyu Gong, Suma Bhat, Pramod Viswanath","year":"2020","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hamilton et al., 2016] William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statistical laws of semantic change. In ACL, pages 1489- 1501, 2016.","title":"Diachronic word embeddings reveal statistical laws of semantic change","authors":"William L Hamilton, Jure Leskovec, Dan Jurafsky","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hofmann et al., 2021] Valentin Hofmann, Janet B. Pierre- humbert, and Hinrich Schütze. Dynamic contextualized word embeddings. In ACL-IJCNLP, pages 6970-6984, 2021.","title":"Dynamic contextualized word embeddings","authors":"Valentin Hofmann, Janet B Pierrehumbert, Hinrich Schütze","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2019] Renfen Hu, Shen Li, and Shichen Liang. Diachronic sense modeling with deep contextualized word embeddings: An ecological view. In ACL, pages 3899- 3908, 2019.","title":"Diachronic sense modeling with deep contextualized word embeddings: An ecological view","authors":"Hu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kaji and Kobayashi, 2017] Nobuhiro Kaji and Hayato Kobayashi. Incremental skip-gram model with negative sampling. In EMNLP, pages 363-371, 2017.","title":"Incremental skip-gram model with negative sampling","authors":"Kobayashi Kaji, Nobuhiro Kaji, Hayato Kobayashi","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2014] Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. Temporal analysis of lan- guage through neural language models. In LTCSS@ACL, pages 61-65, 2014.","title":"Temporal analysis of language through neural language models","authors":"Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, Slav Petrov","year":"2014","venue":"LTCSS@ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulkarni et al., 2015] Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. Statistically significant detection of linguistic change. In WWW, pages 625-635, 2015.","title":"Statistically significant detection of linguistic change","authors":"Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, Steven Skiena","year":"2015","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kutuzov and Pivovarova, 2021] Andrey Kutuzov and Lidia Pivovarova. Rushifteval: A shared task on semantic shift detection for russian. In Dialogue, 2021.","title":"Rushifteval: A shared task on semantic shift detection for russian","authors":"Pivovarova Kutuzov, Andrey Kutuzov, Lidia Pivovarova","year":"2021","venue":"Dialogue","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kutuzov et al., 2018] Andrey Kutuzov, Lilja Øvrelid, Ter- rence Szymanski, and Erik Velldal. Diachronic word em- beddings and semantic shifts: A survey. In COLING, pages 1384-1397, 2018.","title":"Diachronic word embeddings and semantic shifts: A survey","authors":"Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, Erik Velldal","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lazaridou et al., 2021] Angeliki Lazaridou, Adhiguna Kun- coro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Mas- son d'Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yo- gatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the gap: Assessing temporal generalization in neural language models. In NeurIPS, pages 29348-29363, 2021.","title":"Mind the gap: Assessing temporal generalization in neural language models","authors":"Lazaridou","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NIPS, pages 2177-2185, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McMahon, 1994] M. S. McMahon. Understanding language change. Cambridge University Press, 1994.","title":"Understanding language change","authors":"M S Mcmahon","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013a] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR (Workshop Poster), 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":"ICLR (Workshop Poster)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013b] Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their composi- tionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rosin et al., 2022] Guy D Rosin, Ido Guy, and Kira Radin- sky. Time masking for temporal language models. In WSDM, pages 833-841, 2022.","title":"Time masking for temporal language models","authors":"Ido Guy D Rosin, Kira Guy, Radinsky","year":"2022","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saeed et al., 2019] Zafar Saeed, Rabeeh Ayaz Abbasi, Muhammad Imran Razzak, and Guandong Xu. Event de- tection in twitter stream using weighted dynamic heartbeat graph approach [application notes]. IEEE Computational Intelligence Magazine, 14(3):29-38, 2019.","title":"Event detection in twitter stream using weighted dynamic heartbeat graph approach [application notes","authors":"Zafar Saeed, Rabeeh Ayaz Abbasi, Muhammad Imran Razzak, Guandong Xu","year":"2019","venue":"IEEE Computational Intelligence Magazine","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schlechtweg et al., 2020] Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. Semeval-2020 task 1: Unsupervised lex- ical semantic change detection. In SemEval@COLING, pages 1-23, 2020.","title":"Semeval-2020 task 1: Unsupervised lexical semantic change detection","authors":"Dominik Schlechtweg, Barbara Mcgillivray, Simon Hengchen, Haim Dubossarsky, Nina Tahmasebi","year":"2020","venue":"SemEval@COLING","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Spitz and Gertz, 2018] Andreas Spitz and Michael Gertz. Exploring entity-centric networks in entangled news streams. In WWW (Companion Volume), pages 555-563, 2018.","title":"Exploring entity-centric networks in entangled news streams","authors":"Gertz Spitz, Andreas Spitz, Michael Gertz","year":"2018","venue":"WWW (Companion","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J. Smola, and Eduard H. Hovy. Hi- erarchical attention networks for document classification. In NAACL-HLT, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2018] Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. Dynamic word embeddings for evolving semantic discovery. In WSDM, pages 673- 681, 2018.","title":"Dynamic word embeddings for evolving semantic discovery","authors":"Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, Hui Xiong","year":"2018","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zuckerman and Last, 2019] Matan Zuckerman and Mark Last. Using graphs for word embedding with enhanced semantic relations. In TextGraphs@EMNLP, pages 32-41, 2019.","title":"Using graphs for word embedding with enhanced semantic relations","authors":"Matan Zuckerman, Mark Last","year":"2019","venue":"TextGraphs@EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}