{"type":"ScholarlyArticle","title":"Incorporating Extra Knowledge to Enhance Word Embedding","authors":[{"name":"Arpita Roy","slug":"arpita-roy","given":"Arpita","family":"Roy","affiliation":"University of Maryland , Baltimore County"},{"name":"Shimei Pan","slug":"shimei-pan","given":"Shimei","family":"Pan","affiliation":"University of Maryland , Baltimore County; (UMBC)"}],"abstract":"Word embedding, a process to automatically learn the mathematical representations of words from unlabeled text corpora, has gained a lot of attention recently.  Since words are the basic units of a natural language, the more precisely we can represent the morphological, syntactic and semantic properties of words, the better we can support downstream Natural Language Processing (NLP) tasks. Since traditional word embeddings are mainly designed to capture the semantic relatedness between co-occurred words in a predefined context, it may not be effective in encoding other information that is important for different NLP applications. In this survey, we summarize the recent advances in incorporating extra knowledge to enhance word embedding. We will also identify the limitations of existing work as well as point out a few promising future directions.","doi":"10.24963/ijcai.2020/686","url":"https://ijcai.org/proceedings/2020/686","pdf":"https://ijcai.org/proceedings/2020/0686.pdf","markdown":"https://ijcai.org/proceedings/2020/686.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/686","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Survey track","track":"Survey track","number":686,"pages":{"first":4929,"last":4935},"keywords":["Machine Learning: general","Natural Language Processing: general"],"videos":[],"references":[{"position":1,"raw":"[Bian et al., 2014] Jiang Bian, Bin Gao, and Tie-Yan Liu. Knowledge-powered deep learning for word embedding. In ECML PKDD. Springer, 2014.","title":"Knowledge-powered deep learning for word embedding","authors":"Bin Jiang Bian, Tie-Yan Gao, Liu","year":"2014","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bodenreider, 2004] Olivier Bodenreider. The unified medi- cal language system (umls): integrating biomedical termi- nology. Nucleic acids research, 32(suppl 1):D267-D270, 2004.","title":"The unified medical language system (umls): integrating biomedical terminology","authors":"Olivier Bodenreider, Bodenreider","year":"2004","venue":"Nucleic acids research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolukbasi et al., 2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. In NIPS, 2016.","title":"Man is to computer programmer as woman is to homemaker? debiasing word embeddings","authors":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, Adam T Kalai","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2017] Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. Bridge text and knowledge by learning multi-prototype entity mention embedding. In ACL, 2017.","title":"Bridge text and knowledge by learning multi-prototype entity mention embedding","authors":"Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, Juanzi Li","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2015] Jianpeng Cheng, Zhongyuan Wang, Ji- Rong Wen, Jun Yan, and Zheng Chen. Contextual text understanding in distributional semantic space. In CIKM, 2015.","title":"Contextual text understanding in distributional semantic space","authors":"Cheng","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2016] Wei Fang, Jianwen Zhang, Dilin Wang, Zheng Chen, and Ming Li. Entity disambiguation by knowledge and text jointly embedding. In CoNLL, 2016.","title":"Entity disambiguation by knowledge and text jointly embedding","authors":"Wei Fang, Jianwen Zhang, Dilin Wang, Zheng Chen, Ming Li","year":"2016","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Faruqui et al., 2015] Manaal Faruqui, Jesse Dodge, Su- jay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting word vectors to semantic lexicons. In ACL, 2015.","title":"Retrofitting word vectors to semantic lexicons","authors":"Faruqui","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glavaš and Vulić, 2018] Goran Glavaš and Ivan Vulić. Ex- plicit retrofitting of distributional word vectors. In ACL, 2018.","title":"Explicit retrofitting of distributional word vectors","authors":"Vulić Glavaš, Goran Glavaš, Ivan Vulić","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jauhar et al., 2015] Sujay Kumar Jauhar, Chris Dyer, and Eduard Hovy. Ontologically grounded multi-sense repre- sentation learning for semantic vector space models. In NAACL, 2015.","title":"Ontologically grounded multi-sense representation learning for semantic vector space models","authors":"Sujay Kumar, Jauhar, Chris Dyer, Eduard Hovy","year":"2015","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2018] Zhiwei Jiang, Qing Gu, Yafeng Yin, and Daoxu Chen. Enriching word embeddings with domain knowledge for readability assessment. In COLING, 2018.","title":"Enriching word embeddings with domain knowledge for readability assessment","authors":"Zhiwei Jiang, Qing Gu, Yafeng Yin, Daoxu Chen","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirillov et al., 2011] Ivan Kirillov, Desiree Beck, Penny Chase, and Robert Martin. Malware attribute enumeration and characterization, 2011.","title":"Malware attribute enumeration and characterization","authors":"Ivan Kirillov, Desiree Beck, Penny Chase, Robert Martin","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ling et al., 2017] Yuan Ling, Yuan An, Mengwen Liu, Sa- did A Hasan, Yetian Fan, and Xiaohua Hu. Integrating extra knowledge into word embedding models for biomed- ical nlp tasks. In IJCNN, 2017.","title":"Integrating extra knowledge into word embedding models for biomedical nlp tasks","authors":"Yuan Ling, Yuan An, Mengwen Liu, A Sadid, Yetian Hasan, Xiaohua Fan, Hu","year":"2017","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. Learning semantic word embeddings based on ordinal knowledge constraints. In ACL-IJCNLP, 2015.","title":"Learning semantic word embeddings based on ordinal knowledge constraints","authors":"Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, Yu Hu","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Qian Liu, Heyan Huang, Guangquan Zhang, Yang Gao, Junyu Xuan, and Jie Lu. Semantic structure-based word embedding by incorporating concept convergence and word divergence. In AAAI, 2018.","title":"Semantic structure-based word embedding by incorporating concept convergence and word divergence","authors":"Qian Liu, Heyan Huang, Guangquan Zhang, Yang Gao, Junyu Xuan, Jie Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luong et al., 2013] Minh-Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, 2013.","title":"Better word representations with recursive neural networks for morphology","authors":"Minh-Thang Luong, Richard Socher, Christopher D Manning","year":"2013","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mancini et al., 2017] Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. Embedding words and senses together via joint knowledge-enhanced training. In CoNLL, 2017.","title":"Embedding words and senses together via joint knowledge-enhanced training","authors":"Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, Roberto Navigli","year":"2017","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. CoRR, abs/1802.05365, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peters et al., 2019] Matthew E. Peters, Mark Neumann, IV RobertLLogan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contex- tual word representations. In EMNLP-IJCNLP, 2019.","title":"Knowledge enhanced contextual word representations","authors":"Matthew E Peters, Mark Neumann, Roy Iv Robertllogan, Vidur Schwartz, Sameer Joshi, Noah A Singh, Smith","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roy et al., 2019] Arpita Roy, Youngja Park, and Shimei Pan. Predicting malware attributes from cybersecurity texts. In NAACL, 2019.","title":"Predicting malware attributes from cybersecurity texts","authors":"Arpita Roy, Youngja Park, Shimei Pan","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2014] Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. Learning sentiment- specific word embedding for twitter sentiment classifica- tion. In ACL, 2014.","title":"Learning sentimentspecific word embedding for twitter sentiment classification","authors":"Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, Bing Qin","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tissier et al., 2017] Julien Tissier, Christophe Gravier, and Amaury Habrard. Dict2vec : Learning word embeddings using lexical dictionaries. In EMNLP, 2017.","title":"Dict2vec : Learning word embeddings using lexical dictionaries","authors":"Julien Tissier, Christophe Gravier, Amaury Habrard","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph and text jointly embedding. In EMNLP, 2014.","title":"Knowledge graph and text jointly embedding","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2012] Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Q Zhu. Probase: A probabilistic taxonomy for text understanding. In Proceedings of the 2012 ACM SIG- MOD International Conference on Management of Data, pages 481-492, 2012.","title":"Probase: A probabilistic taxonomy for text understanding","authors":"Wentao Wu, Hongsong Li, Haixun Wang, Kenny Q Zhu","year":"2012","venue":"Proceedings of the 2012 ACM SIG-MOD International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2014] Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, and Tie-Yan Liu. Rc- net: A general framework for incorporating knowledge into word representations. In CIKM, 2014.","title":"Rcnet: A general framework for incorporating knowledge into word representations","authors":"Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, Tie-Yan Liu","year":"2014","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeng et al., 2017] Ziqian Zeng, Yichun Yin, Yangqiu Song, and Ming Zhang. Socialized word embeddings. In IJCAI, 2017.","title":"Socialized word embeddings","authors":"Zeng","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/547"},{"position":28,"raw":"[Zhang et al., 2019] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In ACL, 2019.","title":"ERNIE: Enhanced language representation with informative entities","authors":"Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2015] Guangyou Zhou, Tingting He, Jun Zhao, and Po Hu. Learning continuous word embedding with metadata for question retrieval in community question an- swering. In ACL-IJCNLP, 2015.","title":"Learning continuous word embedding with metadata for question retrieval in community question answering","authors":"Zhou","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}