{"type":"ScholarlyArticle","title":"MGAD: Learning Descriptional Representation Distilled from Distributional Semantics for Unseen Entities","authors":[{"name":"Yuanzheng Wang","slug":"yuanzheng-wang","given":"Yuanzheng","family":"Wang","affiliation":"CAS Key Lab of Network Data Science and Technology , ICT , CAS , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Xueqi Cheng","slug":"xueqi-cheng","given":"Xueqi","family":"Cheng","affiliation":"CAS Key Lab of Network Data Science and Technology , ICT , CAS , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yixing Fan","slug":"yixing-fan","given":"Yixing","family":"Fan","affiliation":"CAS Key Lab of Network Data Science and Technology , ICT , CAS , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Xiaofei Zhu","slug":"xiaofei-zhu","given":"Xiaofei","family":"Zhu","affiliation":"College of Computer Science and Engineering , Chongqing University of Technology"},{"name":"Huasheng Liang","slug":"huasheng-liang","given":"Huasheng","family":"Liang","affiliation":"WeChat , Tencent , Guangzhou , China"},{"name":"Qiang Yan","slug":"qiang-yan","given":"Qiang","family":"Yan","affiliation":"WeChat , Tencent , Guangzhou , China"},{"name":"Jiafeng Guo","slug":"jiafeng-guo","given":"Jiafeng","family":"Guo","affiliation":"CAS Key Lab of Network Data Science and Technology , ICT , CAS , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"Entity representation plays a central role in building effective entity retrieval models. Recent works propose to learn entity representations based on entity-centric contexts, which achieve SOTA performances on many tasks. However, these methods lead to poor representations for unseen entities since they rely on a multitude of occurrences for each entity to enable accurate representations. To address this issue, we propose to learn enhanced descriptional representations for unseen entities by distilling knowledge from distributional semantics into descriptional embeddings. Specifically, we infer enhanced embeddings for unseen entities based on descriptions by aligning the descriptional embedding space to the distributional embedding space with different granularities, i.e., element-level, batch-level and space-level alignment. Experimental results on four benchmark datasets show that our approach improves the performance over all baseline methods. In particular, our approach can achieve the effectiveness of the teacher model on almost all entities, and maintain such high performance on unseen entities.","doi":"10.24963/ijcai.2022/611","url":"https://ijcai.org/proceedings/2022/611","pdf":"https://ijcai.org/proceedings/2022/0611.pdf","markdown":"https://ijcai.org/proceedings/2022/611.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/611","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":611,"pages":{"first":4404,"last":4410},"keywords":["Natural Language Processing: Named Entities","Natural Language Processing: Coreference Resolution","Natural Language Processing: Embeddings","Natural Language Processing: Information Retrieval and Text Mining","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987080","length_sec":153,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985223","length_sec":591,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bevilacqua et al., 2020] Michele Bevilacqua, Marco Maru, and Roberto Navigli. Generationary or \"how we went beyond word sense inventories and learned to gloss\". In EMNLP, 2020.","title":"Generationary or \"how we went beyond word sense inventories and learned to gloss","authors":"Michele Bevilacqua, Marco Maru, Roberto Navigli","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bosc and Vincent, 2018] Tom Bosc and Pascal Vincent. Auto-encoding dictionary definitions into consistent word embeddings. In EMNLP, 2018.","title":"Auto-encoding dictionary definitions into consistent word embeddings","authors":"Vincent ; Bosc, Tom Bosc, Pascal Vincent","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Botha et al., 2020] Jan A. Botha, Zifei Shan, and Daniel Gillick. Entity Linking in 100 Languages. In EMNLP, 2020.","title":"Entity Linking in 100 Languages","authors":"Jan A Botha, Zifei Shan, Daniel Gillick","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2021] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In ICLR, 2021.","title":"Autoregressive entity retrieval","authors":"Nicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio Petroni","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cucerzan, 2007] Silviu Cucerzan. Large-scale named en- tity disambiguation based on Wikipedia data. In EMNLP- CoNLL, 2007.","title":"Large-scale named entity disambiguation based on Wikipedia data","authors":"Silviu Cucerzan","year":"2007","venue":"EMNLP-CoNLL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Firth, 1957] John R Firth. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis, 1957.","title":"A synopsis of linguistic theory, 1930-1955","authors":"John R Firth","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganea and Hofmann, 2017] Octavian-Eugen Ganea and Thomas Hofmann. Deep joint entity disambiguation with local neural attention. In EMNLP, 2017.","title":"Deep joint entity disambiguation with local neural attention","authors":"Hofmann ; Octavian-Eugen Ganea, Thomas Ganea, Hofmann","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gillick et al., 2019] Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. Learning dense representations for entity retrieval. In CoNLL, 2019.","title":"Learning dense representations for entity retrieval","authors":"Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, Diego Garcia-Olano","year":"2019","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gou et al., 2020] Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. CoRR, abs/2006.05525, 2020.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gupta et al., 2017] Nitish Gupta, Sameer Singh, and Dan Roth. Entity linking via joint encoding of types, descrip- tions, and context. In EMNLP, 2017.","title":"Entity linking via joint encoding of types, descriptions, and context","authors":"Nitish Gupta, Sameer Singh, Dan Roth","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Harris, 1954] Zellig S Harris. Distributional structure. Word, 1954.","title":"Distributional structure","authors":"S Zellig, Harris","year":"1954","venue":"Word","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hill et al., 2016] Felix Hill, Kyunghyun Cho, Anna Korho- nen, and Yoshua Bengio. Learning to understand phrases by embedding the dictionary. TACL, 2016.","title":"Learning to understand phrases by embedding the dictionary","authors":"Felix Hill, Kyunghyun Cho, Anna Korhonen, Yoshua Bengio","year":"2016","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoffart et al., 2011] Johannes Hoffart, Mohamed A. Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In EMNLP, 2011.","title":"Robust disambiguation of named entities in text","authors":"Johannes Hoffart, Mohamed A Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, Gerhard Weikum","year":"2011","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR (Poster)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le and Titov, 2019] Phong Le and Ivan Titov. Distant learn- ing for entity linking with automatic noise detection. In ACL, 2019.","title":"Distant learning for entity linking with automatic noise detection","authors":"Le, Titov ;, Phong Le, Ivan Titov","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Logeswaran et al., 2019] Lajanugen Logeswaran, Ming- Wei Chang, Kenton Lee, Kristina Toutanova, Jacob De- vlin, and Honglak Lee. Zero-shot entity linking by reading entity descriptions. In ACL, 2019.","title":"Zero-shot entity linking by reading entity descriptions","authors":"Logeswaran","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Milne and Witten, 2008] David Milne and Ian H. Witten. Learning to link with wikipedia. In CIKM, 2008.","title":"Learning to link with wikipedia","authors":"David Milne, Ian H Witten","year":"2008","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nooralahzadeh and Øvrelid, 2018] Farhad Nooralahzadeh and Lilja Øvrelid. SIRIUS-LTG: An entity linking ap- proach to fact extraction and verification. In FEVER, 2018.","title":"SIRIUS-LTG: An entity linking approach to fact extraction and verification","authors":"Farhad Nooralahzadeh, Lilja Øvrelid","year":"2018","venue":"FEVER","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Onoe et al., 2021] Yasumasa Onoe, Michael Boratko, An- drew McCallum, and Greg Durrett. Modeling fine-grained entity types with box embeddings. In ACL/IJCNLP, 2021.","title":"Modeling fine-grained entity types with box embeddings","authors":"Onoe","year":"2021","venue":"ACL/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Prokhorov et al., 2019] Victor Prokhorov, Mohammad T. Pilehvar, Dimitri Kartsaklis, Pietro Lio, and Nigel Col- lier. Unseen word representation by aligning heteroge- neous lexical semantic spaces. In AAAI, 2019.","title":"Unseen word representation by aligning heterogeneous lexical semantic spaces","authors":"Victor Prokhorov, Mohammad T Pilehvar, Dimitri Kartsaklis, Pietro Lio, Nigel Collier","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ratinov et al., 2011] Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. Local and global al- gorithms for disambiguation to Wikipedia. In ACL, 2011.","title":"Local and global algorithms for disambiguation to Wikipedia","authors":"Ratinov","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van Hulst et al., 2020] Johannes M. van Hulst, Faegheh Ha- sibi, Koen Dercksen, Krisztian Balog, and Arjen P. de Vries. Rel: An entity linker standing on the shoulders of giants. In SIGIR, 2020.","title":"Rel: An entity linker standing on the shoulders of giants","authors":"Johannes M Van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog, Arjen P De Vries","year":"2020","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021] Yixiao Wang, Zied Bouraoui, Luis Es- pinosa Anke, and Steven Schockaert. Deriving word vec- tors from contextualized language models using topic- aware mention selection. In RepL4NLP, 2021.","title":"Deriving word vectors from contextualized language models using topicaware mention selection","authors":"Wang","year":"2021","venue":"RepL4NLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2020a] Junshuang Wu, Richong Zhang, Yongyi Mao, Hongyu Guo, Masoumeh Soflaei, and Jinpeng Huai. Dynamic graph convolutional networks for entity linking. In WWW, 2020.","title":"Dynamic graph convolutional networks for entity linking","authors":"Wu","year":"2020","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2020b] Ledell Wu, Fabio Petroni, Martin Josi- foski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, 2020.","title":"Scalable zero-shot entity linking with dense entity retrieval","authors":"Wu","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yamada and Shindo, 2019] Ikuya Yamada and Hiroyuki Shindo. Pre-training of deep contextualized embeddings of words and entities for named entity disambiguation. CoRR, abs/1909.00426, 2019.","title":"Pre-training of deep contextualized embeddings of words and entities for named entity disambiguation","authors":"Shindo Yamada, Ikuya Yamada, Hiroyuki Shindo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yamada et al., 2020] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. LUKE: Deep contextualized entity representations with entity- aware self-attention. In EMNLP, 2020.","title":"LUKE: Deep contextualized entity representations with entityaware self-attention","authors":"Yamada","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zock and Schwab, 2008] Michael Zock and Didier Schwab. Lexical access based on underspecified input. In COLING, 2008.","title":"Lexical access based on underspecified input","authors":"Michael Zock, Didier Schwab","year":"2008","venue":"COLING","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}