Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings

Li, Na; Bouraoui, Zied; Camacho-Collados, Jose; Espinosa-Anke, Luis; Gu, Qing; Schockaert, Steven

doi:10.24963/ijcai.2021/530

Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings

Na Li, Zied Bouraoui, Jose Camacho-Collados, Luis Espinosa-Anke, Qing Gu, Steven Schockaert

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 3850-3856.

https://doi.org/10.24963/ijcai.2021/530

Keywords

Natural Language Processing: Natural Language Semantics
Natural Language Processing: Natural Language Processing

Abstract

While the success of pre-trained language models has largely eliminated the need for high-quality static word vectors in many NLP applications, static word vectors continue to play an important role in tasks where word meaning needs to be modelled in the absence of linguistic context. In this paper, we explore how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors for such domains, in particular related to knowledge base completion, where our focus is on capturing the semantic properties of nouns. We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors learned by BERT, as well as those from standard word embedding models, in property induction tasks. We notice in particular that masking target words is critical to achieve this strong performance, as the resulting vectors focus less on idiosyncratic properties and more on general semantic properties. Inspired by this view, we propose a filtering strategy which is aimed at removing the most idiosyncratic mention vectors, allowing us to obtain further performance gains in property induction.

Na Li, Zied Bouraoui, Jose Camacho-Collados, Luis Espinosa-Anke, Qing Gu, and Steven Schockaert. 2021. Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 3850–3856. https://doi.org/10.24963/ijcai.2021/530

@inproceedings{ijcai2021p530,
  title     = {Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings},
  author    = {Li, Na and Bouraoui, Zied and Camacho-Collados, Jose and Espinosa-Anke, Luis and Gu, Qing and Schockaert, Steven},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {3850--3856},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/530},
  url       = {https://doi.org/10.24963/ijcai.2021/530},
}

TY  - CPAPER
TI  - Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
AU  - Li, Na
AU  - Bouraoui, Zied
AU  - Camacho-Collados, Jose
AU  - Espinosa-Anke, Luis
AU  - Gu, Qing
AU  - Schockaert, Steven
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 3850
EP  - 3856
DO  - 10.24963/ijcai.2021/530
UR  - https://doi.org/10.24963/ijcai.2021/530
ER  -