{"type":"ScholarlyArticle","title":"EViLBERT: Learning Task-Agnostic Multimodal Sense Embeddings","authors":[{"name":"Agostina Calabrese","slug":"agostina-calabrese","given":"Agostina","family":"Calabrese","affiliation":"Sapienza NLP Group , Department of Computer Science , Sapienza University of Rome"},{"name":"Michele Bevilacqua","slug":"michele-bevilacqua","given":"Michele","family":"Bevilacqua","affiliation":"Sapienza NLP Group , Department of Computer Science , Sapienza University of Rome"},{"name":"Roberto Navigli","slug":"roberto-navigli","given":"Roberto","family":"Navigli","affiliation":"Sapienza NLP Group , Department of Computer Science , Sapienza University of Rome"}],"abstract":"The problem of grounding language in vision is increasingly attracting scholarly efforts. As of now, however, most of the approaches have been limited to word embeddings, which are not capable of handling polysemous words. This is mainly due to the limited coverage of the available semantically-annotated datasets, hence forcing research to rely on alternative technologies (i.e., image search engines). To address this issue, we introduce EViLBERT, an approach which is able to perform image classification over an open set of concepts, both concrete and non-concrete. Our approach is based on the recently introduced Vision-Language Pretraining (VLP) model, and builds upon a manually-annotated dataset of concept-image pairs. We use our technique to clean up the image-to-concept mapping that is provided within a multilingual knowledge base, resulting in over 258,000 images associated with 42,500 concepts. \r\nWe show that our VLP-based model can be used to create multimodal sense embeddings starting from our automatically-created dataset. In turn, we also show that these multimodal embeddings improve the performance of a Word Sense Disambiguation architecture over a strong unimodal baseline. We release code, dataset and embeddings at http://babelpic.org.","doi":"10.24963/ijcai.2020/67","url":"https://ijcai.org/proceedings/2020/67","pdf":"https://ijcai.org/proceedings/2020/0067.pdf","markdown":"https://ijcai.org/proceedings/2020/67.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/67","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":67,"pages":{"first":481,"last":487},"keywords":["Computer Vision: Language and Vision","Natural Language Processing: Natural Language Semantics","Natural Language Processing: Embeddings"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25190/file_25190.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proc. of CVPR, pages 6077- 6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barsalou and Wiemer-Hastings, 2005] Lawrence W Barsalou and Katja Wiemer-Hastings. Situating abstract concepts. Grounding Cognition: The Role of Perception and Action in Memory, Lan- guage, and Thought, pages 129-163, 2005.","title":"Situating abstract concepts","authors":"W Lawrence, Katja Barsalou, Wiemer-Hastings","year":"2005","venue":"Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thought","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bevilacqua and Navigli, 2020] Michele Bevilacqua and Roberto Navigli. Breaking through the 80% glass ceiling: Raising the state of the art in Word Sense Disambiguation by incorporating knowledge graph information. In Proc. of ACL, 2020.","title":"Breaking through the 80% glass ceiling: Raising the state of the art in Word Sense Disambiguation by incorporating knowledge graph information","authors":"Michele Bevilacqua, Roberto Navigli","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Calabrese et al., 2020] Agostina Calabrese, Michele Bevilacqua, and Roberto Navigli. Fatality killed the cat or: BabelPic, a multi- modal dataset for non-concrete concepts. In Proc. of ACL, 2020.","title":"Fatality killed the cat or: BabelPic, a multimodal dataset for non-concrete concepts","authors":"Agostina Calabrese, Michele Bevilacqua, Roberto Navigli","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Collell and Moens, 2018] Guillem Collell and Marie-Francine Moens. Do neural network cross-modal mappings really bridge modalities? In Proc. of ACL, pages 462-468, 2018.","title":"Do neural network cross-modal mappings really bridge modalities?","authors":"Moens Collell, Guillem Collell, Marie-Francine Moens","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In Proc. of CVPR, pages 248-255, 2009.","title":"ImageNet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proc. of NAACL- HLT, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional Transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proc. of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frome et al., 2013] Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. DeViSE: A deep visual-semantic embedding model. In Proc. of NeurIPS, pages 2121-2129, 2013.","title":"DeViSE: A deep visual-semantic embedding model","authors":"Andrea Frome, Gregory S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'aurelio Ranzato, Tomas Mikolov","year":"2013","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA mat- ter: Elevating the role of image understanding in visual question answering. In Proc. of CVPR, pages 6325-6334, 2017.","title":"Making the V in VQA matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hill and Korhonen, 2014] Felix Hill and Anna Korhonen. Learn- ing abstract concept embeddings from multi-modal data: Since you probably can't see what I mean. In Proc. of EMNLP, pages 255-265, 2014.","title":"Learning abstract concept embeddings from multi-modal data: Since you probably can't see what I mean","authors":"Korhonen ; Hill, Felix Hill, Anna Korhonen","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2019] Luyao Huang, Chi Sun, Xipeng Qiu, and Xu- anjing Huang. GlossBERT: BERT for Word Sense Disambigua- tion with gloss knowledge. In Proc. of EMNLP-IJCNLP, pages 3507-3512, 2019.","title":"GlossBERT: BERT for Word Sense Disambiguation with gloss knowledge","authors":"Huang","year":"2019","venue":"Proc. of EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kiela et al., 2014] Douwe Kiela, Felix Hill, Anna Korhonen, and Stephen Clark. Improving multi-modal representations using im- age dispersion: Why less is sometimes more. In Proc. of ACL, pages 835-841, 2014.","title":"Improving multi-modal representations using image dispersion: Why less is sometimes more","authors":"Douwe Kiela, Felix Hill, Anna Korhonen, Stephen Clark","year":"2014","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiros et al., 2018] Jamie Ryan Kiros, William Chan, and Geof- frey E. Hinton. Illustrative language understanding: Large-scale visual grounding with image search. In Proc. of ACL, pages 922- 933, 2018.","title":"Illustrative language understanding: Large-scale visual grounding with image search","authors":"Jamie Ryan Kiros, William Chan, Geoffrey E Hinton","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bern- stein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Interna- tional Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual Genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuznetsova et al., 2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The Open Images dataset V4. International Journal of Computer Vision, 2020.","title":"The Open Images dataset V4","authors":"Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, Vittorio Ferrari","year":"2020","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A simple and perfor- mant baseline for vision and language. CoRR, abs/1908.03557, 2019.","title":"VisualBERT: A simple and performant baseline for vision and language","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in con- text. In Proc. of ECCV, pages 740-755, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loureiro and Jorge, 2019] Daniel Loureiro and Alípio Jorge. Lan- guage modelling makes sense: Propagating representations through WordNet for full-coverage Word Sense Disambiguation. In Proc. of ACL, pages 5682-5691, 2019.","title":"Daniel Loureiro and Alípio Jorge. Language modelling makes sense: Propagating representations through WordNet for full-coverage Word Sense Disambiguation","authors":"Jorge Loureiro","year":"2019","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic repre- sentations for vision-and-language tasks. In Proc. of NeurIPS, pages 13-23, 2019.","title":"ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Proc. of ICLR, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Miller, 1995] George A. Miller. WordNet: A lexical database for English. Commun. ACM, 38(11):39-41, 1995.","title":"WordNet: A lexical database for English","authors":"George A Miller","year":"1995","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moro and Navigli, 2015] Andrea Moro and Roberto Navigli. SemEval-2015 task 13: Multilingual all-words sense disam- biguation and entity linking. In Proc. of SemEval@NAACL-HLT, pages 288-297, 2015.","title":"SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking","authors":"Andrea Moro, Roberto Navigli","year":"2015","venue":"Proc. of SemEval@NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Navigli and Ponzetto, 2012] Roberto Navigli and Simone Paolo Ponzetto. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell., 193:217-250, 2012.","title":"BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network","authors":"Roberto Navigli, Simone Paolo, Ponzetto","year":"2012","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Navigli, 2009] Roberto Navigli. Word Sense Disambiguation: A Survey. ACM Computing Surveys, 41(2):1-69, 2009.","title":"Word Sense Disambiguation: A Survey","authors":"Roberto Navigli","year":"2009","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Plummer et al., 2015] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting region-to-phrase cor- respondences for richer image-to-sentence models. In Proc. of ICCV, pages 2641-2649, 2015.","title":"Flickr30k Entities: Collecting region-to-phrase correspondences for richer image-to-sentence models","authors":"Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, Svetlana Lazebnik","year":"2015","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raganato et al., 2017] Alessandro Raganato, José Camacho- Collados, and Roberto Navigli. Word Sense Disambiguation: A unified evaluation framework and empirical comparison. In Proc. of EACL, pages 99-110, 2017.","title":"Word Sense Disambiguation: A unified evaluation framework and empirical comparison","authors":"Alessandro Raganato, José Camacho-Collados, Roberto Navigli","year":"2017","venue":"Proc. of EACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proc. of NeurIPS, pages 91- 99, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silberer and Lapata, 2014] Carina Silberer and Mirella Lapata. Learning grounded meaning representations with autoencoders. In Proc. of ACL, pages 721-732, 2014.","title":"Learning grounded meaning representations with autoencoders","authors":"Lapata Silberer, Carina Silberer, Mirella Lapata","year":"2014","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tan and Bansal, 2019] Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from Trans- formers. In Proc. of EMNLP-IJCNLP, pages 5099-5110, 2019.","title":"LXMERT: Learning cross-modality encoder representations from Transformers","authors":"Hao Tan, Mohit Bansal","year":"2019","venue":"Proc. of EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2020] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and VQA. In Proc. of AAAI, 2020.","title":"Unified visionlanguage pre-training for image captioning and VQA","authors":"Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, Jianfeng Gao","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}