Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?

de Varda, Andrea Gregor; Strapparava, Carlo

doi:10.24963/ijcai.2021/89

Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?

Andrea Gregor de Varda, Carlo Strapparava

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 643-649.

https://doi.org/10.24963/ijcai.2021/89

Keywords

Computer Vision: Language and Vision
Natural Language Processing: Psycholinguistics
Natural Language Processing: Phonology, Morphology, and Word Segmentation

Abstract

The present paper addresses the study of cross-linguistic and cross-modal iconicity within a deep learning framework. An LSTM-based Recurrent Neural Network is trained to associate the phonetic representation of a concrete word, encoded as a sequence of feature vectors, to the visual representation of its referent, expressed as an HCNN-transformed image. The processing network is then tested, without further training, in a language that does not appear in the training set and belongs to a different language family. The performance of the model is evaluated through a comparison with a randomized baseline; we show that such an imaginative network is capable of extracting language-independent generalizations in the mapping from linguistic sounds to visual features, providing empirical support for the hypothesis of a universal sound-symbolic substrate underlying all languages.

Andrea Gregor de Varda, and Carlo Strapparava. 2021. Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 643–649. https://doi.org/10.24963/ijcai.2021/89

@inproceedings{ijcai2021p89,
  title     = {Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?},
  author    = {de Varda, Andrea Gregor and Strapparava, Carlo},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {643--649},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/89},
  url       = {https://doi.org/10.24963/ijcai.2021/89},
}

TY  - CPAPER
TI  - Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?
AU  - de Varda, Andrea Gregor
AU  - Strapparava, Carlo
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 643
EP  - 649
DO  - 10.24963/ijcai.2021/89
UR  - https://doi.org/10.24963/ijcai.2021/89
ER  -