{"type":"ScholarlyArticle","title":"Compositionality Decomposed: How do Neural Networks Generalise? (Extended Abstract)","authors":[{"name":"Dieuwke Hupkes","slug":"dieuwke-hupkes","given":"Dieuwke","family":"Hupkes","affiliation":"ILLC , University of Amsterdam"},{"name":"Verna Dankers","slug":"verna-dankers","given":"Verna","family":"Dankers","affiliation":"University of Amsterdam"},{"name":"Mathijs Mul","slug":"mathijs-mul","given":"Mathijs","family":"Mul","affiliation":"University of Amsterdam"},{"name":"Elia Bruni","slug":"elia-bruni","given":"Elia","family":"Bruni","affiliation":"University of Pompeu Fabra {dieuwkehupkes ,"}],"abstract":"Despite a multitude of empirical studies, little consensus exists on whether neural networks are able to generalise compositionally. As a response to this controversy, we present a set of tests that provide a bridge between, on the one hand, the vast amount of linguistic and philosophical theory about compositionality of language and, on the other, the successful neural models of language. We collect different interpretations of compositionality and translate them into five theoretically grounded tests for models that are formulated on a task-independent level. To demonstrate the usefulness of this evaluation paradigm, we instantiate these five tests on a highly compositional data set which we dub PCFG SET,  apply the resulting tests to three popular sequence-to-sequence models and provide an in-depth analysis of the results.","doi":"10.24963/ijcai.2020/708","url":"https://ijcai.org/proceedings/2020/708","pdf":"https://ijcai.org/proceedings/2020/0708.pdf","markdown":"https://ijcai.org/proceedings/2020/708.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/708","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Journal track","track":"Journal track","number":708,"pages":{"first":5065,"last":5069},"keywords":["Natural Language Processing: Natural Language Processing","Machine Learning: Interpretability","Machine Learning: Deep Learning","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25116/file_25116.m3u8","length_sec":301,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2018] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.","title":"Systematic generalization: What is required and can it be learned?","authors":"Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm De Vries, Aaron Courville","year":"2018","venue":"Proceedings of the 6th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojar et al., 2017] Ondrej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, and Julia Kreutzer, editors. Proceedings of the Sec- ond Conference on Machine Translation (WMT), 2017.","title":"Proceedings of the Second Conference on Machine Translation (WMT)","authors":"Ondrej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bowman et al., 2015] Samuel R Bowman, Christopher D Manning, and Christopher Potts. Tree-structured compo- sition in neural networks without tree-structured architec- tures. In Proceedings of the 2015th International Confer- ence on Cognitive Computation: Integrating Neural and Symbolic Approaches, pages 37-42, 2015.","title":"Tree-structured composition in neural networks without tree-structured architectures","authors":"Christopher D Samuel R Bowman, Christopher Manning, Potts","year":"2015","venue":"Proceedings of the 2015th International Conference on Cognitive Computation: Integrating Neural and Symbolic Approaches","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carnap, 1947] Rudolf Carnap. Meaning and necessity: A study in semantics and modal logic. University of Chicago Press, 1947.","title":"Rudolf Carnap. Meaning and necessity: A study in semantics and modal logic","authors":"Carnap","year":"1947","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chomsky, 1956] Noam Chomsky. Three models for the de- scription of language. IRE Transactions on information theory, 2(3):113-124, 1956.","title":"Noam Chomsky. Three models for the description of language","authors":"Chomsky","year":"1956","venue":"IRE Transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dessì and Baroni, 2019] Roberto Dessì and Marco Baroni. CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3919-3923, 2019.","title":"CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networks","authors":"Roberto Dessì, Marco Baroni","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fodor and Pylyshyn, 1988] Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: a critical analysis. Cognition, 28(1-2):3-71, 1988.","title":"Connectionism and cognitive architecture: a critical analysis","authors":"Jerry A Fodor, Zenon W Pylyshyn","year":"1988","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolu- tional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1243-1252, 2017.","title":"Convolutional sequence to sequence learning","authors":"Gehring","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hupkes et al., 2018] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks pro- cess hierarchical structure. Journal of Artificial Intelli- gence Research, 61:907-926, 2018.","title":"Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure","authors":"Hupkes","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hupkes et al., 2020] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decom- posed: how do neural networks generalise? Journal of Artificial Intelligence Research, (67):757-795, 2020.","title":"Compositionality decomposed: how do neural networks generalise","authors":"Hupkes","year":"2020","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Janssen, 1983] Theo Janssen. Foundations and applications of Montague grammar. Mathematisch Centrum, 1983.","title":"Foundations and applications of Montague grammar","authors":"Theo Janssen","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for com- positional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1988-1997, 2017.","title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lake and Baroni, 2018] Brenden Lake and Marco Baroni. Generalization without systematicity: On the composi- tional skills of sequence-to-sequence recurrent networks. In proceedings of the 35th International Conference on Machine Learning (ICML), pages 4487-4499, 2018.","title":"Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks","authors":"Baroni Lake, Brenden Lake, Marco Baroni","year":"2018","venue":"proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liška et al., 2018] Adam Liška, Germán Kruszewski, and Marco Baroni. Memorize or generalize? Searching for a compositional RNN in a haystack. In Proceedings of AE- GAP (FAIM Joint Workshop on Architectures and Evalua- tion for Generality, Autonomy and Progress in AI), 2018.","title":"Memorize or generalize? Searching for a compositional RNN in a haystack","authors":"Adam Liška, Germán Kruszewski, Marco Baroni","year":"2018","venue":"Proceedings of AE-GAP (FAIM Joint Workshop on Architectures and Evaluation for Generality, Autonomy and Progress in AI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Loula et al., 2018] João Loula, Marco Baroni, and Bren- den M Lake. Rearranging the familiar: Testing compo- sitional generalization in recurrent networks. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Ana- lyzing and Interpreting Neural Networks for NLP, pages 108-114, 2018.","title":"Rearranging the familiar: Testing compositional generalization in recurrent networks","authors":"João Loula, Marco Baroni, Brenden M Lake","year":"2018","venue":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mul and Zuidema, 2019] Mathijs Mul and Willem Zuidema. Siamese recurrent networks learn first- order logic reasoning and exhibit zero-shot compositional generalization. CoRR, abs/1906.00180, 2019.","title":"Siamese recurrent networks learn firstorder logic reasoning and exhibit zero-shot compositional generalization","authors":"Zuidema ; ] Mathijs Mul, Willem Mul, Zuidema","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pagin, 2003] Peter Pagin. Communication and strong com- positionality. Journal of Philosophical Logic, 32(3):287- 322, 2003.","title":"Communication and strong compositionality","authors":"Peter Pagin, Pagin","year":"2003","venue":"Journal of Philosophical Logic","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Partee, 1995] Barbara Partee. Lexical semantics and com- positionality. An invitation to cognitive science: Lan- guage, 1:311-360, 1995.","title":"Lexical semantics and compositionality","authors":"Barbara Partee, Partee","year":"1995","venue":"An invitation to cognitive science: Language","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Penke, 2012] Martina Penke. The dual-mechanism debate. In The Oxford handbook of compositionality. Oxford Uni- versity Press, 2012.","title":"The dual-mechanism debate","authors":"Martina Penke","year":"2012","venue":"The Oxford handbook of compositionality","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saxton et al., 2019] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.","title":"Analysing mathematical reasoning abilities of neural models","authors":"David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems (NIPS), pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[von Humboldt, 1836] Wilhelm von Humboldt. On lan- guage: The diversity of human language-structure and its influence on the mental development of mankind. 1836.","title":"Wilhelm von Humboldt. On language: The diversity of human language-structure and its influence on the mental development of mankind","authors":"Von Humboldt","year":"1836","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zadrozny, 1994] Wlodek Zadrozny. From compositional to systematic semantics. Linguistics and philosophy, 17(4):329-342, 1994.","title":"From compositional to systematic semantics","authors":"Wlodek Zadrozny, Zadrozny","year":"1994","venue":"Linguistics and philosophy","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}