{"type":"ScholarlyArticle","title":"Interpretable Tensor Fusion","authors":[{"name":"Saurabh Varshneya","slug":"saurabh-varshneya","given":"Saurabh","family":"Varshneya","affiliation":"RPTU Kaiserslautern-"},{"name":"Antoine Ledent","slug":"antoine-ledent","given":"Antoine","family":"Ledent","affiliation":"Singapore Management University"},{"name":"Philipp Liznerski","slug":"philipp-liznerski","given":"Philipp","family":"Liznerski","affiliation":"RPTU Kaiserslautern-"},{"name":"Andriy Balinskyy","slug":"andriy-balinskyy","given":"Andriy","family":"Balinskyy","affiliation":"RPTU Kaiserslautern-"},{"name":"Purvanshi Mehta","slug":"purvanshi-mehta","given":"Purvanshi","family":"Mehta","affiliation":"Lica World"},{"name":"Waleed Mustafa","slug":"waleed-mustafa","given":"Waleed","family":"Mustafa","affiliation":"RPTU Kaiserslautern-"},{"name":"Marius Kloft","slug":"marius-kloft","given":"Marius","family":"Kloft","affiliation":"RPTU Kaiserslautern-"}],"abstract":"Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method training a neural network to simultaneously learn multiple data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of the data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations, respectively. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on four real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.","doi":"10.24963/ijcai.2024/557","url":"https://ijcai.org/proceedings/2024/557","pdf":"https://ijcai.org/proceedings/2024/0557.pdf","markdown":"https://ijcai.org/proceedings/2024/557.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/557","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":557,"pages":{"first":5037,"last":5045},"keywords":["Machine Learning: ML: Multi-modal learning","Machine Learning: ML: Explainable/Interpretable machine learning","Natural Language Processing: NLP: Sentiment analysis, stylistic analysis, and argument mining","Machine Learning: ML: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Arabacı et al., 2021] Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, and Alptekin Temizel. Multi-modal egocentric activity recognition using multi- kernel learning. Multimedia Tools and Applications, 80(11):16299-16328, 2021.","title":"Multi-modal egocentric activity recognition using multikernel learning","authors":"Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, Alptekin Temizel","year":"2021","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bagher Zadeh et al., 2018] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.","title":"Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph","authors":"Bagher Zadeh","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Behravan et al., 2018] Hamid Behravan, Jaana M Har- tikainen, Maria Tengström, Katri Pylkäs, Robert Winqvist, Veli-Matti Kosma, and Arto Mannermaa. Machine learn- ing identifies interacting genetic variants contributing to breast cancer risk: A case study in finnish cases and con- trols. Scientific reports, 8(1):13149, 2018.","title":"Machine learning identifies interacting genetic variants contributing to breast cancer risk: A case study in finnish cases and controls","authors":"Hamid Behravan, Maria Jaana M Hartikainen, Katri Tengström, Robert Pylkäs, Winqvist, Veli-Matti, Arto Kosma, Mannermaa","year":"2018","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Büchel et al., 1998] Christian Büchel, Cathy Price, and Karl Friston. A multimodal language region in the ventral vi- sual pathway. Nature, 394(6690):274-277, 1998.","title":"A multimodal language region in the ventral visual pathway","authors":"Büchel","year":"1998","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2020] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part VI 16, pages 565-580. Springer, 2020.","title":"Behind the scene: Revealing the secrets of pre-trained vision-and-language models","authors":"Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, Jingjing Liu","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Castro et al., 2019] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihal- cea, and Soujanya Poria. Towards multimodal sarcasm de- tection, 2019. [Chagas et al., 2020] Paulo Chagas, Luiz Souza, Ikaro Araújo, Nayze Aldeman, Angelo Duarte, Michele Angelo, Washington LC Dos-Santos, and Luciano Oliveira. Classi- fication of glomerular hypercellularity using convolutional features and support vector machine. Artificial intelligence in medicine, 103:101808, 2020.","title":"Classification of glomerular hypercellularity using convolutional features and support vector machine","authors":"Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, Soujanya Poria, ; Chagas","year":"2019","venue":"Artificial intelligence in medicine","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chandrasekaran et al., 2018] Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make vqa models more pre- dictable to a human? arXiv preprint arXiv:1810.12366, 2018.","title":"Do explanations make vqa models more predictable to a human","authors":"Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, Devi Parikh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2014] JunKai Chen, Zenghai Chen, Zheru Chi, and Hong Fu. Emotion recognition in the wild with feature fusion and multiple kernel learning. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 508-513, 2014.","title":"Emotion recognition in the wild with feature fusion and multiple kernel learning","authors":"Chen","year":"2014","venue":"Proceedings of the 16th International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elgart et al., 2022] Michael Elgart, Genevieve Lyons, San- tiago Romero-Brufau, Nuzulul Kurniansyah, Jennifer A Brody, Xiuqing Guo, Henry J Lin, Laura Raffield, Yan Gao, Han Chen, et al. Non-linear machine learning models incorporating snps and prs improve polygenic prediction in diverse human populations. Communications Biology, 5(1):856, 2022.","title":"Non-linear machine learning models incorporating snps and prs improve polygenic prediction in diverse human populations","authors":"Michael Elgart, Genevieve Lyons, Santiago Romero-Brufau, Nuzulul Kurniansyah, Jennifer A Brody, Xiuqing Guo, Laura Henry J Lin, Yan Raffield, Han Gao, Chen","year":"2022","venue":"Communications Biology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frank et al., 2021] Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-for- language? on cross-modal influence in multimodal trans- formers. arXiv preprint arXiv:2109.04448, 2021.","title":"Vision-and-language or vision-forlanguage? on cross-modal influence in multimodal transformers","authors":"Stella Frank, Emanuele Bugliarello, Desmond Elliott","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gat et al., 2021] Itai Gat, Idan Schwartz, and Alex Schwing. Perceptual score: What data modalities does your model perceive? Advances in Neural Information Processing Sys- tems, 34:21630-21643, 2021.","title":"Perceptual score: What data modalities does your model perceive?","authors":"Itai Gat, Idan Schwartz, Alex Schwing","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hasan et al., 2019] Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, et al. Ur-funny: A multimodal lan- guage dataset for understanding humor. arXiv preprint arXiv:1904.06618, 2019.","title":"Ur-funny: A multimodal language dataset for understanding humor","authors":"Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] Zhipeng He, Zina Li, Fuzhou Yang, Lei Wang, Jingcong Li, Chengju Zhou, and Jiahui Pan. Ad- vances in multimodal emotion recognition based on brain- computer interfaces. Brain sciences, 10(10):687, 2020.","title":"Advances in multimodal emotion recognition based on braincomputer interfaces","authors":"Zhipeng He, Zina Li, Fuzhou Yang, Lei Wang, Jingcong Li, Chengju Zhou, Jiahui Pan","year":"2020","venue":"Brain sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hessel and Lee, 2020] Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 861-877, 2020.","title":"Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think!","authors":"Lee ; Hessel, Jack Hessel, Lillian Lee","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Interna- tional conference on machine learning, pages 448-456.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kanehira et al., 2019] Atsushi Kanehira, Kentaro Take- moto, Sho Inayoshi, and Tatsuya Harada. Multimodal ex- planations by predicting counterfactuality in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8594-8602, 2019.","title":"Multimodal explanations by predicting counterfactuality in videos","authors":"Atsushi Kanehira, Kentaro Takemoto, Sho Inayoshi, Tatsuya Harada","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kloft et al., 2011] Marius Kloft, Ulf Brefeld, Sören Son- nenburg, and Alexander Zien. Lp-norm multiple kernel learning. The Journal of Machine Learning Research, 12:953-997, 2011.","title":"Lp-norm multiple kernel learning","authors":"Marius Kloft, Ulf Brefeld, Sören Sonnenburg, Alexander Zien","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leiva et al., 2020] Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A dataset for topic modeling of mobile ui designs. In 22nd International Conference on Human- Computer Interaction with Mobile Devices and Services, pages 1-4, 2020.","title":"Enrico: A dataset for topic modeling of mobile ui designs","authors":"Asutosh Luis A Leiva, Antti Hota, Oulasvirta","year":"2020","venue":"22nd International Conference on Human-Computer Interaction with Mobile Devices and Services","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2021] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Pe- ter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Mul- tiscale benchmarks for multimodal representation learn- ing. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track (Round 1), 2021.","title":"Multibench: Multiscale benchmarks for multimodal representation learning","authors":"Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu","year":"2021","venue":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lunghi et al., 2019] Giacomo Lunghi, Raul Marin, Mario Di Castro, Alessandro Masi, and Pedro J Sanz. Multi- modal human-robot interface for accessible remote robotic interventions in hazardous environments. IEEE Access, 7:127290-127319, 2019.","title":"Multimodal human-robot interface for accessible remote robotic interventions in hazardous environments","authors":"Giacomo Lunghi, Raul Marin, Mario Di Castro, Alessandro Masi, Pedro J Sanz","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Park et al., 2018] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Marcus Rohrbach. Multimodal explanations: Jus- tifying decisions and pointing to the evidence. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 8779-8788, 2018.","title":"Multimodal explanations: Justifying decisions and pointing to the evidence","authors":"Dong Huk, Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, Marcus Rohrbach","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pérez-Rúa et al., 2019] Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6966-6975, 2019.","title":"Mfas: Multimodal fusion architecture search","authors":"Pérez-Rúa","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Poria et al., 2015] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance- level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural lan- guage processing, pages 2539-2544, 2015.","title":"Deep convolutional neural network textual features and multiple kernel learning for utterancelevel multimodal sentiment analysis","authors":"Poria","year":"2015","venue":"Proceedings of the 2015 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rakotomamonjy et al., 2008] Alain Rakotomamonjy, Fran- cis Bach, Stéphane Canu, and Yves Grandvalet. Sim- plemkl. Journal of Machine Learning Research, 9:2491- 2521, 2008.","title":"Simplemkl","authors":"Alain Rakotomamonjy, Francis Bach, Stéphane Canu, Yves Grandvalet","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sabour et al., 2017] Sara Sabour, Nicholas Frosst, and Ge- offrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.","title":"Dynamic routing between capsules","authors":"Sara Sabour, Nicholas Frosst, Geoffrey E Hinton","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tan and Bansal, 2019] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019.","title":"Lxmert: Learning cross-modality encoder representations from transformers","authors":"Hao Tan, Mohit Bansal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for un- aligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguis- tics. Meeting, volume 2019, page 6558. NIH Public Ac- cess, 2019.","title":"Multimodal transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tsai et al., 2020] Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multimodal routing: Improving local and global interpretability of multimodal language analysis. In Pro- ceedings of the Conference on Empirical Methods in Natu- ral Language Processing. Conference on Empirical Meth- ods in Natural Language Processing, volume 2020, page 1823. NIH Public Access, 2020.","title":"Multimodal routing: Improving local and global interpretability of multimodal language analysis","authors":"Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Ruslan Salakhutdinov, Louis-Philippe Morency","year":"2020","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vielzeuf et al., 2018] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a mul- tilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.","title":"Centralnet: a multilayer approach for multimodal fusion","authors":"Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, Frédéric Jurie","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wörtwein et al., 2022] Torsten Wörtwein, Lisa Shee- ber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency. Beyond additive fusion: Learning non-additive multimodal interactions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4681-4696, 2022.","title":"Beyond additive fusion: Learning non-additive multimodal interactions","authors":"Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, Louis-Philippe Morency","year":"2022","venue":"Findings of the Association for Computational Linguistics: EMNLP 2022","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zadeh et al., 2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos, 2016.","title":"Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos","authors":"Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zadeh et al., 2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103-1114, 2017.","title":"Tensor fusion network for multimodal sentiment analysis","authors":"Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2023] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023.","title":"Meta-transformer: A unified framework for multimodal learning","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}