{"type":"ScholarlyArticle","title":"SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities","authors":[{"name":"Pengbo Hu","slug":"pengbo-hu","given":"Pengbo","family":"Hu","affiliation":"School of Mathematical Sciences , University of Science and Technology of China , Hefei , China; Shanghai Center for Brain Science and Brain-Inspired Technology , Shanghai , China"},{"name":"Xingyu Li","slug":"xingyu-li","given":"Xingyu","family":"Li","affiliation":"Shanghai Center for Brain Science and Brain-Inspired Technology , Shanghai , China"},{"name":"Yi Zhou","slug":"yi-zhou","given":"Yi","family":"Zhou","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application , School of Information Science and Technology , University of Science and Technology of China , Hefei , China; Shanghai Center for Brain Science and Brain-Inspired Technology , Shanghai , China"}],"abstract":"As deep learning advances, there is an ever-growing demand for models capable of synthesizing information from multi-modal resources to address the complex tasks raised from real-life applications. Recently, many large multi-modal datasets have been collected, on which researchers actively explore different methods of fusing multi-modal information. However, little attention has been paid to quantifying the contribution of different modalities within the proposed models. \r\nIn this paper, we propose the SHapley vAlue-based PErceptual (SHAPE) scores that measure the marginal contribution of individual modalities and the degree of cooperation across modalities. Using these scores, we systematically evaluate different fusion methods on different multi-modal datasets for different tasks. Our experiments suggest that for some tasks where different modalities are complementary, the multi-modal models still tend to use the dominant modality alone and ignore the cooperation across modalities. On the other hand, models learn to exploit cross-modal cooperation when different modalities are indispensable for the task. In this case, the scores indicate it is better to fuse different modalities at relatively early stages. We hope our scores can help improve the understanding of how the present multi-modal models operate on different modalities and encourage more sophisticated methods of integrating multiple modalities.","doi":"10.24963/ijcai.2022/425","url":"https://ijcai.org/proceedings/2022/425","pdf":"https://ijcai.org/proceedings/2022/0425.pdf","markdown":"https://ijcai.org/proceedings/2022/425.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/425","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":425,"pages":{"first":3064,"last":3070},"keywords":["Machine Learning: Multi-modal learning","Machine Learning: Evaluation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987092","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985235","length_sec":635,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on com- puter vision, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2021] Chongyang Bai, Haipeng Chen, Srijan Ku- mar, Jure Leskovec, and V. S. Subrahmanian. M2P2: Multimodal Persuasion Prediction using Adaptive Fusion. arXiv preprint arXiv:2006.11405, December 2021.","title":"M2P2: Multimodal Persuasion Prediction using Adaptive Fusion","authors":"Chongyang Bai, Haipeng Chen, Srijan Kumar, Jure Leskovec, V S Subrahmanian","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baltrušaitis et al., 2018] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443, 2018.","title":"Multimodal machine learning: A survey and taxonomy","authors":"Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Delbrouck et al., 2020] Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. A transformer-based joint-encoding for emotion recognition and sentiment analysis. arXiv preprint arXiv:2006.15955, 2020.","title":"A transformer-based joint-encoding for emotion recognition and sentiment analysis","authors":"Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, Stéphane Dupont","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gat et al., 2021] Itai Gat, Idan Schwartz, and Alex Schwing. Perceptual score: What data modalities does your model perceive? Advances in Neural Information Processing Systems, 34, 2021.","title":"Perceptual score: What data modalities does your model perceive","authors":"Itai Gat, Idan Schwartz, Alex Schwing","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hessel and Lee, 2020] Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! arXiv preprint arXiv:2010.06572, 2020.","title":"Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think!","authors":"Lee ; Hessel, Jack Hessel, Lillian Lee","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiela et al., 2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950, 2019.","title":"Supervised multimodal bitransformers for classifying images and text","authors":"Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, Davide Testuggine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2021] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. MultiBench: Multiscale Bench- marks for Multimodal Representation Learning. arXiv preprint arXiv:2107.07502, November 2021.","title":"MultiBench: Multiscale Benchmarks for Multimodal Representation Learning","authors":"Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2019] Fei Liu, Jing Liu, Zhiwei Fang, Richang Hong, and Hanqing Lu. Densely connected attention flow for visual question answering. In IJCAI, pages 869-875, 2019.","title":"Densely connected attention flow for visual question answering","authors":"Fei Liu, Jing Liu, Zhiwei Fang, Richang Hong, Hanqing Lu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/122"},{"position":13,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems, pages 4768-4777, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Proceedings of the 31st international conference on neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Molnar, 2020] Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.","title":"Interpretable machine learning","authors":"Christoph Molnar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vec- tors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rodrigues Makiuchi et al., 2019] Mariana Rodrigues Mak- iuchi, Tifani Warnita, Kuniaki Uto, and Koichi Shinoda. Multimodal fusion of bert-cnn and gated cnn represen- tations for depression detection. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, pages 55-63, 2019.","title":"Multimodal fusion of bert-cnn and gated cnn representations for depression detection","authors":"Mariana Rodrigues Makiuchi, Tifani Warnita, Kuniaki Uto, Koichi Shinoda","year":"2019","venue":"Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shapley, 1953] LS Shapley. The value of n-person games. Annals of Math. Studies, 28:307-317, 1953.","title":"LS Shapley. The value of n-person games","authors":"Shapley","year":"1953","venue":"Annals of Math. Studies","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tiwari, 2010] Vibha Tiwari. Mfcc and its applications in speaker recognition. International journal on emerging technologies, 1(1):19-22, 2010.","title":"Mfcc and its applications in speaker recognition","authors":"Vibha Tiwari","year":"2010","venue":"International journal on emerging technologies","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vadicamo et al., 2017] Lucia Vadicamo, Fabio Carrara, An- drea Cimino, Stefano Cresci, Felice Dell'Orletta, Fabrizio Falchi, and Maurizio Tesconi. Cross-media learning for image sentiment analysis in the wild. In Proceedings of the IEEE International Conference on Computer Vision Work- shops, pages 308-317, 2017.","title":"Cross-media learning for image sentiment analysis in the wild","authors":"Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell'orletta, Fabrizio Falchi, Maurizio Tesconi","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Weiyao Wang, Du Tran, and Matt Feis- zli. What Makes Training Multi-Modal Classification Net- works Hard? arXiv preprint arXiv:1905.12681, April 2020.","title":"What Makes Training Multi-Modal Classification Networks Hard","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang, 2021] Yang Wang. Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion. ACM Trans- actions on Multimedia Computing, Communications, and Applications (TOMM), 17(1s):1-25, 2021.","title":"Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion","authors":"Yang Wang","year":"2021","venue":"ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Weber, 1988] Robert J Weber. Probabilistic values for games. The Shapley Value. Essays in Honor of Lloyd S. Shapley, pages 101-119, 1988.","title":"Probabilistic values for games. The Shapley Value","authors":"J Robert, Weber","year":"1988","venue":"Essays in Honor of Lloyd S. Shapley","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zadeh et al., 2018] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu- mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018.","title":"Multimodal language analysis in the wild: Cmumosei dataset and interpretable dynamic fusion graph","authors":"Zadeh","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}