{"type":"ScholarlyArticle","title":"An Entanglement-driven Fusion Neural Network for Video Sentiment Analysis","authors":[{"name":"Dimitris Gkoumas","slug":"dimitris-gkoumas","given":"Dimitris","family":"Gkoumas","affiliation":"The Open University , Milton Keynes , UK; School of Electronic Engineering and Computer Science , Queen Mary University of London , UK"},{"name":"Qiuchi Li","slug":"qiuchi-li","given":"Qiuchi","family":"Li","affiliation":"University of Padua , Padua , Italy"},{"name":"Yijun Yu","slug":"yijun-yu","given":"Yijun","family":"Yu","affiliation":"The Open University , Milton Keynes , UK"},{"name":"Dawei Song","slug":"dawei-song","given":"Dawei","family":"Song","affiliation":"The Open University , Milton Keynes , UK; Beijing Institute of Technology , Beijing , China"}],"abstract":"Video data is multimodal in its nature, where an utterance can involve linguistic, visual and acoustic information. Therefore, a key challenge for video sentiment analysis is how to combine different modalities for sentiment recognition effectively. The latest neural network approaches achieve state-of-the-art performance, but they neglect to a large degree of how humans understand and reason about sentiment states. By contrast, recent advances in quantum probabilistic neural models have achieved comparable performance to the state-of-the-art, yet with better transparency and increased level of interpretability. However, the existing quantum-inspired models treat quantum states as either a classical mixture or as a separable tensor product across modalities, without triggering their interactions in a way that they are correlated or non-separable (i.e., entangled). This means that the current models have not fully exploited the expressive power of quantum probabilities. To fill this gap, we propose a transparent quantum probabilistic neural model. The model induces different modalities to interact in such a way that they may not be separable, encoding crossmodal information in the form of non-classical correlations. Comprehensive evaluation on two benchmarking datasets for video sentiment analysis shows that the model achieves significant performance improvement. We also show that the degree of non-separability between modalities optimizes the post-hoc interpretability.","doi":"10.24963/ijcai.2021/239","url":"https://ijcai.org/proceedings/2021/239","pdf":"https://ijcai.org/proceedings/2021/0239.pdf","markdown":"https://ijcai.org/proceedings/2021/239.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/239","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Humans and AI","track":"Main Track","number":239,"pages":{"first":1736,"last":1742},"keywords":["Natural Language Processing: Embeddings","Humans and AI: Cognitive Modeling"],"videos":[],"references":[{"position":1,"raw":"[Banchi et al., 2018] Leonardo Banchi, Edward Grant, An- drea Rocchetto, and Simone Severini. Modelling non- markovian quantum processes with recurrent neural net- works. New Journal of Physics, 20(12):123030, 2018.","title":"Leonardo Banchi, Edward Grant, Andrea Rocchetto, and Simone Severini. Modelling nonmarkovian quantum processes with recurrent neural networks","authors":"Banchi","year":"2018","venue":"New Journal of Physics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bruza et al., 2008] Peter Bruza, Kirsty Kitto, and Doug McEvoy. Entangling words and meaning. In Quantum Interaction: Proceedings of the Second Quantum Interac- tion Symposium (QI-2008), pages 118-124. College Pub- lications, 2008.","title":"Entangling words and meaning","authors":"Peter Bruza, Kirsty Kitto, Doug Mcevoy","year":"2008","venue":"Quantum Interaction: Proceedings of the Second Quantum Interaction Symposium (QI-2008)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruza et al., 2009] Peter Bruza, Kirsty Kitto, Douglas Nel- son, and Cathy McEvoy. Is there something quantum-like about the human mental lexicon? Journal of mathematical psychology, 53(5):362-377, 2009.","title":"Is there something quantum-like about the human mental lexicon","authors":"Peter Bruza, Kirsty Kitto, Douglas Nelson, Cathy Mcevoy","year":"2009","venue":"Journal of mathematical psychology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Busemeyer and Bruza, 2012] Jerome R Busemeyer and Pe- ter D Bruza. Quantum models of cognition and decision. Cambridge University Press, 2012.","title":"Quantum models of cognition and decision","authors":"R Jerome, Peter D Busemeyer, Bruza","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghosal et al., 2018] Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, Asif Ekbal, and Push- pak Bhattacharyya. Contextual inter-modal attention for multi-modal sentiment analysis. In proceedings of the 2018 conference on empirical methods in natural lan- guage processing, pages 3454-3466, 2018.","title":"Contextual inter-modal attention for multi-modal sentiment analysis","authors":"Ghosal","year":"2018","venue":"proceedings of the 2018 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gkoumas et al., 2018] Dimitris Gkoumas, Sagar Uprety, and Dawei Song. Investigating non-classical correlations between decision fused multi-modal documents. In Inter- national Symposium on Quantum Interaction, pages 163- 176. Springer, 2018.","title":"Investigating non-classical correlations between decision fused multi-modal documents","authors":"Dimitris Gkoumas, Sagar Uprety, Dawei Song","year":"2018","venue":"International Symposium on Quantum Interaction","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gkoumas et al., 2021a] Dimitris Gkoumas, Qiuchi Li, Shahram Dehdashti, Massimo Melucci, Yijun Yu, and Dawei Song. Quantum cognitively motivated decision fusion for video sentiment analysis. arXiv preprint arXiv:2101.04406, 2021.","title":"Quantum cognitively motivated decision fusion for video sentiment analysis","authors":"Gkoumas","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gkoumas et al., 2021b] Dimitris Gkoumas, Qiuchi Li, Christina Lioma, Yijun Yu, and Dawei Song. What makes the difference? an empirical comparison of fusion strategies for multimodal language analysis. Information Fusion, 66:184-197, 2021.","title":"What makes the difference? an empirical comparison of fusion strategies for multimodal language analysis","authors":"Gkoumas","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Halmos, 1987] P.R. Halmos. Finite-Dimensional Vector Spaces. Undergraduate Texts in Mathematics. Springer, New York, USA, 1987.","title":"Finite-Dimensional Vector Spaces","authors":"P R Halmos","year":"1987","venue":"Undergraduate Texts in Mathematics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lai et al., 2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In AAAI-2015, 2015.","title":"Recurrent convolutional neural networks for text classification","authors":"Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao","year":"2015","venue":"AAAI-2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Qiuchi Li, Benyou Wang, and Massimo Melucci. Cnm: An interpretable complex-valued network for matching. arXiv preprint arXiv:1904.05298, 2019.","title":"Cnm: An interpretable complex-valued network for matching","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Qiuchi Li, Dimitris Gkoumas, Christina Li- oma, and Massimo Melucci. Quantum-inspired multi- modal fusion for video sentiment analysis. Information Fusion, 65:58-71, 2021.","title":"Quantum-inspired multimodal fusion for video sentiment analysis","authors":"Li","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064, 2018.","title":"Efficient low-rank multimodal fusion with modality-specific factors","authors":"Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Melucci, 2015] Massimo Melucci. Introduction to informa- tion retrieval and quantum mechanics. Springer, 2015.","title":"Introduction to information retrieval and quantum mechanics","authors":"Massimo Melucci","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sordoni et al., 2013] Alessandro Sordoni, Jing He, and Jian-Yun Nie. Modeling latent topic interactions using quantum interference for information retrieval. In ACL- 2013, pages 1197-1200, 2013.","title":"Modeling latent topic interactions using quantum interference for information retrieval","authors":"Alessandro Sordoni, Jing He, Jian-Yun Nie","year":"2013","venue":"ACL-2013","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for un- aligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguis- tics. Meeting, volume 2019, page 6558. NIH Public Ac- cess, 2019.","title":"Multimodal transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Uprety et al., 2020] Sagar Uprety, Dimitris Gkoumas, and Dawei Song. A survey of quantum theory inspired ap- proaches to information retrieval. ACM Computing Sur- veys (CSUR), 53(5):1-39, 2020.","title":"A survey of quantum theory inspired approaches to information retrieval","authors":"Sagar Uprety, Dimitris Gkoumas, Dawei Song","year":"2020","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Van Rijsbergen, 2004] Cornelis Joost Van Rijsbergen. The geometry of information retrieval. Cambridge University Press, 2004.","title":"The geometry of information retrieval","authors":"; Van Rijsbergen, Joost Cornelis, Van Rijsbergen","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vivo et al., 2016] Pierpaolo Vivo, Mauricio P Pato, and Gleb Oshanin. Random pure states: Quantifying bipartite entanglement beyond the linear statistics. Physical Review E, 93(5):052106, 2016.","title":"Random pure states: Quantifying bipartite entanglement beyond the linear statistics","authors":"Pierpaolo Vivo, Mauricio P Pato, Gleb Oshanin","year":"2016","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2010] Jun Wang, Dawei Song, and Leszek Kaliciak. Tensor product of correlated text and visual fea- tures. In In QI'10. Citeseer, 2010.","title":"Tensor product of correlated text and visual features","authors":"Wang","year":"2010","venue":"QI'10","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zadeh et al., 2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016.","title":"Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos","authors":"Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zadeh et al., 2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250, 2017.","title":"Tensor fusion network for multimodal sentiment analysis","authors":"Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zadeh et al., 2018a] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32, 2018.","title":"Memory fusion network for multi-view sequential learning","authors":"Zadeh","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zadeh et al., 2018b] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Multi-attention recurrent network for human communication comprehension","authors":"Zadeh","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zadeh et al., 2018c] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu- mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018.","title":"Multimodal language analysis in the wild: Cmumosei dataset and interpretable dynamic fusion graph","authors":"Zadeh","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}