{"type":"ScholarlyArticle","title":"MMT: Multi-way Multi-modal Transformer for Multimodal Learning","authors":[{"name":"Jiajia Tang","slug":"jiajia-tang","given":"Jiajia","family":"Tang","affiliation":"Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province , School of Computer Science and Technology , Hangzhou Dianzi University , China"},{"name":"Kang Li","slug":"kang-li","given":"Kang","family":"Li","affiliation":"Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province , School of Computer Science and Technology , Hangzhou Dianzi University , China"},{"name":"Ming Hou","slug":"ming-hou","given":"Ming","family":"Hou","affiliation":"RIKEN Center for Advanced Intelligence Project (AIP) , Japan"},{"name":"Xuanyu Jin","slug":"xuanyu-jin","given":"Xuanyu","family":"Jin","affiliation":"Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province , School of Computer Science and Technology , Hangzhou Dianzi University , China"},{"name":"Wanzeng Kong","slug":"wanzeng-kong","given":"Wanzeng","family":"Kong","affiliation":"Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province , School of Computer Science and Technology , Hangzhou Dianzi University , China"},{"name":"Yu Ding","slug":"yu-ding","given":"Yu","family":"Ding","affiliation":"Virtual Human Group , Netease Fuxi AI Lab"},{"name":"Qibin Zhao","slug":"qibin-zhao","given":"Qibin","family":"Zhao","affiliation":"RIKEN Center for Advanced Intelligence Project (AIP) , Japan"}],"abstract":"The heart of multimodal learning research lies the challenge of effectively exploiting fusion representations among multiple modalities.However, existing two-way cross-modality unidirectional attention could only exploit the intermodal interactions from one source to one target modality. This indeed fails to unleash the complete expressive power of multimodal fusion with restricted number of modalities and fixed interactive direction.In this work, the multiway multimodal transformer (MMT) is proposed to simultaneously explore multiway multimodal intercorrelations for each modality via single block rather than multiple stacked cross-modality blocks. The core idea of MMT is the multiway multimodal attention, where the multiple modalities are leveraged to compute the multiway attention tensor. This naturally benefits us to exploit comprehensive many-to-many multimodal interactive paths. Specifically, the multiway tensor is comprised of multiple interconnected modality-aware core tensors that consist of the intramodal interactions. Additionally, the tensor contraction operation is utilized to investigate intermodal dependencies between distinct core tensors.Essentially, our tensor-based multiway structure allows for easily extending MMT to the case associated with an arbitrary number of modalities. Taking MMT as the basis, the hierarchical network is further established to recursively transmit the low-level multiway multimodal interactions to high-level ones. The experiments demonstrate that MMT can achieve state-of-the-art or comparable performance.","doi":"10.24963/ijcai.2022/480","url":"https://ijcai.org/proceedings/2022/480","pdf":"https://ijcai.org/proceedings/2022/0480.pdf","markdown":"https://ijcai.org/proceedings/2022/480.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/480","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":480,"pages":{"first":3458,"last":3465},"keywords":["Machine Learning: Multi-modal learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986504","length_sec":62,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984647","length_sec":340,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ain et al., 2017] Qurat Tul Ain, Mubashir Ali, Amna Riaz, Amna Noureen, Muhammad Kamran, Babar Hayat, and A Rehman. Sentiment analysis using deep learning tech- niques: a review. Int J Adv Comput Sci Appl, 8(6):424, 2017.","title":"Sentiment analysis using deep learning techniques: a review","authors":"Ain","year":"2017","venue":"Int J Adv Comput Sci Appl","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Hazarika et al., 2020] Devamanyu Hazarika, Roger Zim- mermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1122-1131, 2020.","title":"Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis","authors":"Hazarika","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Liang et al., 2018] Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. Multimodal language anal- ysis with recurrent multistage fusion. arXiv preprint arXiv:1808.03920, 2018.","title":"Multimodal language analysis with recurrent multistage fusion","authors":"Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Liu et al., 2018] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064, 2018.","title":"Efficient low-rank multimodal fusion with modality-specific factors","authors":"Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Nojavanasghari et al., 2016] Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 284-288, 2016.","title":"Deep multimodal fusion for persuasiveness prediction","authors":"Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, Louis-Philippe Morency","year":"2016","venue":"Proceedings of the 18th ACM International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Park et al., 2014] Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. Computational analysis of persuasiveness in social mul- timedia: A novel dataset and multimodal prediction ap- proach. In Proceedings of the 16th International Confer- ence on Multimodal Interaction, pages 50-57, 2014.","title":"Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach","authors":"Sunghyun Park, Suk Han, Moitreya Shim, Kenji Chatterjee, Louis-Philippe Sagae, Morency","year":"2014","venue":"Proceedings of the 16th International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Pham et al., 2019] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 6892-6899, 2019.","title":"Found in translation: Learning robust joint representations by cyclic translations between modalities","authors":"Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Poria et al., 2017] Soujanya Poria, Erik Cambria, Deva- manyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computa- tional linguistics (volume 1: Long papers), pages 873- 883, 2017.","title":"Context-dependent sentiment analysis in user-generated videos","authors":"Poria","year":"2017","venue":"Proceedings of the 55th annual meeting of the association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Rahman et al., 2020] Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis- Philippe Morency, and Ehsan Hoque. Integrating mul- timodal information in large pretrained transformers. In Proceedings of the conference. Association for Computa- tional Linguistics. Meeting, volume 2020, page 2359. NIH Public Access, 2020.","title":"Integrating multimodal information in large pretrained transformers","authors":"Rahman","year":"2020","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Rajagopalan et al., 2016] Shyam Sundar Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, and Roland Goecke. Extending long short-term memory for multi- view structured learning. In European Conference on Computer Vision, pages 338-353. Springer, 2016.","title":"Extending long short-term memory for multiview structured learning","authors":"Shyam Sundar Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, Roland Goecke","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Soleymani et al., 2017] Mohammad Soleymani, David Gar- cia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3-14, 2017.","title":"A survey of multimodal sentiment analysis","authors":"Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, Maja Pantic","year":"2017","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sun et al., 2020] Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships be- tween text, audio, and video via deep canonical correla- tion for multimodal language analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8992-8999, 2020.","title":"Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis","authors":"Sun","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tsai et al., 2018] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal represen- tations. arXiv preprint arXiv:1806.06176, 2018.","title":"Learning factorized multimodal representations","authors":"Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for un- aligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguis- tics. Meeting, volume 2019, page 6558. NIH Public Ac- cess, 2019.","title":"Multimodal transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wang et al., 2019] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Words can shift: Dynamically adjusting word represen- tations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223, 2019.","title":"Words can shift: Dynamically adjusting word representations using nonverbal behaviors","authors":"Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wu et al., 2020] Liangqing Wu, Dong Zhang, Qiyuan Liu, Shoushan Li, and Guodong Zhou. Speaker personality recognition with multimodal explicit many2many interac- tions. In 2020 IEEE International Conference on Multi- media and Expo (ICME), pages 1-6, 2020.","title":"Speaker personality recognition with multimodal explicit many2many interactions","authors":"Liangqing Wu, Dong Zhang, Qiyuan Liu, Shoushan Li, Guodong Zhou","year":"2020","venue":"IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yu et al., 2021] Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-specific representations with self- supervised multi-task learning for multimodal sentiment analysis. arXiv preprint arXiv:2102.04830, 2021.","title":"Learning modality-specific representations with selfsupervised multi-task learning for multimodal sentiment analysis","authors":"Yu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zadeh et al., 2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal mes- sages. IEEE Intelligent Systems, 31(6):82-88, 2016.","title":"Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages","authors":"Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency","year":"2016","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zadeh et al., 2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250, 2017.","title":"Tensor fusion network for multimodal sentiment analysis","authors":"Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zadeh et al., 2018a] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32, 2018.","title":"Memory fusion network for multi-view sequential learning","authors":"Zadeh","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zadeh et al., 2018b] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Multi-attention recurrent network for human communication comprehension","authors":"Zadeh","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}