{"type":"ScholarlyArticle","title":"Joint Multimodal Aspect Sentiment Analysis with Aspect Enhancement and Syntactic Adaptive Learning","authors":[{"name":"Linlin Zhu","slug":"linlin-zhu","given":"Linlin","family":"Zhu","affiliation":"Xi'an Jiaotong University"},{"name":"Heli Sun","slug":"heli-sun","given":"Heli","family":"Sun","affiliation":"Xi'an Jiaotong University"},{"name":"Qunshu Gao","slug":"qunshu-gao","given":"Qunshu","family":"Gao","affiliation":"Xi'an Jiaotong University"},{"name":"Tingzhou Yi","slug":"tingzhou-yi","given":"Tingzhou","family":"Yi","affiliation":"Xi'an Jiaotong University"},{"name":"Liang He","slug":"liang-he","given":"Liang","family":"He","affiliation":"Xi'an Jiaotong University"}],"abstract":"As an important task in sentiment analysis, joint multimodal aspect sentiment analysis (JMASA) has received increasing attention in recent years. However, previous approaches either i) directly fuse multimodal data without fully exploiting the correlation between multimodal input data, or ii) equally utilize the dependencies of words in the text for sentiment analysis, ignoring the differences in the importance of different words. To address these limitations, we propose a joint multimodal sentiment analysis method based on Aspect Enhancement and Syntactic Adaptive Learning (AESAL). Specifically, we construct an aspect enhancement pre-training task to enable the model to fully learn the correlation of aspects between multimodal input data. In order to capture the differences in the importance of different words in the text, we design a syntactic adaptive learning mechanism. First, we construct different syntactic dependency graphs based on the distance between words to learn global and local information in the text. Second, we use a multi-channel adaptive graph convolutional network to maintain the uniqueness of each modality while fusing the correlations between different modalities. Experimental results on benchmark datasets show that our method outperforms state-of-the-art methods.","doi":"10.24963/ijcai.2024/738","url":"https://ijcai.org/proceedings/2024/738","pdf":"https://ijcai.org/proceedings/2024/0738.pdf","markdown":"https://ijcai.org/proceedings/2024/738.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/738","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":738,"pages":{"first":6678,"last":6686},"keywords":["Natural Language Processing: NLP: Sentiment analysis, stylistic analysis, and argument mining","Computer Vision: CV: Multimodal learning"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2020] Guimin Chen, Yuanhe Tian, and Yan Song. Joint aspect extraction and sentiment analysis with directional graph convolutional networks. In Proceedings of the 28th international conference on computational lin- guistics, pages 272-279, 2020.","title":"Joint aspect extraction and sentiment analysis with directional graph convolutional networks","authors":"Chen","year":"2020","venue":"Proceedings of the 28th international conference on computational linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Das and Singh, 2023] Ringki Das and Thoudam Doren Singh. Multimodal sentiment analysis: A survey of meth- ods, trends and challenges. ACM Computing Surveys, 2023.","title":"Ringki Das and Thoudam Doren Singh. Multimodal sentiment analysis: A survey of methods, trends and challenges","authors":"Singh Das","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hu et al., 2019] Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. Open-domain targeted senti- ment analysis via span-based extraction and classification. arXiv preprint arXiv:1906.03820, 2019.","title":"Open-domain targeted sentiment analysis via span-based extraction and classification","authors":"Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, Yiwei Lv","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ju et al., 2021] Xincheng Ju, Dong Zhang, Rong Xiao, Jun- hui Li, Shoushan Li, Min Zhang, and Guodong Zhou. Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4395-4405, 2021.","title":"Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection","authors":"Ju","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Khan and Fu, 2021] Zaid Khan and Yun Fu. Exploiting bert for multimodal target sentiment classification through in- put space translation. In Proceedings of the 29th ACM in- ternational conference on multimedia, pages 3034-3042, 2021.","title":"Exploiting bert for multimodal target sentiment classification through input space translation","authors":"Zaid Khan, Yun Fu","year":"2021","venue":"Proceedings of the 29th ACM international conference on multimedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2021a] Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, and Eduard Hovy. Dual graph convolutional networks for aspect-based sentiment analy- sis. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Process- ing (Volume 1: Long Papers), pages 6319-6329, 2021.","title":"Dual graph convolutional networks for aspect-based sentiment analysis","authors":"Li","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021b] Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. Learning implicit senti- ment in aspect-based sentiment analysis with supervised contrastive pre-training. arXiv preprint arXiv:2111.02194, 2021.","title":"Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liang et al., 2022] Bin Liang, Hang Su, Lin Gui, Erik Cam- bria, and Ruifeng Xu. Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional net- works. Knowledge-Based Systems, 235:107643, 2022.","title":"Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks","authors":"Bin Liang, Hang Su, Lin Gui, Erik Cambria, Ruifeng Xu","year":"2022","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ling et al., 2022] Yan Ling, Jianfei Yu, and Rui Xia. Vision-language pre-training for multimodal aspect-based sentiment analysis. arXiv preprint arXiv:2204.07955, 2022.","title":"Vision-language pre-training for multimodal aspect-based sentiment analysis","authors":"Yan Ling, Jianfei Yu, Rui Xia","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023a] Juhua Liu, Qihuang Zhong, Liang Ding, Hua Jin, Bo Du, and Dacheng Tao. Unified instance and knowledge alignment pretraining for aspect-based senti- ment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2629-2642, 2023.","title":"Unified instance and knowledge alignment pretraining for aspect-based sentiment analysis","authors":"Liu","year":"2023","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2023b] Kuanghong Liu, Jin Wang, and Xuejie Zhang. Entity-related unsupervised pretraining with vi- sual prompts for multimodal aspect-based sentiment anal- ysis. In CCF International Conference on Natural Lan- guage Processing and Chinese Computing, pages 481- 493. Springer, 2023.","title":"Entity-related unsupervised pretraining with visual prompts for multimodal aspect-based sentiment analysis","authors":"Liu","year":"2023","venue":"CCF International Conference on Natural Language Processing and Chinese Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sun et al., 2021] Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. Rpbert: a text-image relation propagation-based bert model for multimodal ner. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860-13868, 2021.","title":"Rpbert: a text-image relation propagation-based bert model for multimodal ner","authors":"Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, Fangsheng Weng","year":"2021","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tian et al., 2021] Yuanhe Tian, Guimin Chen, and Yan Song. Aspect-based sentiment analysis with type-aware graph convolutional networks and layer ensemble. In Pro- ceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 2910-2922, 2021.","title":"Aspect-based sentiment analysis with type-aware graph convolutional networks and layer ensemble","authors":"Tian","year":"2021","venue":"Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2020] Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. Relational graph atten- tion network for aspect-based sentiment analysis. arXiv preprint arXiv:2004.12362, 2020.","title":"Relational graph attention network for aspect-based sentiment analysis","authors":"Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, Rui Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2023] Di Wang, Changning Tian, Xiao Liang, Lin Zhao, Lihuo He, and Quan Wang. Dual-perspective fu- sion network for aspect-based multimodal sentiment anal- ysis. IEEE Transactions on Multimedia, 2023.","title":"Dual-perspective fusion network for aspect-based multimodal sentiment analysis","authors":"Di Wang, Changning Tian, Xiao Liang, Lin Zhao, Lihuo He, Quan Wang","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2020a] Hanqian Wu, Siliang Cheng, Jingjing Wang, Shoushan Li, and Lian Chi. Multimodal aspect extraction with region-aware alignment network. In Nat- ural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part I 9, pages 145-156. Springer, 2020.","title":"Multimodal aspect extraction with region-aware alignment network","authors":"Wu","year":"2020","venue":"Natural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC 2020","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2020b] Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In Pro- ceedings of the 28th ACM International Conference on Multimedia, pages 1038-1046, 2020.","title":"Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts","authors":"Wu","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2019] Nan Xu, Wenji Mao, and Guandan Chen. Multi-interactive memory network for aspect based mul- timodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 371-378, 2019.","title":"Multi-interactive memory network for aspect based multimodal sentiment analysis","authors":"Nan Xu, Wenji Mao, Guandan Chen","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yan et al., 2021] Hang Yan, Junqi Dai, Xipeng Qiu, Zheng Zhang, et al. A unified generative framework for aspect-based sentiment analysis. arXiv preprint arXiv:2106.04300, 2021.","title":"A unified generative framework for aspect-based sentiment analysis","authors":"Yan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2022] Li Yang, Jin-Cheon Na, and Jianfei Yu. Cross-modal multitask transformer for end-to-end multi- modal aspect-based sentiment analysis. Information Pro- cessing & Management, 59(5):103038, 2022.","title":"Cross-modal multitask transformer for end-to-end multimodal aspect-based sentiment analysis","authors":"Li Yang, Jin-Cheon Na, Jianfei Yu","year":"2022","venue":"Information Processing & Management","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2023] Xiaocui Yang, Shi Feng, Daling Wang, Sun Qi, Wenfang Wu, Yifei Zhang, Pengfei Hong, and Soujanya Poria. Few-shot joint multimodal aspect- sentiment analysis based on generative multimodal prompt. arXiv preprint arXiv:2305.10169, 2023.","title":"Few-shot joint multimodal aspectsentiment analysis based on generative multimodal prompt","authors":"Xiaocui Yang, Shi Feng, Daling Wang, Sun Qi, Wenfang Wu, Yifei Zhang, Pengfei Hong, Soujanya Poria","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu and Jiang, 2019] Jianfei Yu and Jing Jiang. Adapting bert for target-oriented multimodal sentiment classifica- tion. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 5408- 5414, 2019.","title":"Adapting bert for target-oriented multimodal sentiment classification","authors":"Jianfei Yu, Jing Jiang","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/751"},{"position":25,"raw":"[Yu et al., 2019] Jianfei Yu, Jing Jiang, and Rui Xia. Entity- sensitive attention and fusion network for entity-level mul- timodal sentiment classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:429-439, 2019.","title":"Entitysensitive attention and fusion network for entity-level multimodal sentiment classification","authors":"Jianfei Yu, Jing Jiang, Rui Xia","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2020] Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In Annual Meeting of the Association for Computational Linguistics, 2020.","title":"Improving multimodal named entity recognition via entity span detection with unified multimodal transformer","authors":"Jianfei Yu, Jing Jiang, Li Yang, Rui Xia","year":"2020","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2022] Jianfei Yu, Kai Chen, and Rui Xia. Hierar- chical interactive multimodal transformer for aspect-based multimodal sentiment analysis. IEEE Transactions on Af- fective Computing, 2022.","title":"Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis","authors":"Jianfei Yu, Kai Chen, Rui Xia","year":"2022","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2022a] Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. A survey on aspect-based sentiment analysis: tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2022.","title":"A survey on aspect-based sentiment analysis: tasks, methods, and challenges","authors":"Zhang","year":"2022","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2022b] Zheng Zhang, Zili Zhou, and Yanna Wang. Ssegcn: Syntactic and semantic enhanced graph convolutional network for aspect-based sentiment analy- sis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4916- 4925, 2022.","title":"Ssegcn: Syntactic and semantic enhanced graph convolutional network for aspect-based sentiment analysis","authors":"Zhang","year":"2022","venue":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2020] Pinlong Zhao, Linlin Hou, and Ou Wu. Modeling sentiment dependencies with graph convolu- tional networks for aspect-level sentiment classification. Knowledge-Based Systems, 193:105443, 2020.","title":"Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification","authors":"Zhao","year":"2020","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2021] Jie Zhou, Jiabao Zhao, Jimmy Xiangji Huang, Qinmin Vivian Hu, and Liang He. Masad: A large- scale dataset for multimodal aspect-based sentiment anal- ysis. Neurocomputing, 455:47-58, 2021.","title":"Masad: A largescale dataset for multimodal aspect-based sentiment analysis","authors":"Jie Zhou, Jiabao Zhao, Jimmy Xiangji Huang, Qinmin Vivian Hu, Liang He","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2023] Ru Zhou, Wenya Guo, Xumeng Liu, Shenglong Yu, Ying Zhang, and Xiaojie Yuan. Aom: Detecting aspect-oriented information for multi- modal aspect-based sentiment analysis. arXiv preprint arXiv:2306.01004, 2023.","title":"Aom: Detecting aspect-oriented information for multimodal aspect-based sentiment analysis","authors":"Ru Zhou, Wenya Guo, Xumeng Liu, Shenglong Yu, Ying Zhang, Xiaojie Yuan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2023] Linan Zhu, Zhechao Zhu, Chenwei Zhang, Yifei Xu, and Xiangjie Kong. Multimodal sentiment anal- ysis based on fusion methods: A survey. Information Fu- sion, 95:306-325, 2023.","title":"Multimodal sentiment analysis based on fusion methods: A survey","authors":"Linan Zhu, Zhechao Zhu, Chenwei Zhang, Yifei Xu, Xiangjie Kong","year":"2023","venue":"Information Fusion","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}