{"type":"ScholarlyArticle","title":"Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching","authors":[{"name":"Jianfei Yu","slug":"jianfei-yu","given":"Jianfei","family":"Yu","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Jieming Wang","slug":"jieming-wang","given":"Jieming","family":"Wang","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Rui Xia","slug":"rui-xia","given":"Rui","family":"Xia","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Junjie Li","slug":"junjie-li","given":"Junjie","family":"Li","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"}],"abstract":"Targeted Multimodal Sentiment Classification (TMSC) aims to identify the sentiment polarities over each target mentioned in a pair of sentence and image. Existing methods to TMSC failed to explicitly capture both coarse-grained and fine-grained image-target matching, including 1) the relevance between the image and the target and 2) the alignment between visual objects and the target. To tackle this issue, we propose a new multi-task learning architecture named coarse-to-fine grained Image-Target Matching network (ITM), which jointly performs image-target relevance classification, object-target alignment, and targeted sentiment classification. We further construct an Image-Target Matching dataset by manually annotating the image-target relevance and the visual object aligned with the input target. Experiments on two benchmark TMSC datasets show that our model consistently outperforms the baselines, achieves state-of-the-art results, and presents interpretable visualizations.","doi":"10.24963/ijcai.2022/622","url":"https://ijcai.org/proceedings/2022/622","pdf":"https://ijcai.org/proceedings/2022/0622.pdf","markdown":"https://ijcai.org/proceedings/2022/622.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/622","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":622,"pages":{"first":4482,"last":4488},"keywords":["Natural Language Processing: Sentiment Analysis and Text Mining"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986741","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984884","length_sec":331,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai et al., 2021] Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, and Xipeng Qiu. Does syntax matter? a strong baseline for aspect-based sentiment analysis with roberta. In NAACL, 2021.","title":"Does syntax matter? a strong baseline for aspect-based sentiment analysis with roberta","authors":"Dai","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.","title":"Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2018] Feifan Fan, Yansong Feng, and Dongyan Zhao. Multi-grained attention network for aspect-level sentiment classification. In EMNLP, 2018.","title":"Multi-grained attention network for aspect-level sentiment classification","authors":"Fan","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jiang et al., 2011] Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. Target-dependent twitter sentiment classification. In ACL, 2011.","title":"Target-dependent twitter sentiment classification","authors":"Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, Tiejun Zhao","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kazemzadeh et al., 2014] Sahar Kazemzadeh, Vicente Or- donez, Mark Matten, et al. Referitgame: Referring to ob- jects in photographs of natural scenes. In EMNLP, 2014.","title":"Referitgame: Referring to objects in photographs of natural scenes","authors":"Sahar Kazemzadeh, Vicente Ordonez, Mark Matten","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Khan et al., 2021] Zaid Khan, Yun Fu, et al. Exploiting bert for multimodal target sentimentclassification through in- put space translation. In ACM MM, 2021.","title":"Exploiting bert for multimodal target sentimentclassification through input space translation","authors":"Zaid Khan, Yun Fu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lei et al., 2020] Jie Lei, Licheng Yu, Tamara Berg, and Mo- hit Bansal. Tvqa+: Spatio-temporal grounding for video question answering. In ACL, 2020.","title":"Tvqa+: Spatio-temporal grounding for video question answering","authors":"Jie Lei, Licheng Yu, Tamara Berg, Mohit Bansal","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lu et al., 2018] Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. Visual attention model for name tagging in multimodal social media. In ACL, 2018.","title":"Visual attention model for name tagging in multimodal social media","authors":"Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, Heng Ji","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, et al. Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks. In NIPS, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pontiki et al., 2016] Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, et al. Semeval-2016 task 5: Aspect based sentiment analysis. In SemEval-2016, 2016.","title":"Semeval-2016 task 5: Aspect based sentiment analysis","authors":"Maria Pontiki, Dimitris Galanis, Haris Papageorgiou","year":"2016","venue":"SemEval-2016","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Poria et al., 2017] Soujanya Poria, Erik Cambria, Deva- manyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In ACL, 2017.","title":"Context-dependent sentiment analysis in user-generated videos","authors":"Poria","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Su et al., 2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2019.","title":"Vl-bert: Pre-training of generic visual-linguistic representations","authors":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tang et al., 2016] Duyu Tang, Bing Qin, Ting Liu, et al. As- pect level sentiment classification with deep memory net- work. In EMNLP, 2016.","title":"Aspect level sentiment classification with deep memory network","authors":"Tang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tang et al., 2020] Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. Dependency graph enhanced dual- transformer structure for aspect-based sentiment classifi- cation. In ACL, 2020.","title":"Dependency graph enhanced dualtransformer structure for aspect-based sentiment classification","authors":"Hao Tang, Donghong Ji, Chenliang Li, Qiji Zhou","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.","title":"Multimodal transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2018] Shuai Wang, Sahisnu Mazumder, Bing Liu, Mianwei Zhou, et al. Target-sensitive memory net- works for aspect sentiment classification. In ACL, 2018.","title":"Target-sensitive memory networks for aspect sentiment classification","authors":"Shuai Wang, Sahisnu Mazumder, Bing Liu, Mianwei Zhou","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2020] Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, et al. Relational graph attention network for aspect-based sentiment analysis. In ACL, 2020.","title":"Relational graph attention network for aspect-based sentiment analysis","authors":"Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Jiawei Wang, Zhe Liu, Victor Sheng, Yuqing Song, and Chenjian Qiu. Saliencybert: Recur- rent attention network for target-oriented multimodal sen- timent classification. In PRCV, 2021.","title":"Saliencybert: Recurrent attention network for target-oriented multimodal sentiment classification","authors":"Jiawei Wang, Zhe Liu, Victor Sheng, Yuqing Song, Chenjian Qiu","year":"2021","venue":"PRCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Nan Xu, Wenji Mao, and Guandan Chen. A co-memory network for multimodal sentiment analysis. In SIGIR, 2018.","title":"A co-memory network for multimodal sentiment analysis","authors":"Nan Xu, Wenji Mao, Guandan Chen","year":"2018","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2019a] Hu Xu, Bing Liu, Lei Shu, et al. Bert post- training for review reading comprehension and aspect- based sentiment analysis. In NAACL, 2019.","title":"Bert posttraining for review reading comprehension and aspectbased sentiment analysis","authors":"Xu","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2019b] Nan Xu, Wenji Mao, and Guandan Chen. Multi-interactive memory network for aspect based multi- modal sentiment analysis. In AAAI, 2019.","title":"Multi-interactive memory network for aspect based multimodal sentiment analysis","authors":"Xu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2020] Lu Xu, Lidong Bing, Wei Lu, and Fei Huang. Aspect sentiment classification with aspect- specific opinion spans. In EMNLP, 2020.","title":"Aspect sentiment classification with aspectspecific opinion spans","authors":"Lu Xu, Lidong Bing, Wei Lu, Fei Huang","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2021] Xiaocui Yang, Shi Feng, Yifei Zhang, and Daling Wang. Multimodal sentiment detection based on multi-channel graph neural networks. In ACL, 2021.","title":"Multimodal sentiment detection based on multi-channel graph neural networks","authors":"Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.","title":"Modeling context in referring expressions","authors":"Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, Tamara L Berg","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2018] Zhou Yu, Jun Yu, Chenchao Xiang, Zhou Zhao, Qi Tian, and Dacheng Tao. Rethinking diversified and discriminative proposal generation for visual ground- ing. In IJCAI, 2018.","title":"Rethinking diversified and discriminative proposal generation for visual grounding","authors":"Zhou Yu, Jun Yu, Chenchao Xiang, Zhou Zhao, Qi Tian, Dacheng Tao","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/155"},{"position":29,"raw":"[Yu et al., 2019] Jianfei Yu, Jing Jiang, et al. Adapting bert for target-oriented multimodal sentiment classification. In IJCAI, 2019.","title":"Adapting bert for target-oriented multimodal sentiment classification","authors":"Jianfei Yu, Jing Jiang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/751"},{"position":30,"raw":"[Yu et al., 2020] Jianfei Yu, Jing Jiang, and Rui Xia. Entity- sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE/ACM TASLP, 28:429-439, 2020.","title":"Entitysensitive attention and fusion network for entity-level multimodal sentiment classification","authors":"Jianfei Yu, Jing Jiang, Rui Xia","year":"2020","venue":"IEEE/ACM TASLP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2021] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowl- edge enhanced vision-language representations through scene graphs. In AAAI, 2021.","title":"Ernie-vil: Knowledge enhanced vision-language representations through scene graphs","authors":"Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hua Hao Tian, Haifeng Wu, Wang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2020] Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. Multi- modal multi-label emotion detection with modality and la- bel dependence. In EMNLP, 2020.","title":"Multimodal multi-label emotion detection with modality and label dependence","authors":"Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, Guodong Zhou","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}