{"type":"ScholarlyArticle","title":"Adapting BERT for Target-Oriented Multimodal Sentiment Classification","authors":[{"name":"Jianfei Yu","slug":"jianfei-yu","given":"Jianfei","family":"Yu","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China; School of Information Systems , Singapore Management University , Singapore"},{"name":"Jing Jiang","slug":"jing-jiang","given":"Jing","family":"Jiang","affiliation":"School of Information Systems , Singapore Management University , Singapore"}],"abstract":"As an important task in Sentiment Analysis, Target-oriented Sentiment Classification (TSC) aims to identify sentiment polarities over each opinion target in a sentence. However, existing approaches to this task primarily rely on the textual content, but ignoring the other increasingly popular multimodal data sources (e.g., images), which can enhance the robustness of these text-based models. Motivated by this observation and inspired by the recently proposed BERT architecture, we study Target-oriented Multimodal Sentiment Classification (TMSC) and propose a multimodal BERT architecture. To model intra-modality dynamics, we first apply BERT to obtain target-sensitive textual representations. We then borrow the idea from self-attention and design a target attention mechanism to perform target-image matching to derive target-sensitive visual representations. To model inter-modality dynamics, we further propose to stack a set of self-attention layers to capture multimodal interactions. Experimental results show that our model can outperform several highly competitive approaches for TSC and TMSC.","doi":"10.24963/ijcai.2019/751","url":"https://ijcai.org/proceedings/2019/751","pdf":"https://ijcai.org/proceedings/2019/0751.pdf","markdown":"https://ijcai.org/proceedings/2019/751.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/751","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":751,"pages":{"first":5408,"last":5414},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Sentiment Analysis and Text Mining"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Peng Chen, Zhongqian Sun, et al. Re- current attention network on memory for aspect sentiment analysis. In EMNLP, pages 452-461, 2017.","title":"Recurrent attention network on memory for aspect sentiment analysis","authors":"Peng Chen, Zhongqian Sun","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2014] Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. Adaptive recursive neural network for target-dependent twitter sentiment classifica- tion. In ACL, pages 49-54, 2014.","title":"Adaptive recursive neural network for target-dependent twitter sentiment classification","authors":"Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, Ke Xu","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan et al., 2018] Feifan Fan, Yansong Feng, et al. Multi- grained attention network for aspect-level sentiment clas- sification. In EMNLP, pages 3433-3442, 2018.","title":"Feifan Fan, Yansong Feng, et al. Multigrained attention network for aspect-level sentiment classification","authors":"Fan","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regular- izers with gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016.","title":"Bridging nonlinearities and stochastic regularizers with gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard and Ruder, 2018] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text clas- sification. In ACL, pages 328-339, 2018.","title":"Universal language model fine-tuning for text classification","authors":"Ruder Howard, Jeremy Howard, Sebastian Ruder","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2011] Long Jiang, Mo Yu, et al. Target- dependent twitter sentiment classification. In ACL, pages 151-160, 2011.","title":"Targetdependent twitter sentiment classification","authors":"Long Jiang, Mo Yu","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiritchenko et al., 2014] Svetlana Kiritchenko, Xiaodan Zhu, et al. Nrc-canada-2014: Detecting aspects and senti- ment in customer reviews. In SemEval 2014, pages 437- 442, 2014.","title":"Nrc-canada-2014: Detecting aspects and sentiment in customer reviews","authors":"Svetlana Kiritchenko, Xiaodan Zhu","year":"2014","venue":"SemEval 2014","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Xin Li, Lidong Bing, Wai Lam, and Bei Shi. Transformation networks for target-oriented senti- ment classification. In ACL, pages 946-956, 2018.","title":"Transformation networks for target-oriented sentiment classification","authors":"Xin Li, Lidong Bing, Wai Lam, Bei Shi","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu, 2012] Bing Liu. Sentiment analysis and opinion min- ing. Synthesis lectures on human language technologies, 5(1):1-167, 2012.","title":"Sentiment analysis and opinion mining","authors":"Bing Liu, Liu","year":"2012","venue":"Synthesis lectures on human language technologies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2018] Di Lu, Leonardo Neves, et al. Visual atten- tion model for name tagging in multimodal social media. In ACL, pages 1990-1999, 2018.","title":"Visual attention model for name tagging in multimodal social media","authors":"Di Lu, Leonardo Neves","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2017] Dehong Ma, Sujian Li, et al. Interactive at- tention networks for aspect-level sentiment classification. In IJCAI, pages 4068-4074, 2017.","title":"Interactive attention networks for aspect-level sentiment classification","authors":"Dehong Ma, Sujian Li","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/568"},{"position":16,"raw":"[Peters et al., 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representa- tions. In NAACL, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pontiki et al., 2014] Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. Semeval-2014 task 4: Aspect based sentiment analysis. In SemEval, pages 19-30, 2014.","title":"Semeval-2014 task 4: Aspect based sentiment analysis","authors":"Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar","year":"2014","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Poria et al., 2015] Soujanya Poria, Erik Cambria, et al. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In EMNLP, pages 2539-2544, 2015.","title":"Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis","authors":"Poria","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Poria et al., 2017] Soujanya Poria, Erik Cambria, et al. Context-dependent sentiment analysis in user-generated videos. In ACL, pages 873-883, 2017.","title":"Context-dependent sentiment analysis in user-generated videos","authors":"Poria","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2016a] Duyu Tang, Bing Qin, et al. Effective LSTMs for target-dependent sentiment classification. In COLING, pages 3298-3307, 2016.","title":"Effective LSTMs for target-dependent sentiment classification","authors":"Tang","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2016b] Duyu Tang, Bing Qin, Ting Liu, et al. Aspect level sentiment classification with deep memory network. In EMNLP, pages 214-224, 2016.","title":"Aspect level sentiment classification with deep memory network","authors":"Tang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vo and Zhang, 2015] Duy-Tin Vo and Yue Zhang. Target- dependent twitter sentiment classification with rich auto- matic features. In IJCAI, pages 1347-1353, 2015.","title":"Targetdependent twitter sentiment classification with rich automatic features","authors":"Duy-Tin Vo, Yue Zhang","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang and Lu, 2018] Bailin Wang and Wei Lu. Learning la- tent opinions for aspect-level sentiment classification. In AAAI, pages 5537-5544, 2018.","title":"Learning latent opinions for aspect-level sentiment classification","authors":"Bailin Wang, Wei Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Yequan Wang, Minlie Huang, Li Zhao, et al. Attention-based LSTM for aspect-level sentiment classification. In EMNLP, pages 606-615, 2016.","title":"Attention-based LSTM for aspect-level sentiment classification","authors":"Wang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018] Shuai Wang, Sahisnu Mazumder, et al. Target-sensitive memory networks for aspect sentiment classification. In ACL, pages 957-967, 2018.","title":"Target-sensitive memory networks for aspect sentiment classification","authors":"Shuai Wang, Sahisnu Mazumder","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2019] Nan Xu, Wenji Mao, and Guandan Chen. Multi-interactive memory network for aspect based multi- modal sentiment analysis. In AAAI, 2019.","title":"Multi-interactive memory network for aspect based multimodal sentiment analysis","authors":"Nan Xu, Wenji Mao, Guandan Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zadeh et al., 2017] Amir Zadeh, Minghai Chen, et al. Ten- sor fusion network for multimodal sentiment analysis. In EMNLP, pages 1103-1114, 2017.","title":"Tensor fusion network for multimodal sentiment analysis","authors":"Amir Zadeh, Minghai Chen","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zadeh et al., 2018] Amir Zadeh, Paul Pu Liang, et al. Mem- ory fusion network for multi-view sequential learning. In AAAI, pages 5634-5641, 2018.","title":"Memory fusion network for multi-view sequential learning","authors":"Amir Zadeh, Paul Pu Liang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2016] Meishan Zhang, Yue Zhang, and Duy- Tin Vo. Gated neural networks for targeted sentiment anal- ysis. In AAAI, pages 3087-3093, 2016.","title":"Gated neural networks for targeted sentiment analysis","authors":"Zhang","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018a] Lei Zhang, Shuai Wang, and Bing Liu. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, page p.e1253, 2018.","title":"Deep learning for sentiment analysis: A survey","authors":"Zhang","year":"2018","venue":"Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018b] Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. Adaptive co-attention network for named entity recognition in tweets. In AAAI, pages 5674- 5681, 2018.","title":"Adaptive co-attention network for named entity recognition in tweets","authors":"Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}