Adapting BERT for Target-Oriented Multimodal Sentiment Classification

1 Introduction

Target-oriented Sentiment Classification (TSC) is a fundamental task in sentiment analysis, which aims to detect sentiment orientations over individual opinion targets mentioned in a sentence [Liu, 2012]. For example, given a tweet "Georgina Hermitage is a #one2watch since she broke the 400m T37 WR.", the user expresses positive and neutral sentiments towards Georgina Hermitage and 400m T37, respectively.

To address this TSC problem, various supervised learning techniques empowered with both shallow and deep textual features have been proposed [Jiang et al., 2011; Kiritchenko et al., 2014; Dong et al., 2014; Vo and Zhang, 2015;

∗The corresponding author.

1 We make our annotations for the two TMSC datasets publicly available via the link: https://github.com/jefferyYu/TomBERT.

(b). [Arizona] neutral sheriff [Joe Arpaio] negative found in contempt in racial profiling case.

Figure 1: Representative examples for Target-Oriented Multimodal Sentiment Classification in our Twitter datasets. Opinion targets and their corresponding sentiment polarities are highlighted.

Zhang et al., 2016]. With the recent trend of attention mechanism [Tang et al., 2016b], many studies have proposed different attention-based neural architectures to model the interactions between opinion targets and their surrounding context words [Wang et al., 2016; Wang et al., 2018], which can further improve the state-of-the-art performance on several benchmark datasets [Li et al., 2018].

However, all these aforementioned approaches suffer from two common limitations. First, most of them only randomly initialize their model parameters, which may lead to suboptimal solutions by only optimizing them with a small, taskspecific corpus. With the recent trend of pre-training model parameters with unsupervised language models for various NLP tasks [Peters et al., 2018; Howard and Ruder, 2018], it is natural to expect that these well initialized models can capture each word's semantic and syntactic meaning in different contexts and lead to better solutions for TSC.

More importantly, these existing methods primarily rely on textual content, and fail to consider the other associated data sources (e.g., images), which may potentially complement the textual content and enhance these text-based models. As the user-generated content on the Web (e.g., tweets, reviews) are increasingly multimodal, we observe that the associated images are generally useful for TSC for a couple of reasons. First, a user-generated sentence often focuses on one opinion target, and the associated image tends to highlight the focused target (e.g., Georgina Hermitage in Fig.1.a and Joe Arpaio in Fig.1.b). Second, it is sometimes hard to detect the sentiment over the focused target due to the short and informal natural of the sentence, but the associated image may help reflect a user's sentiment over the focused target (e.g., in Fig.1, the two users respectively posted a pleasant image of Georgina Hermitage and an unpleasant image of Joe Arpaio). Third, for those remaining targets, the sentence often expresses neutral sentiment towards them, and the image also tends to pay less or even no attention to them (e.g., 400m T37 and Arizona in Fig.1). Therefore, it would be interesting to explore how to construct the alignment between opinion targets and textual/visual contents to model the intra-modality dynamics, and then fuse the textual and visual representations to uncover their inter-modality alignments in a unified model for Targetoriented Multimodal Sentiment Classification (TMSC).

To address these two limitations, in this paper, we build our model on top of the recent BERT architecture [Devlin et al., 2018], whose pre-trained model parameters from a large corpus can help obtain contextualized word representations, and whose multi-head self-attention mechanism utilized in its Transformer encoder can automatically learn different levels of alignment between any two complex objects. Specifically, we first transform each input sentence into two sub-sentences: individual opinion target words and the remaining context words, and employ BERT to obtain targetsensitive textual representations. Moreover, inspired by the key idea of self-attention, we further design a target attention mechanism to automatically learn the alignment between opinion targets and images, where the targets are leveraged as queries to supervise the model to assign appropriate attention weights to different regions in the associated images to induce the target-sensitive visual representations. After modelling the intra-modality alignments, we further stack a set of self-attention layers on top of them to automatically capture their inter-modality interactions. We refer to this architecture as Target-oriented multimodal BERT or TomBERT for short.

Evaluations on three benchmark datasets for TSC and two manually annotated Twitter datasets for TMSC demonstrate the following: First, the fine-tuned BERT model outperforms the previously-reported best results on three benchmark datasets for TSC. Second, TomBERT can outperform both the state-of-the-art text-based approaches and highly competitive multimodal methods on the Twitter datasets for TMSC. Third, further analysis shows that due to the targetsensitive nature, TomBERT is especially useful when the input sentence has multiple opinion targets.

Our main contributions are summarized as follows:

• We devise a target-oriented multimodal BERT architecture for TMSC, where the two BERT-based modules at the bottom are used to capture intra-modality dynamics including target-text and target-image alignments, and another BERT-based module is stacked on top to capture inter-modality dynamics, i.e., text-image alignments.
• We propose to employ the standard BERT layer to model target-text and text-image alignments, and design a special target-image matching layer coupled with a target attention mechanism to model target-image alignments.

2 RelatedWork

2.1 Target-Oriented Sentiment Classification

As an important task in sentiment analysis, Target-oriented Sentiment Classification (TSC) has been extensively studied in recent years [Zhang et al., 2018a]. One line of work focuses on leveraging external resources to manually design a set of task-specific features, followed by applying traditional statistical learning methods on the features for sentiment prediction [Kiritchenko et al., 2014]. Another line of work centers on incorporating target information into various neural network (NN) models, including Recusive NN-based methods [Dong et al., 2014], RNN-based methods [Tang et al., 2016a] and CNN-based methods [Li et al., 2018]. Inspired by the advantages of attention mechanisms in other NLP tasks, many recent studies design different attention-based methods to model the interactions between the target and the context [Wang et al., 2018; Wang and Lu, 2018]. However, these studies fail to consider visual features that may boost these text-based approaches, which are the focus of this paper.

More recently, Xu et al. [Xu et al., 2019] explored the task of aspect-level multimodal sentiment analysis by proposing a multi-hop memory network to model the cross-modality and single-modality interactions. Different from their work, we aim to explore the usefulness of the recent BERT model for both TSC and TMSC in this paper.

2.2 Multimodal Sentiment Classification

With the growth of multimodal data on the Web, information from different modalities (visual, acoustic, etc.) has recently been leveraged to provide complementary sentiment signals to traditional textual features [Zhang et al., 2018a]. Most existing studies in this area focus on sentiment classification in a dialogue. Specifically, Poria et al. (2015) and Poria et al. (2017) respectively propose a multi-kernel learning approach and an LSTM-based sequential architecture to fuse the textual features, the visual features and the audio features. Following this line of work, Zadeh et al. (2017) and Zadeh et al. (2018) further designed a tensor fusion network and a memory fusion network to better capture the interactions between different modalities. However, these methods are designed for the coarse-grained dialogue sentiment classification, which might not be quite effective for our fine-grained target-oriented sentiment classification.

3 Methodology

In this section, we first formulate our task. We then review the standard BERT model, and present our two multimodal extensions of BERT, i.e., mBERT and TomBERT.

3.1 Task Definition

We are given a set of multimodal samples D. For each sample c ∈ D, it contains a sentence S with n words (w1, . . . ,wn) and an associated image I, as well as an opinion target T (a sub-sequence of words in S). For the opinion target T, it is also associated with a sentiment label y, which can be either positive, negative, or neutral. Our problem can be stated as follows: given D as training corpus, our goal is to learn a target-oriented sentiment classifier so that it can correctly predict sentiment labels for opinion targets in unseen samples.

3.2 Background

As mentioned before, since the BERT [Devlin et al., 2018] model can help derive contextualized word representations with pre-trained model parameters from a large corpus and enjoy the capability of learning alignment between two arbitrary inputs, we leverage it as the base model for our task.

To employ BERT for TSC, we propose to transform each sentence S into two sub-sentences: the opinion target T and the remaining context C, and concatenate them as the input sequence for BERT. For example, the BERT input for Fig.1.a is given in Table 1. Formally, let us use X = (x1, x2, . . . , xN) to denote the transformed input sequence, where xi ∈ Rd is the input representation by summing up the word, segment, and position embeddings, and N is the maximum length of the sequence.

Next, let us briefly review the BERT model [Devlin et al., 2018], which is essentially a multi-layer bi-directional Transformer encoder [Vaswani et al., 2017] illustrated in the sentence encoder part of Fig.2.a.

To capture the global information, an m-head self-attention layer is first used to transform each position in the input sequence into a weighted sum of the input layer. Specifically, for the i-th head attention, the input layer X ∈ Rd×N is transformed based on the dot-product attention mechanism [Luong et al., 2015] as follows:

where {WQi , WKi, WVi} ∈ Rd/m×d are learnable parameters corresponding to queries, keys and values respectively. Then, the outputs of the m attention mechanisms are concatenated together followed by a linear transformation as below:

whereWm ∈ Rd×d is the parameter to learn 2 .

Based on the output from the self-attention layer, BERT adds a residual connection from the input to the output, followed by a layer norm (LN) [Ba et al., 2016] as follows:

Moreover, a standard feed-forward network (a.k.a MLP) with GeLU [Hendrycks and Gimpel, 2016] as the activation function and another residual connection with layer norm are stacked on top to generate the output of the first BERT layer:

Finally, the entire model stacks Ls such BERT layers, and the final hidden state of the first token (i.e., [CLS]) is fed to a linear transformation function for classification.

3.3 Multimodal BERT (mBERT)

As illustrated in Fig.2.a, an intuitive but general solution to incorporate the associated image into the BERT architecture is to directly concatenate the image features with the final hidden states of the input sequence, followed by stacking additional BERT layers on top to model the inter-modal interactions between visual and textual representations.

2 We ignore the bias term to avoid confusion in this paper.

Specifically, for the associated image I, we first resize it to 224×224 pixels, and then adopt one of the state-of-theart image recognition models ResNet-152 (res5c) [He et al., 2016] to obtain the output of the last convolutional layer:

which essentially splits the original image into 7×7 = 49 regions and each region is represented by a 2048-dimensional vector rj. Next, we use a linear transformation function to project the visual features to the same space of textual features: G = WvResNet(I), where Wv ∈ Rd×2048 is the learnable parameter.

Furthermore, the image features G ∈ Rd×49 and the textual features HS ∈ Rd×N (i.e., the hidden states in the final layer, BTLs (X)) are concatenated together, followed by feeding them to the Multimodal Encoder, which contains another set of BERT layers to automatically model the rich interactions between textual and visual features:

where Lm is the number of layers in the Multimodal Encoder. Finally, the final hidden state of the "[CLS]" token is used for classification.

3.4 Target-oriented mBERT (TomBERT)

Although the above mBERT model is expected to well capture the inter-modality dynamics, its main limitation lies in the fact that its visual representation is insensitive to the opinion target, since the visual features for the same input sentence are always the same regardless of the target it considers.

Intuitively, with a specific opinion target as input, it is often the case that only some regions of the associated image are closely related to it, and the other regions should be ignored to eliminate the noise. For example, in Fig.1.a, with "Georgina Hermitage" as the target, we should mainly focus on her smiley face, and ignore the other background. In contrast, with "400m T37" as the target, we should only focus on the regions containing "Women's 400m T37". In this case, if our model takes the entire image into account and mistakenly pays attention to the smiley face, it is highly probable that the model will make an incorrect prediction.

Inspired by this, we design a Target-Image (TI) Matching layer, which employs an m-head target attention mechanism to perform matching between the target and the image to obtain a target-sensitive visual representation. As shown in Fig.2.b, we first apply another BERT encoder to get the hidden representation of the opinion target: HT = BTLc (T), where T ∈ Rd×M and M are respectively the target input representation and the maximum length of the target, and Lc is the number of layers. Next, we treat the hidden states of the target HT as queries, and the regional image features G as keys and values, such that the target is leveraged to guide the model to align it with the appropriate regions, i.e., only assigning high attention weights to the image regions that are closely related to the target. Specifically, the i-th head target attention takes the following form:

where {WQ′i , WK′i , WV′i } ∈ Rd/m×d are parameters. Furthermore, similar as BERT, we adopt a feed-forward network and two layer norms with residual connections to obtain the target-sensitive visual output:

We then stack Lt such TI Matching layers to obtain the final visual representation: HV = TILt (G, T), where HV ∈ Rd×M and each hidden state in HV is essentially a weighted sum of the 49 regions in the associated image.

Next, to form the multimodal input representations, we consider two concatenation types as below:

• All-Text: directly concatenate HV and HS;
• First-Text: only consider the final state of the first element (i.e., the special [CLS] token in the target input) in HV, and concatenate H0V with HS.

Similar to mBERT, we further add a multimodal encoder on top to obtain the final multimodal hidden representations:

To integrate the visual and textual representations for final classification, we consider the following three pooling types to obtain the final output:

• FIRST: The first token of the multimodal input sequence is always a weighted sum of the 49 regional image features. Its final hidden state is regarded as the aggregate multimodal sequence representation with visual representations as queries, and thus can be taken out as the output: O = H0 ;
• CLS: Similarly, the final hidden state for the special token (i.e., [CLS] token in the sentence input) is the aggregate representation with textual representations as queries, and can also be used as the output: O = H[CLS] ;
• BOTH: We concatenate these two hidden states as the hybrid output: O = [H0 , H[CLS] ].

Finally, we feed O to a linear function followed by a softmax function for target-oriented sentiment classification:

whereW ∈ R(2)d×3 is the learnable parameter. To optimize all the parameters in our TomBERT model, the objective is to minimize the standard cross-entropy loss function as below:

4 Experiments

In this section, we carry out extensive experiments to answer the following research questions:

• RQ1: Can the fine-tuned BERT model outperform stateof-the-art text-based approaches on both existing benchmark datasets and our two datasets? (Section 4.2)
• RQ2: Is the associated image generally useful for TSC? Could our TomBERT model bring significant improvements to BERT and achieve the best performance on our two multimodal datasets? (Section 4.2)
• RQ3: What is the effectiveness of TomBERT with respect to the pooling layer, the multimodal concatenation layer, and the number of hidden layers? (Section 4.3)
• RQ4: What is the key advantage of TomBERT over other highly competitive approaches? (Section 4.4)

4.1 Experiment Settings

To evaluate the effect of BERT and TomBERT, we adopt three benchmark datasets for TSC (i.e., LAPTOP and REST from SemEval-2014 Task 4 [Pontiki et al., 2014] as well as TWITTER-14 constructed by [Dong et al., 2014]) and two multimodal datasets for TMSC (i.e., TWITTER-15 and TWITTER-17 respectively collected by [Zhang et al., 2018b] and [Lu et al., 2018]). LAPTOP and REST consist of Amazon customer reviews in laptop and restaurant domains respectively, and the three TWITTER datasets include user tweets posted during 2010-2014, 2014-2015 and 20162017, respectively. Since the two publicly available multimodal datasets TWITTER-15 and TWITTER-17 only provide annotated targets (i.e., entities) in each tweet, we ask three domain experts to annotate the sentiment towards each target, and take the majority label among the three annotators as the gold label. For space limitation, we only show the basic statistics of TWITTER-15 and TWITTER-17 in Table 2.

We build our TomBERT model on top of the pre-trained uncased BERTbase model released by [Devlin et al., 2018], and tune the hyper-parameters on the development set of each dataset. Specifically, for BERT-based models, we set the learning rate as 5e-5, the number of attention heads as m = 12, and the dropout rate as 0.1. The batch size is respectively set as 16 and 32 for all the models for TSC and TMSC, respectively. Besides, for TomBERT, the maximum length of the sentence input and the target input are respectively set as N = 64 and M = 16. The number of layers for encoding the sentence input and the target input are both set to be 12, i.e., Ls = Lc = 12, where the parameters are both initialized from the pre-trained BERTbase model. All the models are fine-tuned for 8 epochs, and are implemented based on PyTorch with a NVIDIA Tesla V100 GPU.

4.2 Main Results

Performance of Fine-Tuned BERT (RQ1)

To demonstrate the effect of the fine-tuned BERT for TSC, we first compare it with a number of representative approaches: 1). SVM [Kiritchenko et al., 2014], including many carefully designed linguistic features; 2). AE-LSTM [Wang et al., 2016], incorporating aspect embeddings and target-specific attention mechanism; 3). TD-LSTM [Tang et al., 2016a], using two LSTMs to model the left context and the right context of the target respectively; 4). IAN [Ma et al., 2017], proposing an interactive attention mechanism to model the interactions between the target and the context; 5). MemNet [Tang et al., 2016b], applying a multi-hop attention mechanism on top of word embeddings and position embeddings with targets as queries; 6). RAM [Chen et al., 2017], constructing a neural architecture by applying a GRU model on top of the representations obtained from multi-hop attention mechanism; 7). TNet [Li et al., 2018], adapting CNN with targetspecific transformation to integrate the target and the context; 8). MGAN [Fan et al., 2018], building up a multi-grained attention network for fusing the target and the context.

We report the accuracy (ACC) and Macro-F1 score of textbased methods on all the five datasets in Table 3 and the text modality part of Table 4. It is easy to find that BERT consistently outperforms all the baselines, which supports our first motivation that the pre-trained model can lead to better optimal solutions, and thus bring improvements for TSC.

Performance of TomBERT (RQ2)

We then consider the following highly competitive approaches for evaluating our TomBERT model: 1). Res-Target: concatenating HT and the max-pooling of G; 2). BERT+BL, adding another BERT layer on top of BERTbase, which is similar to MBERT but without visual features. 3). Res-MGAN, a simple combination of textual and visual contents by concatenating the max pooling of G with the hidden representation of MGAN; 4). Res-MGAN-TFN, using Tensor Fusion Network (TFN) [Zadeh et al., 2017] to fuse the textual and visual representations in Res-MGAN with rich interactions; 5). ResBERT+BL and Res-BERT+BL-TFN, replacing the textual encoder in Res-MGAN and Res-MGAN-TFN with BERT+BL; 6). mBERT (All-Text), the model detailed in Section 3.3; 7). mPBERT, a variant of mBERT, which uses the max pooling of G as input visual features and one of the three pooling types to obtain the final output; 8). TomBERT (All-Text), our model detailed in Section 3.4, where we use All-Text concatenation and CLS pooling for the final output; 9). TomBERT (FIRST, CLS, or BOTH), our model with First-Text concatenation and one of the three pooling types for the final output.

Based on Table 4, we can make a couple of observations: 1). The perforamnce of Res-Target is quite limited, which implies that the associated images only play a supporting role to text, and cannot be treated independently for targetoriented sentiment prediction; 2). Due to the higher capacity of model parameters, BERT+BL brings minor improvements over BERT; 3). Res-MGAN and Res-BERT+BL can generally boost the performance of MGAN and BERT+BL, indicating that the associated images are generally useful to enhance text-based approaches; 4). Interestingly, although TFN learns rich interactions between modalities, it even drops the performance of Res-MGAN and Res-BERT+BL. This suggests that it is hard for the complex fusion matrix to directly capture the interactions between two modalities; 5). mBERT

Associated Image Input Sentence & Predicted Label Associated Image Input Sentence & Predicted Label

Join [iKON] 1 , at their 1 night only, first ever [SG] 2 concert. Go at 3pm!

Human Label: (1-positive, 2-neutral) BERT+BL: (neutral 7, positive 7) mPBERT: (neutral 7, positive 7) TomBERT: (positive 3, neutral 3)

Happy birthday to [Henry Ford] 1 , who was born 150 years ago today ! (Photo: [Getty Images] 2 )

Human Label: (1-positive, 2-neutral)

BERT+BL: (positive 3, positive 7)

mPBERT: (positive 3, positive 7)

TomBERT: (positive 3, neutral 3)

Table 5: Predictions of BERT+BL, mPBERT, and TomBERT on several test samples. 7 and 3 denote incorrect and correct predictions.

TWITTER-15

TWITTER-17

and mPBERT can outperform BERT+BL in most cases, and generally have better performance than Res-BERT+BL-TFN, which shows that the top multimodal encoder can well capture the inter-modality interactions; 6). Finally, regardless of the concatenation and pooling types we use, TomBERT consistently achieves the best results on the two datasets, and most of these gains are significant with p<0.05. These observations support our second motivation that TomBERT can well capture intra-modality and inter-modality dynamics.

4.3 Further Analysis of TomBERT (RQ3)

To answer the questions in RQ3, we investigate the effect of different components in both mBERT and TomBERT.

First, comparing the three pooling types in Table 4, we observe the following: 1). no matter which pooling type we use, TomBERT can generally perform better than mPBERT. Since the only difference between them is the TI Matching module, this suggests that our target attention mechanism is able to generate target-sensitive visual representations, which can lead to significant performance gain; 2). For mPBERT, since its visual representations are not target-sensitive, it is reasonable that using its final hidden state (i.e. FIRST) results in limited performance. In contrast, using CLS and BOTH can learn to pay more attention to textual representations, and have much better results; 3). For TomBERT, since visual and textual representations are both target-sensitive, it is intuitive that all the three pooling types can result in promising results.

Moreover, for the two concatenation types, we can see from Table 4 that incorporating all the visual features (i.e., All-Text) generally leads to slightly worse performance than abstracting all the visual features into a single vector except mPBERT (FIRST). This further confirms that in TMSC, the images are used to support the text for target sentiment detection, and paying too much attention to the visual features may bring some noise and drop the performance.

Finally, since the Multimodal Encoder (ME) and TI Matching (TIM) modules in mPBERT and TomBERT may stack multiple layers, we analyze the impact of their layer number Lm and Lt. As shown in Fig. 3, for ME, mPBERT and TomBERT can achieve the best results when Lm = 4; while for TIM, TomBERT performs the best when Lt is around 5. When further increasing Lm and Lt, the result becomes worse probably due to the increase of model parameters.

4.4 Case Study (RQ4)

To better understand the advantage of TomBERT, we further group test sentences with single target and multiple targets into two categories, and report the results on them in Table 6. It is easy to observe that TomBERT performs significantly better than BERT+BL and mPBERT when input sentences have multiple targets, which is in line with our motivation.

Furthermore, we select several representative test samples to compare the predictions of different methods. In the left side of Table 5, we can see that although BERT+BL and mPBERT incorrectly predict the sentiment over IKON and SG. With the help of the image, our TomBERT model can identify that the focus of the tweet is the band IKON other than SG, and therefore predict the sentiment over the focused target IKON as positive, and the other target SG as neutral. Similar observations can be made on the targets Henry Ford and Getty Images in the right side of Table 5.

5 Conclusion

In this paper, we study Target-oriented Multimodal Sentiment Classification (TMSC), and propose a Target-oriented multimodal BERT (TomBERT) architecture to effectively capture the intra-modality and inter-modality dynamics. Extensive evaluations on five datasets for TSC and TMSC demonstrate the effectivenss of BERT and our TomBERT model in detecting the sentiment polarity for individual opinion target.

Acknowledgments

This research is supported by the National Research Foundation, Prime Minister's Office, Singapore under its International Research Centres in Singapore Funding Initiative.