Multi-Domain Sentiment Classification Based on Domain-Aware Embedding and Attention

1 Introduction

Sentiment classification is one of the most fundamental tasks in Natural Language Processing. Over the past decades, many supervised machine learning methods such as Naive Bayes, Support Vector Machines, Neural Networks are applied to this task [McCallum et al., 1998; Choi and Cardie, 2008; Kim, 2014; Yang et al., 2016; Zhang et al., 2018]. However, sentiment classification models are highly domaindependent, resulting in the demand of a large amount of training data for each domain. The reason is due to that there are usually different words and expressions in different domains and moreover, even the same word in different domains may reflect different sentiment polarities. For example, the word easy often exists when the sentence conveys positive sentiment in the domain of baby products (e.g. It is easy for her to hold ...). But in the domain of movie reviews, easy sometimes may express negative sentiment (e.g. The ending of this movie is easy to guess.). Similarly, the word infantile cannot show speakers' sentiment when they make a comment on baby products, while this word can convey strong negative sentiment when it comes to book reviews or movie reviews.

Therefore, it is worthwhile to leverage available resources across all domains to improve the performance of sentiment classification on some certain domains. One approach is to first pre-train models on large unlabeled data, and then finetune those models like BERT [Devlin et al., 2018] and OpenAI GPT [Radford et al., 2018].

Another approach is multi-task learning [Caruana, 1997], which is the focus of this work and effective in improving the performance of one task with the help of related tasks. Some existing Share-Private models exploit multiple LSTMs, CNNs, Memory Networks and Fully-Connected Networks to represent shared and private layers. Shared layers are trained across all data, while private layers are trained on data from a certain domain [Liu et al., 2016a; Liu et al., 2017; Chen and Cardie, 2018]. However, training these models are difficult and time-consuming due to their large number of parameters. The performance of these models are not very well. Another Share-Private model utilizes shared sentence encoders but private query features to select domain-specific information from shared sentence representation [Zheng et al., 2018]. However, this model only focuses on detecting the important words but ignores identifying sentiment polarity of the same word in different domains.

In order to solve the above problem, we propose a novel completely-shared multi-task learning model in this work. The key characteristic of our model is that it can learn domain-aware word embeddings and make use of domainaware attention mechanism with completely-shared parameters across different domains. Our model first utilizes BiLSTM for domain classification and extracts domain-specific features for words, which are then combined with general word embeddings to form domain-aware word embeddings. Domain-aware word embeddings are fed into another BiLSTM to extract sentence features. The domain-aware attention mechanism is used for selecting significant features by using domain-aware query vector. As can be seen, we share the general word embeddings across different domains, but learn domain-specific word features for each domain. The use of domain-aware word embeddings and domain-aware attention makes the final representation of an input text in a domain contain both shared information across domains and private information in the specific domain.

Evaluation results on public datasets with 16 different domains demonstrate the efficacy of our proposed model, which achieves state-of-the-art performance on the multi-domain sentiment classification task. Further experiments show the generalization ability and the transferability of our model. Taking sentences with the word easy as example, our model can not only focus on the word easy for sentiment classification, but also identify different meanings of it according to different domains and different instances. 澳 澳

Our main contributions are summarized as follows:

• We propose a novel completely-shared neural model based on domain-aware word embeddings and domainaware attention mechanism to make use of training data among all domains for multi-domain sentiment classification. Our model can not only focus on the significant words in texts but also distinguish their sentiment polarity with the help of domain classifier. 澳 澳
• We conduct experiments on datasets with 16 different domains. The results of multi-domain sentiment classification show the effectiveness of our model. 澳
• We further perform experiment on cross-domain sentiment classification and knowledge transferring. Results reveal the generalization ability and transferability of our model. Our code will be released. 澳

2 RelatedWork

Multi-task learning (MTL) [Caruana, 1997] is a popular method to solve multi-domain text classification or sentiment classification. Many works of deep learning based multitask learning [Zhang et al., 2014; Liu et al., 2016b] share first several layers for each task to extract low-level features and generate outputs with task-specific parameters. Recent works apply adversarial training to multi-task learning. Liu et al. [2017] ,Chen and Cardie [2018], Liu et al. [2018] and Li et al. [2017] use LSTMs, CNNs and memory networks to extract features and attempt to confuse domain classifiers by maximizing cost function in order to guarantee the taskindependence of features.

Attention mechanism has become popular since it enables models to focus on the more important words and phrases. Bahdanau et al. [2014], Luong et al. [2015], Vaswani et al. [2017] apply attention to the machine translation task to capture long distance dependency. Yang et al. [2016] apply attention at word level as well as sentence level so as to introduce importance of sentences to document classification. In multi-task learning, Zheng et al. [2018], Yuan et al. [2018] share feature extractors but use attention mechanism with task-independent query vectors to generate different representations for different tasks.

Language model[Bengio et al., 2003] can express context information of a word compared to general word embedding. ELMo [Peters et al., 2018] and BERT [Devlin et al., 2018] have proven the reliability of language model containing context information. Inspired by these works, we exploit the output of an LSTM for domain classification to capture domain and context information.

3 ExistingMTLModels

澳

澳

澳 澳 Recently, many researches focus on multi-domain sentiment classification or text classification. We will introduce several latest models, which are also considered strong baselines for comparison.

澳 Figure (1a) is the architecture of Fully-Shared Multi-Task Learning framework (FS-MTL). This framework is one of the most common multi-task learning frameworks. There is a shared encoder to extract low-level features for all domains, and then a domain-specific classifier is used to make prediction for each domain. However, this model cannot extract domain-specific features since feature extractors are trained on all domains.

澳 Many share-private models utilize multiple encoders to extract domain-specific and domain-independent features. Shared encoders are trained on all training data and private encoders are trained on domain-specific data. Adversarial training is applied to enable shared encoders to extract domain-independent features. ASP-MTL (Figure 1b) [Liu et al., 2017] and DSR-at (Figure 1c) [Liu et al., 2018] are two specific models of this share-private scheme. ASP-MTL uses another encoder as private feature extractor while DSRat uses memory network. However, if the number of domains increases, these models require large space and cost a lot of time to train.

Another share-private model DA-MTL (Figure 1d) [Zheng et al., 2018] uses a shared BiLSTM encoder to extract lowlevel features. This model first builds domain representations. Domain representations are used for attention to select domain-specific sentiment information from the shared sentence representation. This model can focus on more significant words and is space-efficient, but cannot represent different meanings and different sentiment polarities of the same word in different domains.

4 Our ProposedModel

The goal of our proposed model is to pay more attention to significant words in a text and identify their sentiment polarities in each specific domain. Our model consists of two parts: domain feature extraction and classification, sentiment feature extraction and classification. The first part aims to extract domain-specific features with a BiLSTM and a domain classifier. This part attempts to capture domain information and context information in the given text. Therefore, the training loss of the domain classifier is one part of the total training loss in order to guarantee the BiLSTM to extract domain information. The second part aims to get the text representation for sentiment classification. It first forms domain-aware word embedding by combining domain-specific features and general word embedding, and then obtain the initial representation of the text by a BiLSTM with the domain-aware word embedding. After that, it uses a domain-aware attention mechanism to refine the initial representation and obtains the final representation for sentiment classification. In the attention mechanism, different query vectors are formed for different texts in different domains, aiming at paying attention to important words for sentiment classification. The training loss for sentiment classification and the loss for domain classification are added up, making two objectives learned simultaneously. The architecture of our proposed model is shown in Figure 2. Note that all the parameters in the two parts are completely shared across different domains. We will introduce key components in the following subsections, respectively.

4.1 Domain Feature Extraction and Domain Classification

Representation of each word that involves domain information and context information is required. After the general word embeddings for a sequence of words in a text are taken as input in a BiLSTM, the output vectors of the BiLSTM at different time steps are considered to contain context information of the text. The output vectors are then combined by average pooling to get the text representation, and the training objective of predicting the domain correctly by the text representation forces the BiLSTM to extract domain information. We exploit BiLSTM as domain feature extractor and a fullyconnected network as domain classifier. The output vectors of BiLSTM at different time steps are considered domainspecific features for the words. The formulas can be written as follows.

where -→ hft and ←- hbt are forward and backward LSTM output vectors at time step t. et is the word embedding for the t-th word. θf and θb are parameters of two LSTMs. ⊕ is the concatenation operation. E is the length of text. d is the predicted probabilities (a vector) of domains for the text. W1, W2, b1 and b2 are weight matrices and bias vectors. Ld is the cross entropy loss function of domain classifier and N is the number of domains. ˆ d is the gold probability distribution over domains.

4.2 Domain-AwareWord Embedding

We attempt to form domain-aware word embedding that contains domain features as well as context information for sentiment feature extraction. Therefore we combine domain features mentioned in Section 4.1 and general word embedding for each word. The formulas of building domain-aware word embedding xt are listed as follows.

where W3, b3 are weight matrix and bias vector and the linear transformation aims to extract higher-level domainspecific information. ht is the output vector of domain feature extractor, et is the word embedding for the t-th word, as mentioned in Section 4.1, ⊕ is concatenation operation.

Domain-aware word embedding xt is then fed into another BiLSTM to extract sentiment features.

4.3 Domain-Aware Attention and Sentiment Feature Extraction

Our proposed model generates query vectors in the attention mechanism dynamically, expecting the model can leverage domain information to select important words in a text. The formulas of the attention mechanism are listed below. In the attention mechanism, we apply additive attention instead of dot-product attention, since the performance of former attention is better.

where ot is the output vector (by concatenating the forward and backward output vectors) of the BiLSTM for extracting sentiment features, as mentioned in the last paragraph of Section 4.2. htext is the vector mentioned in Equation (1), αt is the weight of the t-th word, E is the length of text, and otext is the weighted sum of output vector ot, which is the extracted sentiment features for sentiment classification. W4, W5, b4, b5 are weight matrices and bias vectors.

4.4 Sentiment Classification

We use a two layer fully-connected neural network as our final sentiment classifier rather than several classifiers corresponding to different domains. The loss function Ls is cross entropy between the predicted labels and true labels.

The formulas can be written as follows.

where W6, W7, b6, b7 are weight matrices and bias vectors, p is the probability of predicting the text as positive, and y is 1 if the gold label is positive and 0 otherwise.

The total loss of our model can be computed as follows:

where γd and γs are weights of loss functions Ld and Ls, respectively.

5 Experiments

5.1 Datasets

We use the datasets released by [Liu et al., 2017] for multidomain sentiment classification, which consist of product and movie reviews in 16 different domains. The data in each domain is randomly split into training set, development set and test set according to the proportion of 70%, 10%, 20%. Statistics of the 16 datasets are listed in Table 1.

5.2 Training Details

We initialize word embedding with 200-dimension Glove vectors [Pennington et al., 2014]. The word embedding is fixed during training. Other parameters are initialized by sampling from normal distribution whose standard deviation is 0.1. The minibatch is 128. Each batch contains 8 samples from every domain. We use Adam optimizer [Kingma and Ba, 2014] with an initial learning rate of 0.004. The hidden size of each LSTM is 128. Weights γd and γs of domain classification loss and sentiment classification loss are set to 0.1 and 1 respectively after a small grid search over [1, 0.1, 0.05]. To alleviate overfitting, we use dropout with probability of 0.5 and L2 regularization with parameter of 1e-8. We train the domain classifier and the sentiment classifier jointly at first. After 10 epochs, we only train the sentiment classifier, setting γd to 0. At last, we finetune our model on each task.

5.3 Multi-Domain Sentiment Classification

We first conduct experiments of multi-domain sentiment classification. Models are tested on 16 test sets, respectively. Latest models mentioned in Section 3 are used for comparison. In addition, we also compare with the following baselines.

• Single Task. BiLSTM, BiLSTM with attention mechanism (att-BiLSTM) and text CNN are three of the most popular models designed for the sentiment classification task. The model is trained on each domain independently.
• BERT. BERT [Devlin et al., 2018] is a pre-trained language model with deep bidirectional transformer. Finetuning BERT model can outperform state-of-the-art models among many tasks, including text classification. We use the pre-trained BERT-base model available online and finetune it on each task.
• SA-MTL. SA-MTL [Zheng et al., 2018] is the simplified version of DA-MTL. It shares BiLSTM as low-level feature extractor. This model uses different query vector for each domain rather than generating query vector for each instance. The query vector is updated during the training process.

Results of our proposed model and baseline models are listed in Table 2. The table shows that both pre-trained language models and multi-task learning models can improve the accuracy of multi-domain sentiment classification a lot. Multi-task learning baseline models except FS-MTL achieve similar performance with BERT model, but they demand less space and are easier to train. Our proposed model outperforms other models by at least 1.9 points on accuracy.

5.4 Cross-Domain Sentiment Classification

In the task of cross-domain sentiment classification, models are trained on the training data in 15 domains and tested on the test data in the remaining one domain. It means we do not use the sentiment labels of the training data in the test domain, but we still use the data as unlabeled data for domain classification. In other words, all training data in 16 domains are used for training domain feature extractor and domain classifier, but only the training data in 15 domains are used for training sentiment feature extractor and sentiment classifier. We assume that the combination of domain features as well as general word embedding contains domain and context based sentiment features, which is beneficial for sentiment classification and improves the generalization ability of the sentiment classifier.

Previous models ASP-MTL and DSR-at mentioned in Section 5.3 can also be adapted for cross-domain sentiment classification. In ASP-MTL, we use average pooling to combine outputs from all domains. As for DSR-at, the test data is predicted by 15 memory networks, after which the majority voting is applied to classify the sentiment polarity of the test data.

The results are shown in Table 3. Compared to Table 2, the accuracy of cross-domain sentiment classification is worse than that of multi-domain sentiment classification, revealing that the sentiment classification task is highly domaindependent. Our proposed model outperforms DSR-at by 1.8 points on accuracy and outperforms ASP-MTL by 3.3 points on accuracy, demonstrating the generalization ability of our model.

5.5 Transferability Test

We assume that the shared domain feature extractor can learn enough knowledge about domain and context to be applied to unseen domains. Therefore, we test the transferability of our model, following the transfer learning setting of [Zheng et al., 2018]. We first use the last 10 domains to train the model with multi-task learning. We then keep the parameters of domain feature extractor fixed and finetune other parameters.

We compare our model with previous models SA-MTL and DA-MTL. These two models fix their sentence encoders after pre-training it on the last 10 tasks. We also compare models with transfer learning mentioned above and models with multi-task learning based only on the first 6 domains.

In our model, we exploit domain feature extractor to collect domain information as well as context information. Recently, the success of finetuning language models and applying their features to text classification has demonstrated that pre-trained language models can better capture context information of sentences. Therefore, we introduce BERT features to our model to enrich the context information.

We adapt Equation (2) as follows.

where ˜ ht, ht and so on are the same as that in Equation (2), Bt is the output of BERT model for the t-th word and ˜ Bt is the linear transformation of Bt. xt is the concatenation of ˜ Bt , et, and ˜ ht .

We also introduce BERT features to SA-MTL and DAMTL. In SA-MTL, domain query vectors are replaced by sentence representations extracted by the final layer of BERT. In DA-MTL model, we concatenate generated instance specific query vector and its BERT features as new query vector.

Table 5 shows the results of original models and models with BERT features. The improved performance of models with BERT features shows the positive effect of more context information. The improvement of SA-MTL is larger than DA-MTL and our proposed model, revealing that DA-MTL and our model have already captured more context information than SA-MTL.

6 Visualization Analysis

Figure 3 shows the attention in our model for some running examples. In Figure (3a), our model attends to the words "easy", "rigth"(right), "recommend" and phrase "very well done" to predict positive sentiment for the text, because the phrase "easy to follow", "models are just right", "recommend it" and "well done" convey strong positive polarity to the products. In Figure (3b), our model pays attention to "repeating himself", "getting less and less", "easy" and "spoiling" and succeeds in making correct prediction that the text conveys negative sentiment. In Figure (3c), words "perfect", "easy", "best", "excellent" are selected as the evidences of sentiment prediction. Examples above show that our model can choose words and phrases correctly, demonstrating the effectiveness of regarding domain features as query vectors of attention. Besides, in Figure (3a), (3b), (3c), our model succeeds in predicting correct sentiment while selecting the same word "easy", revealing that domain information and context information are beneficial for some instances.

7 Conclusion

In this paper, we propose a novel completely-shared neural model to make use of different training data across all domains. Our model builds domain-aware word embedding to express domain and context information for words, and proposes domain-aware attention mechanism to focus on more significant words in the text. Experiments on multi-domain sentiment classification and cross-domain sentiment classification on 16 different domains demonstrate the effectiveness and advantages of our proposed model.

Acknowledgments

This work was supported by National Natural Science Foundation of China (61772036) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.