Transition-based Adversarial Network for Cross-lingual Aspect Extraction

1 Introduction

Different from coarse-grained sentiment classification which predicts an overall sentiment polarity for each sentence, finegrained opinion mining involves the identification of aspect terms describing product features, which is important for information extraction. However, for this task, a crucial issue is the difficulty in collecting sufficient labeled data to train a precise classifier, especially for resource-limited languages. Existing cross-lingual approaches mainly focused on coarsegrained sentiment classification. To the best of our knowledge, there is yet no studies on cross-lingual aspect extraction, due to the difficulties in word-level feature adaptation.

Aspect extraction in supervised mono-lingual setting has been studied extensively [Qiu et al., 2011; Liu et al., 2015; Yin et al., 2016; Wang et al., 2016; 2017]. Given a sentence e.g., "We love the pink pony and atmosphere.", the aspect terms to be extracted are pink pony and atmosphere.

Most existing work treat this task as a sequence labeling problem that predicts a label for each token. It has also been shown that syntactic relations are crucial to identify target terms [Qiu et al., 2011; Yin et al., 2016; Wang et al., 2016]. Token-level knowledge transfer has been proposed for cross-domain aspect extraction [Li et al., 2012; Ding et al., 2017], where the models are built upon common sentiment lexicons. However, these methods only work for homogeneous space within the same language. Moreover, token-level knowledge is much more difficult to be transferred across languages, because the segmentation rules may vary in different languages. For instance, an aspect can be expressed by a single word in one language but by multiple words in another language. To conquer this limitation, we adopt transition-based models, which have been used for dependency parsing [Dyer et al., 2015] and named entity recognition [Lample et al., 2016]. Different from sequence labeling, transition-based models learn a sequence of actions that read words sequentially from a buffer and avoid the separation of a phrase. These actions depend on the configuration representation at the previous timestamp. To transfer knowledge, we aim to learn a shared space for configurations from different languages.

Recently, adversarial networks [Goodfellow et al., 2014] have been widely used and shown promising results for domain adaptation [Ganin et al., 2016]. Chen et al. [2016] applied the domain adversarial network to cross-lingual sentiment classification which learns a domain discriminator for sentence representations. However, fine-grained adaptation is much more difficult than sharing of sentence representations. To make adaptation successful, we propose a transition-based adversarial network, where a generator is used to produce language-invariant configuration features, and a discriminator competes to discriminate the configurations between source and target languages. Since dependency relations have proven to play an important role in identification of aspect terms, and are invariant across different languages when using universal dependency parsers [de Marneffe et al., 2014], we build partial dependency trees when computing the configuration representations to capture the key components during transfer. We also use these relations as auxiliary labels that helps the adversarial network to learn better representations. Our main contributions are three-fold: 1) We propose a novel transfer strategy that uses sentence configuration as invariant features for cross-lingual word predictions. 2) We integrate syntactic information to assist knowledge transfer across different languages. 3) We conduct extensive experiments to show the effectiveness of our proposed model.

2 RelatedWork

Aspect extraction has been actively studied for English corpus. Early works have applied unsupervised methods to extract the target words using association rules mining [Hu and Liu, 2004] and manually designed rules based on syntactic relations [Qiu et al., 2011]. Several supervised approaches have later been proposed including HMMs with feature engineering [Jin and Ho, 2009] and deep learning [Liu et al., 2015; Yin et al., 2016; Wang et al., 2016]. These supervised methods all treat the problem as a sequence labeling task that work on each token in a sentence. For cross-domain aspect extraction, Li et al. [2012] proposed a boosting-style method to expand target lexicon through common sentiment words and syntactic relations across domains. Ding et al. [2017] developed a deep model that considers structural correspondences. The above methods depend on common sentiment lexicon, which is not available for heterogeneous spaces across languages.

For cross-lingual sentiment classification, one approach is to use machine translation to translate into source or target language to build a classifier for target language [Wan, 2009]. Another approach is to use parallel corpora or word pairs to learn shared features [Prettenhofer and Stein, 2010; Meng et al., 2012; Zhou et al., 2015]. Bilingual representation learning has also been widely studied [Klementiev et al., 2012; Hermann and Blunsom, 2014; Chandar et al., 2014; Gouws et al., 2015; Zhou et al., 2016]. Moreover, Zhou et al. [2014] used marginalized stack denoise auto-encoder to align document representations from heterogeneous spaces. Chen et al. [2016] applied domain adversarial network [Ganin et al., 2016] to train language-independent document representations. Among all these, fine-grained-level adaptation has not been investigated yet.

3 ProblemDefinition &Motivation

The task studied in this paper involves the identification of explicit aspect terms in each review sentence. Formally, given a sequence of words x=(w1, w2, ..., wn), the transition model reads words sequentially and outputs a sequence of actions y=(y1, y2, ..., ym), resulting in a sequence of configurations c = (c1, c2, ..., cm), where m is the number of timestamps. Each yt ∈ A, where A = {OUTPUT, SHIFT, REDUCE}. Each ct = (cot , cct , cbt ) consists of three components that partition the whole sentence: output stack (cot ), candidate stack (cct ) and buffer (cbt ), where each component consists of a subsequence of tokens (or empty). The aspect terms are the complete segments in cct when yt+1=REDUCE.

In transition-based models, each configuration ct represents a partition of the whole sentence at timestamp t. The concept of configuration is our primary motivation for knowledge transfer across different languages. Since each configuration summarizes the current status and decides on the next action, by aligning them from different languages into a shared space, we can transfer the knowledge of how to take actions in the target language. Compared with tokenlevel representation, the configuration represents the global information including the entire history processed and the future ahead. The adaptation within a transition system is more aligned with cognitive modeling: a reader makes incremental actions in the same way regardless of different languages. Moreover, token-level predictions require accurate learning for different positions within word segments, e.g., the prediction for service staff should be "B" (beginning of aspect) for service and "I" (inside of aspect) for staff. However, the translation is personnel de service in French, which consists of 3 words. When the segments differ, adaptation becomes noisy and inaccurate. This could be addressed using transition systems, which use a candidate stack to store and represent a complete aspect segment to be transfered across languages.

It has been shown that universal dependency relations are invariant across different languages. Since Dyer et al. [2015] showed the effectiveness of a transition model for dependency parsing by predicting the transition or relation at each timestamp according to the current configuration, we take this sequential relation prediction as an auxiliary task to our main task. Moreover, motivated by auxiliary conditioned generative adversarial network (ACGAN) [Odena et al., 2017], we integrate this auxiliary task into the transition-based adversarial network to make the transfer more coherent with the language-invariant syntactic structure. In addition, we integrate syntactic structure into the representation of each configuration. This helps to make the representations more sensitive to the important words in the sentence. For example, if the stack consists of "We love", the dependency tree will output the representation for love, which is the parent of We. This construction is able to attend on the important word love.

4 The ProposedModel

We name our proposed model Transition-based Adversarial Network (TAN) in the sequel. Figure 1 shows the overall structure which consists of a feature extractor (F) and three classifiers: an action classifier (Y), an auxiliary predictor (A) and a domain discriminator (D). At timestamp t, the input sequence is partitioned into an output stack, a candidate stack and a buffer. Each component computes its own representation through a recurrent neural network (RNN) in the last layer, which are then integrated into the configuration representation that is fed into all three classifiers. In the following, we first introduce a basic transition-based model in Section 4.1, then illustrate the integration of an adversarial component in Section 4.2, and finally present the final model by considering the syntactic information in Section 4.3.

4.1 Transition-basedModel

The transition-based method is formalized as a sequence of actions that read words sequentially from a buffer [Dyer et al., 2015]. It consists of an output stack, a candidate stack and a buffer. The output stack stores the processed words, the candidate stack stores incomplete aspect candidates until all the words for an aspect are processed. The buffer contains the words that have yet to be processed. There are three possible transition actions: Output, Shift, and Reduce.

1. Output - Move the word on top of the buffer to the output stack and label it as "None".
2. Shift - Move the word on top of the buffer to the candidate stack.
3. Reduce - Move every word in the candidate stack to the output stack and label them as "Aspect".

The algorithm starts with the buffer containing every word in the sentence. At each configuration, one action is taken, until the algorithm terminates when both the candidate stack and buffer are empty. An illustration of the process is shown in Figure 2. The next action depends on the current configuration, which is computed from the representations of the 3 components: output stack, candidate stack and buffer. We use Gated Recurrent Unit (GRU), one of the RNN architectures to compute the representation of each component. A GRU takes the word embedding as the input and produces the output of the last cell to represent the corresponding subsequence, as shown in Figure 3 on the left. We take the reverse GRU to compute the buffer representation and use another vector to represent empty stack when the candidate stack is empty.

Formally, let xOt , xCt and xBt denote the input sequence for output stack, candidate stack and buffer, respectively, at timestamp t. Denote the hidden representations by hOt for output stack, hCt for candidate stack, and hBt for buffer. The final feature representation for the configuration is

concatenation of vectors, the model fGRU produces the last hidden representation of a forward GRU network, and the model fRGRU indicates the reverse GRU network that reads words in the backward direction. We use Θ to denote all the GRU parameters. The action yt+1 to be taken at the next time stamp is obtained via yt+1 =softmax(Wyht + by).

4.2 Transition-based Adversarial Network

To adapt the above model to another language, we apply domain adversarial network (DAN) to learn domain-invariant features [Ganin et al., 2016]. The intuition is to make each configuration representation invariant across languages, but discriminative for action predictions. Indeed, at each state of transition, the configuration should share similar characteristics across languages to inform the next action. To apply DAN, we create two classifiers: a transition action predictor Y and a language discriminator D to discriminate between configurations from source and target languages.

The corresponding structure is shown in Figure 3 on the right. Denote by θf , θy, and θd the parameters for feature learning, action prediction, and language prediction, respectively. The discriminator generates a probability distribution over languages P(S|ht), where ht obtained from (1) is the hidden representation for the whole configuration at time stamp t. Here P(S = 1|ht) indicates the probability of ht belonging to the source language. The predictions for action label and language label are computed on top of a fully connected layer

where hyt =tanh(Wyhht+byh ) and hdt =tanh(Wdhht+bdh ). The model is trained by minimizing

for feature generator, and by minimizing Ld(θf , θd) for the language discriminator. Here Ly and Ld are the cross-entropy losses for action prediction and language classifier, respectively. The hyper parameter α is used to tune the trade-off between the two losses. The saddle points ˆ θf , ˆ θy, and ˆ θd can be found as a stationary point of the following updates:

4.3 Incorporate Syntactic Relations

The previous model ignores the syntactic relations among the words in each sentence, which is, however, crucial for aspect extraction and bridging the gap between source and target languages. To incorporate syntactic information in monolingual setting, Wang et al. [2016] proposed a dependencytree-based recursive neural network that computes the hidden representation of each word as a composition function of its children. Following this idea, a straightforward method is to obtain the hidden representation of each word through recursive neural network and use it as the input to GRU for the output stack, candidate stack and buffer. However, the relation information may have little effect to the action prediction when the dependency path is too long and the model consists of multiple layers. Moreover, the hidden representation for each stack/buffer may contain information from the others due to the dependency tree. This contradicts the idea that each stack only contains partial information given by its content. To overcome these potential limitations, we propose to construct partial trees in the output stack.

Figure 4 presents an example configuration at timestamp t′ . For each sentence, we first produce a dependency tree from the parser. At each configuration, we pick all the edges in the tree that connect the words in the output stack and keep the representation for the highest node in each connected component. The rest of the words in the output stack that are not connected are then linked through recurrent edges. As shown in Figure 4, the output stack consists of "We love the pink pony and". According to the dependency tree, love is the highest node and is connected to all the other words in the output stack, hence, we only keep the hidden representation of love that is recursively computed from all the words below. Formally, given an input sentence with pre-generated dependency tree, at time stamp t, let xOt = {xO1 , ..., xOnOt }, xCt = {xC1 , ..., xCnCt }, xBt = {xB1 , ..., xBnBt } represents the list of word embeddings for the output stack, candidate stack, and buffer, respectively. For the output stack, we compute the hidden representation hOi of each word as:

where KO i denotes the set of children of node i in the output stack, rik denotes the dependency relation between node i and its child node k, and hOk is the hidden vector of node k in the output stack. When KO i = ∅, we ignore the last addition in (8). To compute the final hidden representation hOt for output stack, we select all the hOj 's when the jth node does not have any parent in the output stack or it is the root node in the tree and denote the set of these node indexes as I O t :

The hidden representations for the candidate stack and the buffer are produced by GRU and RGRU, respectively, as

where hCi =tanh(Wv ·xCi +b) and hBi =tanh(Wv ·xBi +b). We only incorporated the syntactic information in hOt because only the output stack contains rich processed information. And we use another embedding to represent the empty stack, which is randomly initialized and trained.

To explicitly make use of the syntactic relations, we integrate an auxiliary task to predict the dependency relation of the word about to be processed in each configuration. Inspired by [Odena et al., 2017] which applies an auxiliary classifier to help the adversarial training, our auxiliary task to predict the relation between the word on top of the buffer and its parent is helpful to decide on next action. Moreover, this implicitly conditions the language representations hdt on the relation labels, which helps to better align language features based on syntactic structure. To avoid syntactic inconsistencies across languages, we apply universal dependency parsers and utilize its taxonomy 1 to group relations into more general categories as the auxiliary labels 2 . The auxiliary prediction is

By plugging the loss of the auxiliary prediction task into (4), our final objective becomes

1 http://universaldependencies.org/

2 There are in total 15 categories. The groupings are: {nsubj, obj, iobj, nsubjpass, dobj}, {csubj, ccomp, xcomp}, {cc, conj}, {fixed, flat, compound, mwe}, {expl, obl, vocative, dislocated}, {advcl}, {advmod, discourse, neg}, {aux, cop, mark, auxpass}, {nmod, appos, nummod}, {acl}, {amod}, {det, clf, case}, {list, parataxis}, {root}, {punct, dep}

where La is the cross-entropy loss for the auxiliary task with θa denoting its parameters. θy and θd are updated using (6)- (7). The updates for θf and θa are:

During training, each sentence is fed into the common feature extractor to produce configuration vectors. For source language, each configuration is fed into the action, auxiliary and language predictors, whereas for target language, only auxiliary and language classifiers are trained.

5 Experiments

For experiments, we use the restaurant reviews from English, French and Spanish taken from SemEval Challenge 2016 task 5. The statistics of the datasets are listed in Table 1. We use labeled training data from the source language and unlabled training data from the target language to train the model. For testing, we conduct both transductive and inductive experiments to test our model on the unlabeled training data and the test data of the target language. The results are shown in F1 scores. For aspect extraction, only exact match is counted as correct when dealing with multi-word aspect terms.

Regarding experimental setting, the word embeddings are pre-trained using multivec [B´erard et al., 2016] that generates bilingual word embeddings with parallel corpus. The parallel corpus are from Europarl 3 that contains 2M sentences for each language. The dependency trees are generated from Stanford universal dependencies 4 . The whole network is trained with SGD using learning rate 0.02. The trade-off parameters are α = 0.1 and β = 1. The size of word embeddings is 100 and the hidden layer size is 50. Each experiment is trained for 20 epochs and the best performance is reported. For the overall complexity, our model takes 10mins for training 1 epoch with 4000 sentences using Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz. Testing is within seconds.

5.1 Experimental Results

To the best of our knowledge, there is not yet any work on cross-lingual aspect extraction. To make fair comparisons, we revise some popular models listed in the following to be fitted into the cross-lingual setting as our baselines:

• Translate-TAN, Translate-CRF: Translate the target data to source language using google translator, and train a TAN or linear-chain CRF with labeled source data to make predictions on the translated target data.

3 http://www.statmt.org/europarl/

4 https://nlp.stanford.edu/software/stanford-dependencies.shtml

• NoAdp: Train the model only on source data using TAN without adaptation and then test on target data.
• A-RNN: With a RNN [Liu et al., 2015] producing a hidden representation for each word, an adversarial network with a language discriminator is applied on top.
• A-R2NN: Apply the model in [Wang et al., 2016] but replace CRF with RNN on top of the dependency-treebased recursive neural network. An adversarial network with a language discriminator is then applied.
• CrossCRF: Linear-chain CRF [Jakob and Gurevych, 2010] with non-lexical features that are similar across domains. In cross-lingual setting, we use universal POS tags, sentiment lexicons, universal dependencies etc.
• CL-DSCL: A deep model with structural correspondence learning proposed by [Ding et al., 2017]. We convert the dependencies in the rules to universal dependencies and use sentiment lexicons for all three languages.

Specifically, for A-RNN, A-R2NN and CL-DSCL, we combine training sentences from labeled source language and unlabeled target language to train the models. Given a training sentence, A-RNN and A-R2NN compute final features for each word with RNN, which is then fed into a language discriminator to predict the language label. For source language, a token classifier is jointly applied to predict "BIO" labels. CL-DSCL integrates auxiliary classifiers into RNN. If a training sentence is from source language, both the auxiliary and "BIO" classifier are applied on top of the hidden vector of each word. Only auxiliary classifier is applied for target training sentences. The auxiliary task is to decide whether the word satisfies some rules describing universal dependency relations among aspect and opinion words. CrossCRF only takes training sentences from source language to train. All the trained models are tested on test sentences for target language to obtain "BIO" label for each token.

The comparison results are shown in Table 2. Obviously, the proposed model TAN achieves the state-of-the-art results most of the time with large performance gain. The performances of both A-RNN and A-R2NN are much worse compared to TAN, even though the syntactic structure is incorporated to bridge the gap between source and target languages (A-R2NN). This proves our assumption that wordlevel knowledge is hard to be transferred across languages. For CL-DSCL, the result shows 18.54%, 5.51% drop compared to TAN for inductive experiments transferring from English to French and Spanish, respectively. The inferior performances for the cross-domain models indicate the difficulty to build the correspondences with heterogeneous spaces across languages, compared to domain adaptation in the same language. This shows the advantage of our proposed model that learns transferable actions across domains.

We also conduct experiments to show the effect of each component of TAN. The results are listed in Table 3. The first column indicates the modification of TAN: -hier replaces the hierarchical grouping of dependency relations with exact relation as the label; -aux removes the auxiliary task; -dan removes domain-adversarial network; -ptree removes partial tree for the output stack and +ftree use the complete dependency tree to compute the hidden representation of each word and feed it into a GRU for the representations of output, candidate and buffer. Among all these variants, -hier is most comparable, although worse than TAN. This shows the benefit of relation grouping in the transfer setting. Indeed, in the hierarchy of universal dependencies, higher level is more general to be shared across languages. By removing the auxiliary task (-aux), the performance drops considerably for the last four experiments. It is even worse when removing DAN (-dan). This shows either the auxiliary task or DAN alone is not good enough to capture shared information. By using auxiliary-conditional DAN, we can bridge the language gap through shared syntactic space. We also observe the effect of using partial dependency tree to compute the output representation, compared with a simple GRU (-ptree) or a complete dependency tree (+ftree). This indicates that RNN fails to capture syntactic structure and may produce unnecessary noise, and a complete dependency tree brings excessive knowledge about the words not contained in the stack, making it hard to propagate information when the path is long.

To show that our model indeed benefits from unlabeled target data during training and learns language-invariant features that are useful for predictions, we conduct experiments to vary the number of unlabeled target data for training. The percentage of the unlabeled target data increases from 0.0 to

1.0. We select two transfer settings: Es→Fr and En→Es and compare TAN with CL-DSCL and A-R2NN. The inductive performances are shown in Figure 5. TAN shows more stable improvements with the increasing number of target training data compared with the other 2 baselines. This proves that TAN is able to learn from unlabeled target for adaptation.

We also conduct sensitivity tests on the hyper-parameters of TAN. As shown in Figure 6, we separately vary the tradeoff parameter β for Fr→En and the hidden layer dimension of the configuration representation for En→Es. We plot both the transductive and inductive results on training and test data, respectively. The results are overall stable for β and change slowly when changing the dimension of hidden layer from 40 to 100. This proves the robustness and stableness of our proposed model against these variations.

6 Conclusion

We propose a novel transition-based deep model for crosslingual aspect extraction that integrates domain adversarial network with syntactic information. Our model focuses on learning transferable configuration for the sentence at each timestamp and aims to learn a shared space that is actionsensitive but language-invariant given the syntactic structure.

Acknowledgements

We thank the support from NTU Singapore Nanyang Assistant Professorship (NAP) grant M4081532.020 and Fuji Xerox Corporation through joint research on Multilingual Semantic Analysis.