Learn from Syntax: Improving Pair-wise Aspect and Opinion Terms Extraction with Rich Syntactic Knowledge

1 Introduction

Fine-grained aspect-based sentiment analysis (ABSA), which aims to analyze people's detailed insights towards a product or service, has become a hot research topic in natural language processing (NLP). The extraction of aspect terms (AT) extraction and opinion terms (OT) as two fundamental subtasks of ABSA have emerged [Wang et al., 2017; Xu et al., 2018; Fan et al., 2019; Chen and Qian, 2020]. In later research, the aspect and opinion terms co-extraction has received much attention for the exploration of mutual benefits in between [Wang et al., 2017; Dai and Song, 2019]. However, these extraction methods do not consider AT and OT as pairs. More recently, some efforts are devoted to detecting the pair of the correlated aspect and opinion terms jointly, namely pair-wise aspect and opinion terms extraction (PAOTE) task [Zhao et al., 2020; Wu et al., 2020a; Chen et al., 2020], as illustrated in Figure 1. Existing works perform end-to-end PAOTE based on joint learning methods for better task performances [Zhao et al., 2020; Wu et al., 2020a; Chen et al., 2020]. Unfortunately, there interactions unused. Hence, how to effectively model the overlapping structure during the term pairing remains unexplored.

∗Equal contribution

†Corresponding Author

In this paper, we aim to address all the aforementioned challenges by presenting a novel joint framework for PAOTE. Figure 2 shows the overall framework. First, we propose a syntax fusion encoder (namely SynFue) for encoding syntactic features (cf. Figure 3), where a label-aware graph convolutional network (LAGCN) models dependency edges and labels as well as POS tags, and the local-attention module encodes POS tags. By capturing rich syntactic knowledge in this manner, SynFue is able to produce span terms more accurately, and on the other hand, it encourages sufficient interactions between syntactic structures and term pair structures. During pairing, we then perform high-order scoring for each candidate aspect-opinion term pair via a Triaffine scorer [Carreras, 2007], which can model the triadic relations of the overlapping term structures with a broader viewpoint. To enhance the semantic pairing, we further consider a syntactic-aware scoring, re-harnessing the syntax-enriched representations in LAGCN. Finally, our system outputs all valid aspect-opinion term pairs based on the overall potential scores.

• ⋆We for the first time in literature propose to incorporate rich syntactic and linguistic knowledge for improving the PAOTE task. We propose a LAGCN to encode the dependency trees with labels as well as POS tags in a unified manner. Also, we promote the term boundary recognition by modeling the POS features via a local attention mechanism.

To sum up, our contributions are three-fold.

• ⋆We present a high-order joint solution for aspect-opinion pairing with a Triaffine scorer, fully exploring the underlying mutual interactions within the overlapping pair structures. The intermediate syntax-enriched representations yielded from LAGCN are re-exploited for further syntactic-aware scoring.
• ⋆ Our method attains state-of-the-art performances on four benchmark datasets for PAOTE. Further analysis reveals that our method can effectively leverage rich syntactic information, and capture the correlations between syntactic structures and aspect-opinion pair structures.

2 Model

As illustrated in Figure 2, our system is built based on the current best-performing span graph-based model [Zhao et al., 2020; Eberts and Ulges, 2020]. The model first takes as inputs the contextualized word representation from the BERT language model [Devlin et al., 2019]. Next, syntactic dependencies and POS tags are injected into the syntax fusion encoder. We then perform term type classification and filtering based on the term representations from the token representations. In the pairing stage, we measure the term-term pairs with the potential scores including high-order scores and syntactic-aware scores, based on which the final pairs will be output. Given an input sentence s={w1, · · · , wT }, our system is expected to produce a set of aspect-opinion pairs P={p1(a, o), · · · , pk(a, o)} ⊂ A × O. A={a1, · · · , aN} is all possible aspect terms, where an can be a single word or a phrase, denoted as an={wi, · · · ,wj}. Likewise, O={o1, · · · , oM} is all the possible opinion terms, where om={wi, · · · ,wj} is a term span.

Predicted Pairs: P = { p1(a, o), , pk(a, o) }

2.1 Word Representation fromBERT

The BERT language model has been proven superior in building contextualized representations for various NLP tasks [Ding et al., 2020; Eberts and Ulges, 2020; Zhao et al., 2020]. Hence we utilize BERT as the underlying encoder to yield the basic contextualized word representations:

where vt is the output representation for the word wt.

2.2 Syntax Fusion Encoder

We inject three types of external syntactic sources, i.e., dependency edges and labels, and POS tags, into our syntax fusion encoder (SynFue). SynFue (with total L layers) consists of a local attention for POS tags and a label-aware GCN for all syntactic inputs at each layer (cf. Figure 3).

Local-attention for encoding POS tags POS tags, as the major word-level linguistic features, provide potential clues for boundary recognition of term spans [Nie et al., 2020]. Instead of adopting the vanilla hard attention that encodes the whole sequence-level information, we encode POS tags via a local attention mechanism [Luong et al., 2015], which is more capable of capturing the local contexts for phrasal term spans. Technically, for each word wt in s, we mark its corresponding POS tag as wp t , and obtain its POS embedding xpt . At the l-th layer, the local attention operation is performed at a scope of d window size:

where [; ] denotes the concatenation operation, W1 is the learnable parameters and ep,l t is the output representations.

Label-aware GCN for rich syntactic features Previous studies employ GCN [Marcheggiani and Titov, 2017] to encode purely the dependency structural edges, while they fail to model the syntactic dependency labels leeched on to the edges, but also ignore the POS category information. We note that these syntactic features should be navigated simultaneously in a unified manner, as they together essentially describe the complete syntactic attributes in different perspectives. We here propose a label-aware GCN (LAGCN) to accomplish it. Given the input sentence s with its corresponding dependency edges and labels, and POS tag embeddings xpt , we define an adjacency matrix {bt,j}T×T for dependency edges between each pair of words ( wt and wj) where bt,j=1 if there is an edge between them, and bt,j=0 vice versa. There is also a dependency label matrix {rt,j}T×T , where rt,j denotes the dependency relation label between wt and wj . We maintain the vectorial embedding xrt,j for each dependency label.

We denote the hidden representation of wt at the l-th LAGCN layer as es,l t :

where αlt,j is the syntactic-aware neighbor connecting-strength distribution calculated by:

where rs t,j entails the syntactic relationship between tokens. The weight distribution αt,j entails the structural information, thus comprehensively reflecting the syntactic attributes.

We explicitly concatenate the representations of L-th layer of local attention POS encoder and LAGCN as the overall token representations eLt = [es,L t ; ep,L t ].

2.3 TermGeneration and Filtering

We concatenate BERT representation vt and SynFue representation eLt as final token representation ht=[vt; eLt ]. We next construct span representation based on token representations:

where hhead and htail are the boundary representation of the start and end token of each term. hS is the overall sentence representation from BERT (i.e., from CLS token). hpool is the max pooling operation (Max-Pooling), and hsize is the term width embedding. 'FFN' refers to feed-forward layers, and 'Dropout' is applied to alleviate overfitting.

Then, we determine the term type via a softmax classifier ci = Softmax(si). We pre-define three categories of terms: {C A,C O ,C ϵ }, more specially, aspect term (C A), opinion term (C O ) and invalid term (C ϵ ). Afterward, we estimate which type each term belongs to, by looking at the highestscored class based on ci, i.e., filtering invalid candidates (C ϵ ), maintaining a set of final terms which supposedly are aspect terms and opinion terms, i.e., an and om. Based on the filtering step, we can effectively prune the pool of span terms and lead to higher efficiency. We denote the representation of aspect term and opinion term as san and som, respectively.

2.4 TermPairing

We measure the relationship between each candidate aspectopinion term pair by calculating the potential scores of the term pairs Qn,m ∈ RN×M . We consider two types of scores: high-order pairing scores and syntactic-aware scores.

High-order scoring. In previous works, the Biaffine scorer is usually utilized for relation determination [Dozat and Manning, 2017; Eberts and Ulges, 2020]. However, Biaffine only models the dyadic relations between each pair, which may lead to an insufficient exploration of the triadic relations that occur in the overlapping aspect-opinion pair structures, as we highlighted earlier. We hence adopt high-order scoring, as in Figure 2. First, a Biaffine scorer is used for measuring each pair under the first-order scope:

whereW6 ∈ R(ds+1)×ds is a parameter. ds is the dimension of the term representations. The Triaffine scorer [Carreras, 2007; Wang et al., 2019] is utilized for second-order scoring of two overlapping pairs simultaneously, over three term representations (i.e., san , som, s∗k ) 2

2 s∗k can be an aspect or opinion term, excluding san and som.

Syntactic-aware scoring. Intuitively, the syntactic representations in LAGCN that depict the syntactic relationship between tokens can also provide rich clues for the detection of term pairs. Here we consider making use of such syntactic features, performing syntactic-aware scoring, cf. Figure 2. Technically, we re-harness the token-level representation Rs=[· · · , rs t,j , · · · ] ∈ RT×T (from Eq. 5) by projecting Rs into span-level syntactic transition representations Ss ∈ RN×M . For an aspect-opinion term pair an and om, we first track its start and end indexes respectively in Rs . We then obtain the transition representations spn,m, i.e., from an to om via the span pooling operation:

We further apply a cross-attention operation [Ding et al., 2020] for each pair spn,m, to propagate the dependencies and impacts from other terms at the same row and column. Concretely, we calculate the row-wise weights ↔β and column-wise weights β↕ on spn,m:

where k is the column or row index of the current pair.

Finally, we build the overall unary potential scores by taking into account all the above scoring items.

where η1 and η2 are factors regulating the contributions of different scores. We then push Qn,m into [0,1] likelihood value:

where those elements yn,m larger than a pre-defined threshold δ will be output as valid pairs, i.e., pk(an, om).

2.5 Training

During training, given an input sentence s with manually annotated gold pairs ˆ P = {ˆ pk(ˆa, ˆ o)}Kk=1 . We define a joint loss for term detection and pair relation detection:

where D is the total sentence number, λ1 is the coupling co-efficiency regulating two loss items, and λ2 is the ℓ2 regularization factor. LType denotes the negative log-likelihood loss for term type detection, and LPair denotes the binary cross-entropy over pair relation classes:

where G is total spans, p′k is the factorized probability of each aspect-opinion pair over input sentence: p′k = ∏a∈A,o∈O p(a, o).

Negative sampling. During term type detection, in addition to the positive samples of the labeled terms, we randomly draw a fixed number (Nt) of negative samples, i.e., non-term spans, to accelerate the training.

3 Experiments

3.1 Experimental Setups

Datasets and resources. We conduct experiment on four benchmark datasets [Wu et al., 2020a], including 14lap, 14res, 15res and 16res. The statistics of four datasets are listed in Table 1. We employ the Stanford CoreNLP Toolkit 3 to obtain the dependency parses and POS tags for all sentences. We adopt the officially released pre-trained BERT parameters. 4

Implementation. Both the BERT representation vt and the term span representation ds have 768 dimensionality. The syntactic label embedding size and POS embedding are set to 100-d, and span width embedding is set to 25-d. We adopt the Adam optimizer with an initial learning rate of 4e-5. We use a batch size of 16 and set unfixed epochs with early-stop training strategy instead. We mainly adopt F1 score as the metric. Our model 5 takes different parameters on different data, which are separately fine-tuned.

Baselines. Our baselines are divided into pipeline methods and joint methods. · 1) One type of pipeline methods uses CMLA [Peng et al., 2020] to co-extract aspect and opinion terms, and then make pairing with CGCN [Zhang et al., 2018]. Another pipeline schemes first perform targeted aspect terms extraction, e.g., with BiLSTM+ATT [Fan et al., 2018], DECNN [Xu et al., 2018] and RINANTE [Dai and Song, 2019] models, and then conduct target-oriented opinion terms extraction with the given aspect terms in the second stage, e.g., by IOG [Fan et al., 2019]. · 2) Joint methods perform unified extraction of aspect terms and opinion terms, as well as pair-wise relation between them, including SpanMlt [Zhao et al., 2020] and GTS [Wu et al., 2020a].

3.2 Results and Analysis

Main performances. The overall results are shown in Table 2. The first observation is that the performances by the joint methods are constantly higher than the two types of pipeline methods. This confirms the previously established viewpoint that the joint scheme of aspect-opinion term extraction can relieve the error propagation issues in the pipeline. More importantly, our proposed model achieves the best results against all the baselines. For example, our model even without using BERT obtains 64.59%, 74.05%, 63.74% and 72.06% F1 scores on each dataset, respectively. By integrating the BERT language model, the performances of the joint models can be further improved. Note that our proposed model outperforms strong baselines by a large margin.

3 https://stanfordnlp.github.io/CoreNLP/, CoreNLP v4.2.0

4 https://github.com/google-research/bert, base cased version.

5 Available at https://github.com/ChocoWu/Synfue-PAOTE

Ablation. We perform ablation experiments (cf. Table 3) to understand the effect of each part of the proposed model. We first remove BERT while using pre-trained Glove embeddings for BiLSTM instead, and we receive the most notable performance drops among all other factors, showing the effectiveness of BERT for downstream tasks [Zhao et al., 2020; Wu et al., 2020a; Fei et al., 2020]. Without the dependency label features, we can find that the performances consistently decrease. By replacing LAGCN with vanilla GCN encoding only the dependency arcs, the results drop further. When LAGCN is ablated, i.e., without encoding the syntactic arcs and labels as well as the POS tags, such drops are magnified significantly. If we strip off the local-attention POS encoder, or remove the POS features away from LAGCN, the performances are downgraded to some extent.

For decoding, we can find that the negative sampling strategy influences the results. Also, three pairing scores show effects in different extent, i.e., the first-order Biaffine gives more effects than the second-order Triaffine scorer and syntactic-aware scoring. One possible reason is that most nonoverlapping pairs prevent Triaffine, which is more capable of modeling triadic relations among overlapping structures, from giving its utmost function. Furthermore, we can find that the syntactic-aware scores are highly crucial to the pairing, and removing the cross-attention mechanism will reduce the effectiveness of syntactic-aware scores.

Term extraction. We further examine our model's capability on aspect term extraction (ATE) and opinion term extraction (OTE), separately. Figure 4 shows the performances of two subtasks on the 14res and 15res datasets. It can be observed that the joint methods consistently outperform the pipeline method (RINANTE), while our model gives the best performances compared with all baselines. We also find that whether we use POS tagging features or not has a significant impact on term extraction.

Effects of pairing strategies. In addition to the base Biaffine scorer for term pairing, we further study how the Triaffine scorer and the syntactic-aware scoring influence the overall pairing performances. We reach this by tuning the regulating factors η1 and η2. From the patterns in Figure 5, we can find that the overall result is the best when η1=0.4 and η2=1.0. For one thing, the comparatively fewer overlapping pairs in the dataset make the contribution by the Triaffine scorer limited. For another, the syntactic dependency information from LAGCN offers prominent hints for determining the semantic relations between aspect-opinion terms, which accordingly requires a higher proportion of scoring weights.

Influence of Layer Number. Syntactic fusion encoder (SynFue) is responsible for fusing syntactic structure features as well as the POS tags. Intuitively, more layers of SynFue should give stronger capability of the syntax modeling. We show the performances by installing different layers of SynFue in our model, based on each dataset. As illustrated in Figure 6, we see that the model can give the best effect with a twolayer of SynFue, in most of the datasets. This implies that too many layers of syntax propagation may partially result in information redundancy and overfitting.

Syntax correlations. Finally, we qualitatively investigate if our proposed LAGCN can genuinely model these syntaxes to improve the task. Technically, for each input sentence we observe the syntax-connecting weights α (in Eq. 6) and collect the weights of the correlated dependencies and POS tags of token words. We render these normalized values in Figure 7. It is quite clear to see that LAGCN well captures the correlations between syntactic dependencies and POS tags. For example, for the dependency arc with the 'nsubj' type, LAGCN learns to assign more connections with those tokens with the POS tags of 'JJ', 'NN' and 'NNS', which essentially depicts the boundary attributes of the constituent spans, as well as the correlated semantic relations between terms. This can explain the task improvement accordingly.

4 RelatedWork

Aspect terms extraction and opinion terms extraction, as two fundamental subtasks of fine-grained aspect-based sentiment analysis (ABSA) [Pang and Lee, 2007; Liu, 2012; Huang et al., 2020; Wang et al., 2020], have received extensive research attentions in recent years [Wang et al., 2017; Xu et al., 2018; Fan et al., 2019; Chen and Qian, 2020]. Considering the relevance between two subtasks, Zhao et al. (2020) propose the pair-wise aspect and opinion terms extraction (PAOTE) task, detecting the pair of the correlated aspect and opinion terms jointly. Preliminary works adopt the pipeline methods, i.e., first extracting the aspect terms and the opinion terms separately, and then making pairings for them [Wang et al., 2017; Xu et al., 2018; Peng et al., 2020; Wu et al., 2020b]. Recent efforts focus on designing joint extraction models for PAOTE [Wu et al., 2020a; Chen et al., 2020], reducing error propagation and bringing better task performances.

Previous studies also reveal that syntactic dependency features are crucial for ABSA [Phan and Ogunbona, 2020; Tang et al., 2020]. These works mostly consider the syntactic dependency edges, while the syntactic labels and POS tags that also provide potential evidences, can not be exploited fully in the PAOTE task. We thus in this work propose a novel label-aware syntactic graph convolutional network for modeling rich syntactic features. Furthermore, we leverage the syntactic information for better term pairing. We also take advantage of the high-order graph-based models [Carreras, 2007; Wang et al., 2019], i.e., using the second-order Triaffine scorer to fully explore the underlying mutual interactions within the overlapping pair structures.

5 Conclusions

In this study, we investigated a novel joint model for pair-wise aspect and opinion terms extraction (PAOTE). Our proposed syntax fusion encoder incorporated rich syntactic features, including dependency edges and labels, as well as the POS tags. During pairing, we considered both the high-order scoring and the syntactic-aware scoring for aspect-opinion term pairs. Experimental results on four benchmark datasets showed that our proposed syntax-enriched model gave improved performance compared with current state-of-the-art models, demonstrating the effectiveness of rich syntactic knowledge for this task.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61772378), the National Key Research and Development Program of China (No. 2017YFC1200500), the Research Foundation of Ministry of Education of China (No. 18JZD015), and the Key Project of State Language Commission of China (No.ZDI135-112).