Automated Rule Selection for Aspect Extraction in Opinion Mining

1 Introduction

Aspect extraction is a fundamental task of opinion mining or sentiment analysis. It aims to extract fine-grained opinion targets from opinion texts. For example, in the sentence "This phone has a good screen," we want to extract "screen." In product reviews, an aspect is basically an attribute or feature of a product. Aspect extraction is important for opinion mining because without knowing the aspects that the opinions are about, the opinions are of limited use [Liu, 2012].

In recent years, aspect extraction has been studied extensively. There are two main approaches: syntactical and statistical. Some existing work has shown that syntactical dependency based methods such as double propagation (DP) [Qiu et al., 2011] can perform better than the statistical learning based method Conditional Random Fields (CRF) [Lafferty et al., 2001]. The key idea of the syntactical approach is that opinions have targets and there are often explicit syntactic relations between opinion words (e.g., "good") and target aspects (e.g., "screen"). By exploiting such relations, the DP method can use a set of seed opinion words to extract aspects and new opinion words, and then use them to extract more aspects and opinion words, and so on, through a propagation process.

Figure 1 shows the dependency relations between words in the sentence "The phone has a good screen." The word "good" is an opinion word, which is known (given or extracted). The word "screen," a noun modified by "good" (i.e., they have a dependency relation amod), is clearly an aspect (or opinion target). Therefore, from a given set of opinion words, we can extract a set of aspects using some syntactic relations, e.g., amod. Similarly, one can also use syntactic relations to extract new aspects and new opinion words from the extracted aspects. For example, using the extracted aspect "screen," we can extract "speaker" from "Both the screen and the speaker are what I expected" using the conjunction "and." Note that, an aspect term can be a single word or a multi-word phrase (e.g., "battery life"). We will explain how phrases are handled in the experiment section.

Clearly, syntactic patterns can extract wrong terms. For example, in "It is a good idea to get this phone," the words "good" and "idea" also have the dependency relation amod, but "idea" is not an aspect of the phone. In order to produce good results, one has to carefully choose rules. Apart from choosing the right rules, some heuristics were also proposed in [Popescu and Etzioni, 2005; Qiu et al., 2011] to prune the extracted aspects to reduce errors. However, simple heuristics such as frequency-based pruning method in [Hu and Liu, 2004] have the problem of pruning many low frequency correct aspects, and complicate heuristics such as the methods in [Qiu et al., 2011] need external knowledge in pruning.

As we will see later, a large number of rules with different qualities can be used for extraction. In the existing works, researchers typically manually examine a set of rules, e.g., [Qiu et al., 2011], and then choose a small subset of more reliable rules. However, since they work by trial and error, no reasons are provided on why some rules are chosen while others are not, how to decide whether a rule is good, and how to combine a set of rules. It is important to have the reasons in order to select rules automatically as the manual approach is very difficult to guarantee quality. This paper addresses this difficulty and proposes an automated rule selection algorithm to choose a subset of good rules from a given set of rules.

Formally, our problem is stated as follows. Given a set of aspect extraction rules R (each has an id), a set of seed opinion words O, and a set of reviews D with labeled aspects, we want to select a subset of rules in R that can be used to extract aspects from reviews across domains. This cross domain capability is important because we do not want to label data in each domain, which is highly labor-intensive. The selected set of rules are obviously domain independent because they are syntactic rules, and thus can be used across domains.

The proposed algorithm can take any number of rules of any quality, and automatically select a good subset of rules for extraction. Selecting the optimal set of rules would involve evaluating all possible subsets of R on D and selecting the subset with the right rule sequence that performs the best. However, as the rule number m grows, the number of subsets grows exponentially, i.e., 2m. Thus, we propose a greedy algorithm which is inspired by rule learning in machine learning and data mining [Liu et al., 1998].

Our experiment is conducted using a popular aspect extract evaluation dataset that contains annotated reviews of five products [Hu and Liu, 2004] as well as some additional datasets annotated by us. Our experimental results show that the proposed method is able to select a subset of rules from those rules used in DP to perform extraction much more accurately than the original rule set used in DP. Furthermore, since our method can take rules of any quality as input, we add many more new rules to the rule set in DP. The proposed method selects a subset of rules that produces even better results than that selected from the DP rules only. It also outperforms the state-of-the-art CRF-based supervised learning method by a large margin.

2 RelatedWork

There are two main approaches for aspect extraction: syntactical and statistical. The former is mainly based on dependency relations [Hu and Liu, 2004; Zhuang et al., 2006; Wang and Wang, 2008; Wu et al., 2009; Zhang et al., 2010; Zhao et al., 2010b; 2014; Liu et al., 2013b], while the latter on CRF [Jakob and Gurevych, 2010; Choi and Cardie, 2010; Mitchell et al., 2013] and topic modeling [Mei et al., 2007; Titov and McDonald, 2008; Li et al., 2010; Brody and Elhadad, 2010; Moghaddam and Ester, 2011; Sauper et al., 2011; Mukherjee and Liu, 2012; Chen et al., 2014].

On the statistical approach, we will show that our method outperforms the CRF-based method [Jakob and Gurevych, 2010] by a large margin. Topic modeling often only gives some rough topics in a document corpus rather than precise aspects as a topical term does not necessarily mean an aspect [Lin and He, 2009; Lu et al., 2009; Zhao et al., 2010a; Jo and Oh, 2011; Fang and Huang, 2012]. For example, in a battery topic, a topic model may find topical terms such as "battery," "life," "day," and "time," etc, which are related to battery life, but each individual word is not an aspect.

There are also frequency-based methods for aspect extraction, which extract frequent noun phrases as aspects [Hu and Liu, 2004; Popescu and Etzioni, 2005; Ku et al., 2006; Zhu et al., 2009]. Additionally, researchers have also proposed different joint models leveraging the relations between opinion words and aspects as well as other information to extract aspects, opinion words, and their relations [Klinger and Cimiano, 2013; Lazaridou et al., 2013]. Other similar works include label propagation [Zhou et al., 2013] and word alignment based approaches [Liu et al., 2013a].

In this paper, we focus on the syntactical approach. Our work is most related to the DP approach [Qiu et al., 2011], which we will discuss further below. Since our rule selection method uses some training data, it can be regarded as an integration of both supervised and unsupervised learning because its resulting rule set can be applied to the test data from any domain. Our work is also related to associative classification [Liu et al., 1998], which uses association rule mining algorithms to generate the complete set of association rules, and then selects a small set of high quality rules for classification, and the pattern mining method in [Kobayashi et al., 2007]. However, we do not need to generate rules as we already have such rules based on syntactical dependency relations. Our rule selection method is also very different.

3 Syntactical Extraction Rules

This work uses dependency relations between opinion words and aspects, as well as between opinion words and aspects themselves to extract aspects. For example, one extraction rule could be "if a word A, whose part-of-speech (POS) is a singular noun (nn), has the dependency relation amod with (i.e., modified by) an opinion word O, then A is an aspect", which can be formulated by the following rule r1:

IF depends(amod,A,O)∧pos(A, nn)∧opinionword(O) THEN aspect(A)

where depends(amod, A, O) means that A and O have a dependency relation amod, pos(A, nn) means that A is a singular noun, opinionword(O) means that O is an opinion word, aspect(A) means that A is an aspect. For example, from "This phone has a great screen," we can extract the aspect "screen" as there is a amod relation between "great" and "screen," "great" is an opinion word, and "screen" is a noun.

As discussed in [Qiu et al., 2011], there are many possible dependency relations that can be exploited for aspect extraction. To reduce incorrect aspects caused by propagation, [Qiu et al., 2011] used only a small subset of manually selected rules based on 8 dependency relations. Since our proposed approach can automatically select a subset of good rules for aspect extraction, it can take as input any kind and any number of syntactical rules regardless of their qualities.

We group rules into three types based on whether they can extract aspects by themselves given a set of opinion words. For some rules, the propagation mechanism in DP is needed before they can be used in the extraction.

Type 1 rules (R1 ): using opinion words to extract aspects (based on some dependency relations between them), e.g., rule r1. A set of seed opinion words are given a priori.

Type 2 rules (R2 ): using aspects to extract aspects. The known aspects are extracted in the previous propagation, e.g., IF depends(conj,Ai,Aj) ∧ pos(Ai, nn) ∧ aspect(Aj) THEN aspect(Ai)

For example, if "screen" has been extracted by a previous rule as an aspect, this rule can extract "speaker" as an aspect from "This phone has a great screen and speaker" because "screen" and "speaker" has the conj dependency relation.

Type 3 rules (R3 ): using aspects and opinion words to extract new opinion words. The given aspects are extracted in the previous propagation, and the given opinion words are the known seeds or extracted in the previous propagation. The following is an example of such rules:

IF depends(amod,A,O) ∧ pos(O, jj) ∧ aspect(A) THEN opinionword(O)

where pos(O, jj) means that O is an adjective. For example, if "screen" has been extracted as an aspect and "nice" was not a seed opinion word, then "nice" will be extracted as an opinion word by this rule from "This phone has a nice screen."

4 Rule Set Selection Algorithm

We are now ready to present the proposed algorithm. The first subsection gives the main idea and steps of the algorithm. The subsequent subsections detail each step.

4.1 The Main Idea and Steps

As mentioned in the introduction, finding the best subset of rules is an infeasible problem, we thus propose a greedy algorithm to perform the task, which has three steps:

Step 1: Rule evaluation. Clearly, rule quality is a key criterion for rule selection. This step first evaluates each rule to assess its quality. Specifically, given a set of rules, a set of seed opinion words, and a training data D with labeled aspects in its sentences, for each type of rules, this step applies each rule to D and outputs the precision and recall values of the rule. We use precision and recall because we want the rules with high precision and recall.

Step 2: Rule ranking. This step ranks the rules in each type first based on their precisions. If two rules have the same precision, the one with the higher recall is ranked higher. We use precision first because high precision rules are more desirable. The recall can be improved by using more rules. This step thus produces three rankings for the three types of rules to be used in the next step for rule selection.

Step 3: Rule selection. Given the ranked rules of each type and training data D, this step adds rules from the ranked rule set one by one in the descending order into the current output rule subset. Once a rule is added, the current rule subset is applied to and evaluated on D, and the F1-score of the current rule subset is recorded. This process continues until all rules in the ranked list are added to the output rule set and evaluated. The algorithm then prunes the rules to produce the final set of rules that gives the best result on D. In this step, F1-score is used as the performance evaluation measure because we want the final rule set to produce overall good aspect extraction result. F1-score is also the final evaluation measure in our experiments.

4.2 Rule Evaluation

Given the training data (e.g., product reviews) D with aspect labels, the evaluation of each rule in R1 is straight forward. We simply apply each rule r1 i ∈ R1 with seed opinion words O to D to extract all possible aspects, denoted by A1i (i ∈ [1,N], where N is the number of rules of type 1). The set of correct aspects in A1i is denoted by T 1 i , the precision of r1 i is defined by r1 i .pre = |T 1 i |/|A1i |, and the recall of r1 i is defined by r1 i .rec = |T 1 i |/|Alllab|, where |.| means the size of a set, Alllab is the set of all labeled aspects in D. The set of all aspects extracted by rules of type 1 is denoted by A1 = ⋃Ni=1 A1i . Note that, the rules in R1 are independent of each other. Also, given some opinion words a priori, the evaluation of rules in R1 is independent to other types of rules.

Rules of type 2 are dependent on rules of type 1 in the sense that they need some known aspects as input in order to extract new aspects. We apply each rule r2 j ∈ R2 to D on the condition that A1 is given, to get new aspects extracted by r2 j , denoted by A2j (j ∈ [1,M], where M is the number of rules of type 2, A2j ∩ A1 = ∅). The set of correct aspects in A2j is denoted by T 2 j , the precision of r2 j is defined by r2 j .pre = |T 2 j |/|A2j |, and the recall of r2 j is definded by r2 j .rec = |T 2 j |/|Alllab|. The set of all aspects extracted by rules of type 2 is denoted by A2 = ⋃Mj=1 A2j .

Rules of type 3 are dependent on rules of both type 1 and type 2 in the sense that rules of type 3 do not extract aspects directly and have to resort to rules of type 1 and type 2 to produce new aspects. Given O, we apply each rule r3 k ∈ R3 to D together with all rules in R1 and R2 , to get new aspects extracted by adding r3 k into R1 ∪ R2 , denoted by A3k (k ∈ [1, L], where L is the number of rules of type 3, A3k ∩ (A1 ∪ A2 ) = ∅). The set of correct aspects in A3k is denoted by T 3 k , the precision of r3 k is defined by r3 k .pre = |T 3 k |/|A3k |, the recall of r3 k is defined by r3 k .rec = |T 3 k |/|Alllab|.

4.3 Rule Ranking

After all the rules are evaluated, they are ranked according to precision and recall. The three types of rules are ranked separately, i.e., rules of each type are ranked among themselves. The order of rules in each type is defined as follows: Given two rules ri and rj of the same type, ri, rj ∈ Rk (k = 1, 2, 3), we have ri ≻ rj, called ri is higher than rj, if

1. ri.pre > rj.pre, or

2. ri.pre = rj.pre and ri.rec > rj.rec, or

3. ri.pre = rj .pre, ri.rec = rj .rec and ri.id < rj .id.

The order is a total order and defines a ranking on the rules.

4.4 Rule Selection

To select the best subset is impossible as we discussed in the introduction section. We thus propose a greedy selection algorithm based on the ranking of the rules of each type. The algorithm is called RS-DP (short for Rule Selection-DP) and given in Algorithm 1. RS-DP has four sub-steps:

Sub-step 1 (lines 1-2): Initialize the sequence S =<> (line 1), and discard the rules in the rule set whose precision and recall are both zero (line 2). It guarantees that the remaining rules can correctly extract at least one aspect.

Sub-step 2 (lines 3-7): Select rules fromR1 following the descending order defined by the ranking. Since the rules in R2 and R3 depend on the rules in R1 , the rules in R1 should be selected and added into S first. For each rule ri ∈ R1 , since ri can correctly extract at least one aspect, it will be a potential rule in S (line 4). We apply S to D and O to find those cases covered by S, i.e., they satisfy the rules in S, and then compute the F1-score of S, called F1-score derived from ri, and denoted by score(ri) = 2×S.pre×S.rec S.pre+S.rec (lines 5-6).

After inserting all the rules in R1 into S, we perform pruning (line 7) by discarding those rules in S that do not improve the F1-score of the rule set. Since when a rule r is added to the end of S, score(r), i.e., the F1-score of the current S is recorded, we simply find the first rule in S from which the highest F1-score is obtained. All the rules after this rule can be discarded because they only produce more errors.

Sub-step 3 (lines 8-12): Select rules fromR2 following the descending order defined by the ranking. Given O, rules in R2 cannot extract any new aspects directly, because they rely on known aspects. However, S now contains the selected rules from R1 , which can extract aspects using the seed opinion words. The extracted aspects can be fed to the rules in R2 . Thus, like that in sub-step 2, for each rule rj ∈ R2 , we first add it to the end of S (lines 8-9), and apply S to D and O to find those cases covered by S, and then compute score(rj), i.e., F1-score of S after adding rj (lines 10-11).

After inserting all the rules in R2 into S, line 12 performs rule pruning by discarding the rules that do not improve the F1-score of S. This sub-step works in the same way as line 7.

Sub-step 4 (lines 13-19): Select rules fromR3 following the descending order defined by the ranking. Unlike sub-steps 2 and 3, not all the rules in R3 are potential rules in the final rule set, only those rules that improve the performance of S should be selected. If we use the same evaluation strategy in this sub-step as in sub-steps 2 and 3, it can result in many useless rules, which generate no additional extractions. Thus, we use a different evaluation strategy here. That is, for each rule rk ∈ R3 , we evaluate S ∪ {rk} on D and O (lines 13-14). If S ∪ {rk} performs better than S, then rk is inserted into S (lines 15-18). Since every added rule can improve the results in this sub-step, it has no pruning.

5 Experiments

We now evaluate the proposed technique to assess the performance of aspect extraction of the selected rules.

5.1 Datasets

We use two customer review collections in our experiments. One is from [Hu and Liu, 2004], which contains five review datasets of four domains: digital cameras (D1, D2), cell phone (D3), MP3 player (D4), and DVD player (D5). The other one is built by us in order to further verify the effectiveness

Algorithm 1 RS-DP

Input: Ranked rule sets R1 , R2 and R3 , training data D, seed opinion words O

Output: The best subset S of rules

• 1: S =<>; // S is represented as a sequence
• 2: Discard all the rules in R1 , R2 and R3 whose precision and recall are both zero;
• 3: for each rule ri ∈ R1 in descending order do
• 4: insert ri at the end of S;
• 5: compute F1-score x of S on D and O, score(ri) = x;
• 6: end for
• 7: Find the first rule p in S with the highest derived F1-score among all the rules of S, drop all the rules after p in S;
• 8: for each rule rj ∈ R2 in descending order do
• 9: insert rj at the end of S;
• 10: compute F1-score x of S on D and O, score(rj) = x;
• 11: end for
• 12: Find the first rule q in S with the highest derived F1-score and drop all the rules after q in S, let maxF = score(q);
• 13: for each rule rk ∈ R3 in descending order do
• 14: compute F1-score x of S ∪ {rk} on D and O;
• 15: if (x >maxF) then
• 16: insert rk at the end of S, maxF = x;
• 17: end if
• 18: end for
• 19: Output S as the final rule set.

of our approach in more domains. It contains three review datasets of three domains: computer (D6), wireless router (D7), and speaker (D8). Aspects in these review datasets are annotated manually. The first collection has been widely used in aspect extraction evaluation by researchers [Hu and Liu, 2004; Popescu and Etzioni, 2005; Qiu et al., 2011; Liu et al., 2013a]. The second one is annotated by two annotators. Since they did not identify the same number of aspects, instead of using Kappa statistics, we use Dice coefficient to measure the inter-annotator agreement for our annotations. The average Dice coefficient for three domains is 0.7, which indicates a reasonable high degree of inter-annotator agreement. The detailed information about these datasets is shown in Table 1. For seed opinion words, we used all (and only) the adjective opinion words in the opinion lexicon of [Hu and Liu, 2004] 1 .

5.2 EvaluationMetrics

Precision, recall, and F1-score are employed as our evaluation metrics. There are two ways to compute the results: (1) based on multiple occurrences of each aspect term, and (2) based on distinct occurrence of each aspect term.

1 http://www.cs.uic.edu/∼liub/FBS/sentiment-analysis.html

In a dataset, an important aspect often occurs many times, e.g., the aspect "picture" occurred 10 times in a set of camera reviews. For (1), if any occurrence of "picture" is extracted, then all occurrences of "picture" are considered extracted, i.e., 10. If none of its occurrences is extracted, it is considered as 10 losses. In (2), if any occurrence of "picture" is extracted, it is considered as one extraction. If none is extracted, it is considered as one loss. (2) clearly makes sense, but (1) also makes good sense because it is crucial to get those important aspects extracted. Extracting (or missing) a more frequent aspect term is rewarded (or penalized) more heavily than extracting (or missing) a less frequent one.

Let an extraction method return a set A of distinct aspect terms, and the set of distinct aspect terms labeled by human annotators be T . TP (true positives) is |A ∩ T |, FP (false positives) is |A\ T |, FN (false negatives) is |T \A|.

For (2), the evaluation metrics are defined as follows:

For (1), F1-score is computed in the same way, but precision and recall computations need to change because we now consider multiple occurrences of the same aspect:

where fi is the term frequency of ai, E(ai,A) (or E(ai, T )) equals to 1 if ai is an element of A (or T ), otherwise E(ai, A) (or E(ai, T )) equals to 0.

5.3 Compared Approaches

In the experiments, we compare our approach with DP [Qiu et al., 2011] and CRF [Jakob and Gurevych, 2010]. The reason to compare with DP is that we use DP rules as the input of our algorithm. We compare also with CRF as our approach is supervised. In total, we consider the following approaches, DP, DP+, RS-DP, RS-DP+, CRF and CRF+.

DP denotes the original double propagation algorithm in [Qiu et al., 2011]. It uses 8 aspect extraction patterns, which can be expanded into rules after instantiating the relation variables in each pattern with 8 dependency relations (mod, pnmod, subj, s, obj, obj2, desc and conj) defined in MiniPar 2 . Since Stanford Parser 3 is employed in our experiments, we use the corresponding dependency relations (amod, prep, nsubj, csubj, xsubj, dobj, iobj and conj) defined in Stanford Parser.

DP+ still uses the 8 aspect extraction patterns as in DP. The difference is that DP+ uses more dependency relations in the patterns. DP+ uses 18 dependency relations, i.e., amod, prep, nsubj, csubj, xsubj, dobj, iobj, conj, advmod, dep, cop, mark, nsubjpass, pobj, acomp, xcomp, csubjpass, and poss.

RS-DP implements the proposed Algorithm 1 using all the rules of DP.

2 http://webdocs.cs.ualberta.ca/∼lindek/minipar.htm

3 http://nlp.stanford.edu:8080/parser/

RS-DP+ implements the proposed Algorithm 1 using all the rules of DP+.

CRF is the supervised CRF-based aspect extraction method proposed in [Jakob and Gurevych, 2010].

CRF+ is the same CRF-based method but with much more dependency features than CRF. The dependency relations between opinion words and other words used in CRF+ are the same as those in DP+ and RS-DP+.

DP+, RS-DP+ and CRF+ are designed to explore the effectiveness of adding more rules into the rule set.

Cross domain test. Unlike traditional supervised learning where when a model is learned from a domain, it is also tested in the same domain, we test across domains because syntactical extraction rules are meant to be domain independent. Otherwise, it defeats the purpose of using such rules. Since the datasets D1 to D5 are widely used, and D6 to D8 are newly built, we evaluate these approaches on D1 to D5, and D6 to D8 separately. In testing RS-DP, RS-DP+, CRF and CRF+, to reflect cross domain aspect extraction, we use leave-oneout cross validation for D1 to D5, i.e., the algorithm selects rules based on the annotated data from four products, and tests the selected rules using the unseen data from the remaining product; for D6 to D8, the algorithm selects rules based on the annotated data from D1 to D5, and tests the selected rules using each of the data from D6 to D8. This simulates the situation that the selected rules can be applied to any domain (or in a domain independent matter).

In our experiments, all the approaches use Stanford Parser for part-of-speech tagging and dependency parsing. Note that, we extract not only single noun aspects but also noun phrases (multi-word expressions) in all the six approaches. For the approaches DP, DP+, RS-DP and RS-DP+, noun phrases are identified based on the dependency relation nn (noun compound modifier) defined in Stanford Parser, then the identified noun phrases are treated as ordinary nouns in the rules. In CRF and CRF+, noun phrases are identified by the CRF algorithm, which is the same as in [Jakob and Gurevych, 2010].

5.4 Experimental Results

Table 2 shows the results of DP, DP+, RS-DP, RS-DP+, CRF and CRF+ tested on D1 to D8 and evaluated based on multiple occurrences of each aspect term (evaluation (1)). Table 3 shows the corresponding results of the approaches evaluated based on distinct aspect terms (evaluation (2)).

In our experiments, DP and DP+ are directly tested on each dataset as the rules are domain or dataset independent. For RS-DP, RS-DP+, CRF and CRF+, as mentioned earlier, to test across domains D1 to D5, we use leave-one-out cross validation, and to test D6 to D8, we use the rules selected from D1 to D5.

From the tables, we observe that the F1-scores of RS-DP and RS-DP+ are markedly better than those of DP, DP+, CRF and CRF+. RS-DP+ is also better than RS-DP because RSDP+ uses more rules although some of the additional rules were not of high quality, which can be seen from the average F1-scores of DP and DP+.

Specifically, as shown in Table 2 and Table 3, the average precision of DP is higher than that of DP+, and the average recall of DP is lower than that of DP+. This is because DP+ uses more rules (due to 10 additional dependency relations) than DP. The new rules bring more correct aspects (higher recall) but also more errors (low precision). Since the average F1-score of DP is higher than that of DP+, we can see that adding rules arbitrarily can harm the overall results for the existing approach in [Qiu et al., 2011].

From Table 2 and Table 3, we can also see that the average precisions of RS-DP and RS-DP+ are dramatically higher than those of DP and DP+ respectively. Although there is some loss in recall, this is expected because RS-DP and RS-DP+ have less rules. However, the final F1-scores of RS-DP and RS-DP+ are markedly better than those of DP and DP+ respectively.

We also observe that the average precisions of RS-DP and RS-DP+ are almost the same, but the average recall of RSDP+ is higher. This shows that the proposed method is able to take good advantage of the additional rules (in RS-DP+), maintaining a high recall with almost no loss in precision, which finally translates to the high F1-score of RS-DP+.

From the tables, we further see that the recall of CRF+ is improved on almost every dataset compared with CRF, although the precision of CRF+ decreases slightly on some datasets. The overall F1-score of CRF+ is improved by about 6% on D1 to D5, and about 3% on D6 to D8 compared with CRF. This shows that the CRF-based approach is also able to make good use of additional rules. Note that, CRF+ uses the same dependency relations as in RS-DP+. However, both CRF and CRF+ are much poorer than RS-DP and RS-DP+. We believe one of the key reasons is that the rule-based approach performs propagation and make improvements iteratively, which the CRF-based method does not do.

In summary, we can conclude that the proposed approach can take rules of any quality and select a good subset to produce much better results than existing state-of-the-art rulebased and CRF-based approaches.

6 Conclusion

This paper proposed an automated rule set selection/learning method with the goal of improving the syntactical rule-based approach to aspect extraction in opinion mining. The original set of rules can be user-designed or learned by a system and the rules can be of any quality. The proposed technique can select a good subset of the given rules to perform much better extraction than the original rule set because our method can select rules which work best when used together. The experimental results demonstrated its superior performance. We also compared it with the state-of-the-art supervised statistical/learning method CRF. The proposed technique is much more effective. In our future work, we plan to employ semantic rule patterns, which can be learned or designed based on semantic parsing in addition to syntactic rule patterns as in the DP method. We also plan to explore other possible algorithms for rule selection, such as simulated annealing strategies and genetic algorithms.

Acknowledgments

Zhiqiang Gao's research was supported by the 863 program grant 2015AA015406 and the NSF of China grant 61170165. Bing Liu's research was partially supported by the NSF grants IIS-1111092 and IIS-1407927, and a Google faculty award. Yuanlin Zhang's work was partially supported by the NSF grant IIS-1018031. We would also like to thank Yun Lou and Pooja Sampelly for valuable discussions.