Knowledge-enhanced Hierarchical Attention for Community Question Answering with Multi-task and Adaptive Learning

1 Introduction

Community question answering (CQA) websites, e.g., Stack Overflow and Quora have attracted increasing attention for people to get free advice directly from experienced users. With the influx of new questions and the varied quality of provided answers, it is time-consuming for users to inspect them all. Therefore, developing automated methods to identify valuable answers for a given question is of practical importance.

Earlier efforts for community question answering pay particular attention to designing various features, such as frequency features and translation features, to rank the answers. Nevertheless, these approaches are built on plenty of hand-crafted features and their performance easily stagnates. Recently, many deep learning techniques, such as long short-term memory (LSTM) and convolutional neural network (CNN) have been employed for answer selection in CQA. These methods can capture the semantics of documents and avoid labor-intensive feature engineering. Instead of encoding the question and answer representations separately, attention mechanisms have been applied to capture the correlations between the question and answer, and give more weights on relevant parts of the input for better answer selection.

∗Ying Shen is corresponding author.

Despite the successes achieved by prior work, we argue that CQA in real-world remains challenging for several reasons. First, the external factual knowledge in knowledge bases (KBs), such as Freebase [Bollacker et al., 2008], provides rich information of entities and relations between them, and highlights the features that are essential for CQA. Despite its usefulness, the external knowledge from KBs is underutilized in recent deep learning based CQA systems. Second, the principle of semantic compositionality of a document is vital for text comprehension. To that end, a CQA model should focus on different levels of semantics of the input sequences and acquire comprehensive information to identify suitable answers. However, most existing methods do not exploit the semantic compositionality when learning the document representation. Third, a CQA system that does not identify the categories of input questions may learn a document encoder losing the crucial and discriminative features in questions. Forth, one of the most challenging issues of existing deep neural networks is that their performance may not be stable and can not effectively handle the quite imbalanced and noisy data in CQA.

In this paper, the mentioned challenges are considered and alleviated to some extent. We propose a Knowledgeenhanced Hierarchical Attention for community question answering with multi-task learning and adaptive learning. First, we propose a knowledge-enhanced hierarchical attention mechanism to fully explore the knowledge from input text documents and KB at different levels of granularity. In particular, we design a three-stage attention mechanism, including word-level attention, phrase-level attention, document-level attention, to integrate factual knowledge from KB into the representation learning of questions and answers, exploiting the semantic compositionality of the context and knowledge sequences. In addition, a multi-head co-attention network is proposed to capture correlations between the question and answer. Second, we propose a multi-task learning framework, in which the knowledge-enhanced representation learning is simultaneously optimized by two coupled tasks: CQA (primary task) and question categorization (auxiliary task). The main purpose of multi-task learning is to improve the quality of locating the salient information. Third, for CQA task, we construct multiple classifiers and ensemble their results as the final prediction result that is expected to more effectively and robustly solve the CQA task.

We summarize the main contributions of this paper as follows. (1) We incorporate the external knowledge from KB into deep neural networks to extract important information from the questions/answers that may be noisy and redundant by proposing a knowledge-enhanced hierarchical attention mechanism. (2) We leverage the question categorization to enhance the question representation learning for CQA, which improves the quality of identifying discriminative features in questions. (3) To the best of our knowledge, we are the first to combine adaptive boosting and deep neural networks for CQA task. The ensemble results from multiple models can be more accurate and robust. (4) Experiments on two benchmark CQA datasets show that KHAMA significantly outperforms the state-of-the-art baseline methods.

2 RelatedWork

Conventional CQA methods focused on designing different kinds of features to enhance the performance of CQA. Surdeanu et al. [2008] investigated a variety of features, e.g., frequency features, similarity features, and translation features to rank answers for the given question. Heilman and Smith [2010] applied the logistic regression algorithm and tree kernel function to recognize the correlations between the question and answer. Tran et al. [2015] utilized topic model based features to forecast the quality of answers. Although these methods have achieved remarkable success in answer selection, the performance of above methods depends on the hand-crafted features, whose definition is time-consuming.

Recently, deep learning methods make a breakthrough and become the mainstream to tackle the CQA task by encoding questions and answers into continuous vector representations without heavy feature engineering. Qiu and Huang [2015] introduced a convolutional neural tensor network to model the interaction between sentences with tensor layers. Wang and Nyberg [2015] applied the bidirectional LSTM to learn the representations of the question and answer, and the semantic matching scores of QA pairs were then computed. Tay et al. [2017] presented Holographic Dual LSTM (HDLSTM) to incorporate holographic representational learning into question answering (QA) semantic matching. Guo et al. [2017] introduced a skip CNN to learn the lexical semantic features.

Attention mechanism has been proved to be able to significantly improve experimental results on answer selection task by enhancing the interaction between QA pairs [Dos Santos et al., 2016; Chen et al., 2017; Shen et al., 2018; Yang et al., 2019]. Yin et al. [2015] presented an attentionbased CNN to incorporate mutual influence between sentences into CNNs. Dos Santos et al. [2016] presented attention-pooling (AP), a two-way attention mechanism, to map the question and answer into a common feature representation space so as to capture the comprehensive correlations of QA pairs for semantic matching. Tan et al. [2016a] integrated CNN and LSTM to capture the complex semantic relations of questions and answers, which combined the advantages of capturing linguistic information from both networks. Chen et al. [2017] proposes a positional attention based RNN model to integrate positional information into attentive representation. Despite the effectiveness of these studies, they exclusively consider context information rather than real-world background knowledge and hidden information beyond the context. Tay et al. [2018] proposed a cross temporal recurrent network to control the information retained or discarded over time, in which temporal gates are first learned and then applied to amend the question and answer representations temporally.

3 Our Methodology

3.1 ProblemDefinition

Given a question q and an answer a, CQA task aims at inferring the label Y ∈ {Good, Bad}, where y = Good means that answer a is qualified for question q. We denote the question q and the answer a as q = [wq1 ,wq2 , . . . ,wqn ] and a = [wa 1 , wa 2 , . . . , wam], where n and m represent the lengths of q and a respectively. Since we leverage question categorization task to improve the performance of CQA, we also assume that there is a category label x for each question q. To prevent conceptual confusion, we use superscripts "q" and "a" to represent the variables that are related to questions and answers, respectively.

3.2 The Overall Architecture

KHAMA consists of five modules. Knowledge-enhanced representation learning module contains three key layers: word-level mutual attention layer, phrase-level attention layer, document-level attention layer, motivated by [Lei et al., 2018]. Interactive question/answer representation module uses a multi-head co-attention network to model the interaction information between questions and answers. Text categorization module predicts a category label to the input question, and the category information is then fed into the question representation to learn a category-specific document representation. In community question answeringmodule, we construct multiple CQA classifiers and ensemble their results as the final prediction result to more effectively and robustly solve the CQA problem. KHAMA is trained with the multitask learning module, which regards CQA as primary task and question categorization as auxiliary task. Next, we will introduce each part of our KHAMA model in detail.

3.3 Knowledge-enhanced Representation Learning with Hierarchical Attention

Word-level Mutual Attention

Word Embedding The words which share common components (e.g., prefix, root, suffix) may be potentially related. Therefore, we design a joint word embedding layer to combine the merits of word-level and character-level representations. For the word-level embedding model, we adopt the popular word2vec [Mikolov et al., 2013] embeddings which are widely used in NLP domain. For the character-level embedding model, the ELMo language model [Peters et al., 2018] is used due to its superior performance in a wide range of NLP tasks. Then, each word is represented as a concatenation of the character-level embedding and word-level embedding, resulting in a hybrid word embedding vector et ∈ Rde for each word wt. Here, de denotes the size of the hybrid word embedding. The context representations of question q and answer a thus can be represented as Eq = [eq1 , . . . , eqn ] and Ea = [ea1 , . . . , eam], respectively.

We conduct entity mention detection by n-gram matching and obtain top-K entity candidates from KB for each entity mention in input documents. The entity embeddings in KB are learned via DeepWalk [Perozzi et al., 2014]. Formally, the candidate entities for the t-th entity mention as {ent t1, ent t2, ..., ent tK} ∈ RK×dkb , where dkb is the dimension of the entity embedding in KB. A context-guided attention model is designed to compute the knowledge representation of each entity mention in the document, which is computed as:

where ν is the attention function (i.e., a multilayer perceptron), E represents the context representation of the question or answer. Thus, we get the knowledge representations for question q and answer a, denoted as ˜Eq = [˜eq1 , . . . , ˜eqn ] and ˜Ea = [˜ea1 , . . . , ˜eam], respectively.

After obtaining the entity and word embedding, we design a word-level mutual attention mechanism to identify the relations between context and knowledge representations. Formally, we adopt the dot-product between the context and knowledge representations to calculate the correlation matrix Mq for question q as:

where each element inMqi,j refers to the correlation between the i-th element in the context representation and the j-th element in the knowledge representation.

Next, we average the values of each row and each column of Mq as attention sources to calculate attention vectors λq and ˜λ˜λ˜λq for context and knowledge representations respectively.

Finally, the knowledge-enhanced context representation matrix W q and context-enhanced knowledge representation matrix ˜W q can be computed as:

where U w and ˜U w are word-level projection parameters, Iq = [1, . . . , 1]T denotes a d-dimensional all-ones vector, Iq ⊗λq denotes the kronecker product operation between Iq and λq , ⊙ refers to the element-wise multiplication.

Phrase-level Attention

After obtaining W q and ˜W q , we adopt n-gram convolution operation to extract local semantic features. The convolution operation involves a filter Kq , which is applied to l continuous words. We assume that the feature maps for context and knowledge representations are P q and ˜ P q .

where b is a biasmatrix, represents the convolution operator, k indicates the number of filters

A phrase-level attention mechanism is designed to learn important local n-gram chunks. Mathematically, we formulate the chunk-based attention mechanism as follows:

where Cq is the correlation matrix between the context word chunks and the knowledge entities chunks. ⊙ denotes element-wise multiplication. U p and ˜U p are parameters to be learned for phrase-level attention. P q and ˜ P q refer to the phrase-level knowledge-enhanced context representation and context-enhanced knowledge representation.

Next, we employ two independent LSTM networks to encode hidden states of the context and knowledge representations as Hq = LSTM(P q ) and ˜Hq = LSTM( ˜ P q ), respectively.

Document-level Attention

We use knowledge representation as attention source to attend to the context so as to select those crucial knowledgeenhanced context word chunks to compose the knowledgeaware document representation. Formally, document-level attention is defined as follows:

where µ is mean operation, ρ is the attention function, U d 1 and U d 2 are document-level parameters, Oq is the documentlevel knowledge-enhanced representation for question q.

In the same way, we can get the knowledge-enhanced answer representation Oa .

3.4 Interactive Question/Answer Representation Learning

We propose a multi-head co-attention (MC) network to learn the interaction information between questions and answers. MC learns a 2-dimensional attention matrix. Given the knowledge-enhanced question and answer representations (i.e., Oq and Oa ), the attention matrix Σq for the answeraware question representation is computed as:

where Σqi ∈ Rb denotes the i-th row of the attention matrix, b is the number of hops of attention. In the same way, we can compute the attention matrix Σa for the question-aware answer representation.

After computing the multi-perspective co-attention matrices for the question and answer representations, we can obtain the interactive representations for question q and answer a, which are computed as:

where flatten is the operation that flattens matrix into vector form.

3.5 Question Categorization

Given a question q, the goal of question categorization is to assign a category label to the question. Question categorization facilitates category-specific question representation learning, which enhances the quality of identifying discriminative features of the given question. It can be treated as a multi-class text classification task.

The question representation emb q is fed into a task-specific fully-connected layer followed by a softmax layer to predict the category distribution ˆx of question q:

where fFC is a multilayer perceptron (MLP).

We optimize this question categorization model by minimizing the cross-entropy between the predicted category distribution ˆx and the true category distribution x (one-hot vector):

where N is the number of training samples, xi is the ground truth category label (one-hot vector) of the i-th question.

3.6 Community Question Answering

Category-aware Representation Learning

Inspired by [Cao et al., 2017], we develop a category-aware transformation process to make the transformed question embeddings hold the category information. Formally, our model transforms the question representation emb q into a categoryspecific question representation ˜ emb q by

where Wµ ∈ Rdc is a transformation matrix, dc is the dimensionality of the category-aware question representation. It is noteworthy that the knowledge-aware question representation and the category-specific question representation have the same dimensionality.

We develop the category-aware transformation matrix Wµ conforming to the predicted question category so that the transformed question representation could capture category information. To that end, we introduce X sub-matrices (W1µ , · · · ,WX µ ), where each sub-matrix is respective to one question category. The category-aware transformation matrix Wµ can be calculated as the weighted sum of the submatrices: Wµ = ∑Xi=1 ˆxWiµ . Here, X is the number of question categories. In this manner, Wµ can be biased to the sub-matrix of the predicted question category.

For CQA task, we concatenate the final question and answer representations and feed them into a task-specific fullyconnected layer. A softmax function is then employed to predict the answer probability distribution:

where fFC is a multilayer perceptron (MLP).

Similar to Eq. (16), we also optimize the CQA model with supervised learning by minimizing the cross-entropy between the prediction label distribution ˆ Y and the true distribution Y :

where M is the number of question-answer pairs, yi is the ground truth label (one-hot vector) of the i-th questionanswer pair.

Ensemble Learning

We randomly choose 100 questions that are incorrectly predicted by KHAMA from SemEval-2017 test set to perform error analysis. We observe that the incorrectly predicted questions are imbalanced across question categories. The samples in some question categories such as "Family Life in Qatar" and "Moving to Qatar" are more difficult to predict than the samples in other categories such as "Doha Shopping" and "Cars and driving". To develop a more stable and robust CQA model, we construct multiple CQA classifiers and ensemble their results as the final prediction result. Inspired by [Yang et al., 2018], multiple CQA classifiers are combined together according to their weight vector α, learned from the adaptive boosting algorithm. The weight αi of the i-th CQA classifier gi is updated based on the training error ϵi of gi on the training set, computed by:

The final prediction model G is obtained by weighted voting:

where αi means the weight of each CQA classifier gi for our final predictor Y . The softmax function is to predict the labels of CQA.

In the adaptive boosting algorithm, the individual classifiers are trained hierarchically to learn harder and harder samples of the classification problem. The i-th classifier is trained with more emphasis on different input samples, which is based on a probability distribution Di = {d2, . . . , dN} to re-weighing the training samples. The process of the boosting algorithm is given in Algorithm 1.

Algorithm 1 The process of adaptive boosting.

1. Input: The training set {Q1:N,A1:N}; Initialize the sample weights dn = 1/N, n = 1, 2, . . . , N.
2. For i = 1 to T do:
3. (a) Train a CQAmodel gi with the training data.
4. (b) Compute the training error ϵi of gi.
5. (c) Compute the classifier weight αi based on Eq. 20.
6. (d) Update the sample weight: dn = dn ∗ (1 -αi), if Yn = gi(Qn, An); otherwise, dn = dn ∗ (1 + αi).
7. (e) Normalize the sample weight vector Di.
8. (f) Re-sample training data with weights Di.
9. end for
10. Get the final prediction model Y based on Eq. 21.

3.7 Multi-task Learning

Overall, the proposed KHAMAmodel has two subtasks, each has a training objective. To strengthen the shared question representation learning module, the two related tasks are optimized jointly. Formally, the objective function for the multitask learning is minimized by:

where λ1 and λ2 control the importance of L1 and L2. We empirically show that setting λ1 = 0.2 and λ2 = 0.8 achieves the best performance of our model.

4 Experimental Setup

Experimental Datasets We evaluate our model on two benchmark datasets, i.e. SemEval-2015 Task 3 [Nakov et al., 2015] and SemEval-2017 Task 3 [Nakov et al., 2017], which consist of real-life data from the QatarLiving forum 1 . Table 1 shows the statistics of the datasets. Several candidate answers are given for each question, and each answer has a label "Definitely Relevant" (Good), "Potentially Useful" (Potential), or "Bad" (bad, dialog, non-English, other). Following the strategy as used in previous studies [Filice et al., 2016], both the "Potentially Useful" and "Bad" are considered as "Bad" in all experiments because the "Potentially Useful" is the noisiest and smallest class. Some metadata is given for each question, e.g., the question category and the type of question.

Compared Methods We evaluate and compare KHAMA with several state-of-the-art baseline models, including JAIST [Tran et al., 2015], KeLP [Filice et al., 2016], CNN [Yu et al., 2014], LSTM [Tan et al., 2016b], Bi-LSTM-attention [Tan et al., 2016b], BGMN [Wu et al., 2017], CNN-LSTM-CRF [Xiang et al., 2016], AP-LSTM [Dos Santos et al., 2016], AICNN [Zhang et al., 2017].

1 http://www.qatarliving.com/forum

Evaluation Metrics We use the official evaluation measures [Dos Santos et al., 2016; Zhang et al., 2017] to verify KHAMA. Specifically, for SemEval-2015 dataset, two popular classification measures, accuracy and Macro F1 score, are used to evaluate the performance of CQA. For SemEval-2017 dataset, we use three measures to evaluate KHAMA, including mean average precision (MAP), accuracy and Macro F1 score. MAP is a widely used metric for evaluating ranking algorithms, which considers the ranks of the returned documents.

Implementation Details We use a subset of Freebase (FB5M3) as our KB, which includes 4,904,397 entities, 7,523 relations, and 22,441,880 facts. We initialize the graph embeddings through choosing the values from normal distribution N(0, 1) and set the size of the graph embedding as 100. We use the pre-trained word2vec [Mikolov et al., 2013] with 100-dimensional embeddings to initialize the word embeddings, and initialize the word embeddings of out-ofvocabulary words as zero vector. The weight parameters are initialized using the Xavier uniform initializer, and the bias terms are initialized to zero. Both the hidden size of LSTM and the number of feature maps of CNN are set to 200. The size of each convolution filter is set to 2. We set the number of predictors in our ensemble learning to 5 (i.e., T = 5) and the number of entity candidates from KB for each entity mention to 6 (i.e., K = 6). We use Adadelta optimizer with a initial learning rate of 1×10-4 . Batch size is set as 64. L2 regularization (weight decay = 0.001) and dropout strategy (dropout rate = 0.2) are used to avoid overfitting.

5 Experimental Results

5.1 Quantitative Evaluation

Tables 2-3 reports the results of KHAMA and compared models on SemEval-2015 and SemEval-2017 datasets, respectively. KHAMA achieves significantly better performance than the state-of-the-art competitors on the two datasets. For example, for the accuracy the proposed KHAMA method substantially and consistently outperforms other methods (improves 4.66% on SemEval-2015 and 4.78% on SemEval2017). As we know, it is difficult to improve 1 percent of accuracy for CQA.

5.2 Ablation Study

To investigate the effectiveness of different factors of KHAMA, we also conduct ablation test in terms of removing factual knowledge from KB (w/o knowledge), question categorization task (w/o categorization), boosting algorithm (w/o boosting), word-level attention (w/o word-level), phraselevel attention (w/o phrase-level), document-level attention (w/o doc-level), intra-document attention (w/o intra-doc.), respectively. In particular, for the model without knowledge (i.e., w/o IA), we eliminate the word-level, phrase-level and document-level attention layers.

The ablation results are illustrated in Tables 2-3 (last seven rows). In general, combining all the factors achieves the best performance on all evaluation metrics. From Tables 2-3, we can see that the accuracy and F1 score decrease sharply when discarding the factual knowledge in KB and adaptive boosting algorithm. This is within our expectation since KB introduces background knowledge beyond the context to enrich the text representations and helps the model focusing on useful information. In addition, ensembling multiple classifiers with adaptive boosting could build a more effective and robust CQA model. Question categorization task also contributes to the effectiveness of KHAMA. This suggests that the auxiliary task improves the document representation learning and helps to identify discriminative features of the question.

5.3 The Effect of Parameters T and K

T is the number of classifiers in our ensemble learning. In this experiment, we analyze the impact of T on the overall performance of KHAMA by varying its value from 1 to 10 with step size 1 on the two datasets. We illustrate the experimental results in Figure 1 (left). As T increases from 1 to 3, the accuracy and F1 scores increase sharply till T = 3, after which the results become stable and slightly increase.

K represents the number of entity candidates from KB for each entity mention in questions and answers. In this experiment, we investigate the impact of K on the overall performance of KHAMA through varying its value from 1 to 10 with step size 1. The experimental results on SemEval-2015 are reported in Figure 1 (right). KHAMA obtains the best results when K = 5. As K increases from 1 to 10, the accuracy and F1 scores grow sharply till an optimal value (when K = 5), after which the accuracy and F1 scores decrease slightly.

6 Conclusion

In this paper, we propose a knowledge-enhanced hierarchical attention for community question answering with multi-task learning and adaptive learning. We leverage external knowledge from the knowledge base (KB) to learn better representations of questions and answers by exploiting the semantic compositionality of the input sequences. In addition, we combine multiple CQA models with adaptive boosting to learn a more effective and robust CQA system. Extensive experiments on two benchmark datasets show that KHAMA obtains significantly better results than the compared methods.

Acknowledgments

This work was also partially supported by the SIAT Innovation Program for Excellent Young Researchers (No. Y8G027), Shenzhen Fundamental Research Project (No. JCYJ20180302145633177), NSFC (No. 61876208), Guangdong Provincial Scientific and Technological funds (2017B090901008, 2018B010108002). Min Yang was sponsored by CCF-Tencent Open Research Fund.