Quality Matters: Assessing cQA Pair Quality via Transductive Multi-View Learning

1 Introduction

As compared to the traditional factoid QA systems, community-based QA (cQA) systems are user-centered that leverage crowdsourcing platforms to encourage users to ask and/or answer questions, and further provide rich knowledge to search engines. Nevertheless, the quality of cQA contents varies largely due to users' diverse expertise and the low-cost posting behaviors. Low-quality cQA pairs can cause some serious problems including, but not limited to, the followings: 1) They may mislead information seekers and hence impact their search experience. 2) A large portion of storage and computing resources are wasted on low-quality contents. Instead, desired information are not acquired. 3) They decrease the stickiness of community users. Once these sites are overwhelmed by low-quality information, users will be gradually unwilling to keep browsing them. To gain insights into cQA quality, we randomly selected 100 questions and their corresponding answers from a cQA site StackExchange, and invited three volunteers to rate these cQA pairs on the scale from 1 (poor) to 5 (excellent). According to our statistics, we found that more than 45% of cQA pairs are rated as unsatisfactory (bellow 3 on average). The study result is consistent with what has been reported by Jeon et al. [2006]. To handle such issues, we target at assessing the quality of cQA pairs.

∗Heyan Huang is the corresponding author.

In essence, great efforts have been dedicated to predicting the quality of cQA contents [Shah and Pomerantz, 2010; Dalip et al., 2013; Nie et al., 2017]. Despite their success, most existing methods simply concatenated various features into single vectors first and then fed them into regression models. However, this may be suboptimal, due to: 1) Week Capacity in Multi-view Fusion. The quality of cQA pairs is influenced by distinct views, and each of them describes a specific but incomplete angle. We carefully went through aforementioned cQA pairs and found that their quality are determined by five main factors, including question, answer, QA relevance, asker, and answerer. Figure 1 illustrates a selected example. Even the answerer is experienced and he/she resolves the question with a well-organized answer, the cQA pair can hardly be considered as a high-quality one, as the asker is of less experience and the question is full of typos. So, overlooking the agreement among these views may cause an unfair judgement. 2) Data Deficiency. In most previous work, only labeled cQA pairs are utilized to train the regression model. Yet, it is expensive to gather sufficient labeled cQA pairs in practice. Instead, unlabeled cQA pairs are much cheaper to obtain, and they also contains rich information. However, these unlabeled cQA pairs are not fully explored.

Toward this end, we propose a Transductive Multi-view Learning model, dubbed as TMvL, to assess the cQA pair quality. Specifically, multi-view learning is conducted to solve the multi-view fusion problem. It seamlessly takes the view agreement into account by learning a common space shared by all views, and the desired optimal common space maintains the original intrinsic characteristics of cQA pairs in the original spaces. Meanwhile, transductive learning is applied to involve unlabeled cQA pairs into the common space learning, so that the data deficiency problem could be solved.

The main contributions of this paper are three-fold. 1) As far as we know, this is the first work applying multi-view learning to cQA pair quality assessment, where the agreement among different views of cQA pairs are fully explored. 2) We study the utility of unlabeled cQA pairs in quality assessment and propose a transductive model to fully incorporate unlabeled data. 3) Comparative experiments with both automatic and manual evaluation on our self-collected dataset well-validate the proposed model. Additionally, we have released our codes and data to facilitate follow-on researchers 1 .

2 RelatedWork

The most related work is on answer quality prediction via maximum entropy [Jeon et al., 2006]. Most follow-on efforts explored different features, such as answer content [Surdeanu et al., 2008], domain knowledge [Nie et al., 2015b], and embedding features [Zhang et al., 2017]. Dalip et al. [2013] summarized commonly used features and analyzed their effectiveness. Beyond exploring features, other novel methods were also developed, including propagation method [Li et al., 2015] and click model [Wei et al., 2015]. Different from the previous efforts on exploring answer quality or question quality, we mainly focus on assessing the quality of cQA pairs.

Multi-view learning originates to learn from multiple sources or different feature subsets by considering the agreement of different views. They can be roughly categorized into three groups: co-training, multiple kernel learning, and subspace learning. Co-training alternately maximizes the mutual agreement on two distinct views. Co-EM [Nigam and Ghani, 2000], Bayesian Co-Training [Yu et al., 2011], and Co-Regression [Zhou and Li, 2005] belong to this category. Multiple kernel learning [Sonnenburg et al., 2005] exploits different kernels to different views, and then combines them together to enhance the performance. Subspace learning works by finding a latent subspace shared by each view, assuming that each input view is generated from the common subspace. MSNL [Song et al., 2015a] , SM2L [Song et al., 2015b], aM2L [Nie et al., 2015a], and MvDA [Kan et al., 2016] fall into this category.

Transductive learning is quite different from the traditional inductive learning. It leverages rich information in the testing set while training the model. Various efforts have been dedicated for unlabeled data utilization, such as Transductive SVM [Joachims, 2001], CTA [Blum and Mitchell, 2000], and SSR [Kim et al., 2009]. Promising performance of transductive learning has demonstrated its effectiveness of incorporating unlabeled data.

TMvL belongs to both multi-view learning and transductive learning. Different from previous efforts, it learns a latent subspace by minimizing the disagreement between the common space and each view by jointly considering both labeled and unlabeled data.

1 http://datapublication.wixsite.com/tmvl.

3 cQA Pair Quality Assessment

3.1 Notations

We first declare some notations used in this paper. Suppose there are N labeled and M unlabeled cQA pairs in the dataset. Each pair is described by V views. Let x(v) i ∈ RD(v) denotes the feature vector of the i-th cQA pair in the v-th view, whereby D(v) refers to the feature dimension. The v-th view is represented as X(v) = [x(v) 1 , x(v) 2 , . . . , x(v) N+M ]T ∈ R(N+M)×D(v) . Quality labels of the N cQA pairs are vectorized as y(l) = [y(l) 1 , y(l) 2 , . . . , y(l) N ] ∈ RN , where y(l) i is the quality score for the i-th cQA pair. As to the M unlabeled cQA pairs, we aim to assess their quality y = [yN+1, yN+2, . . . , yN+M] ∈ RM , where yi denotes the inferred quality score of the i-th unlabeled cQA pair. Our research objective is to propagate the quality scores of labeled cQA pairs to unlabeled ones. This is a transductive learning model, in which both labeled and unlabeled cQA pairs are used for training. In particular, information of both labeled and unlabeled cQA pairs are simultaneously leveraged to learn a common space shared by multiple views. The common space is denoted as X(0) = [x(0) 1 , x(0) 2 , . . . , x(0) N+M ]T ∈ R(N+M)×D(0) , where x(0) i ∈ RD(0) indicates the representation of the i-th cQA pair in the common space.

3.2 ProposedModel

As reported in [Song et al., 2015a], the squared loss usually yields good performance compared with other complex ones in the regression problem. We followed this scheme by utilizing the squared loss function to measure the training error of the labeled cQA pairs. In TMvL, the quality of cQA pairs is assessed relied on the representation in the common space. We thus have the empirical loss,

where w ∈ RD(0) is our desired coefficient vector, mapping the cQA pair representation in the common feature space to the quality value space.

The common space is induced from V different views, and each view conveys a specific aspect of cQA pairs. To informatively and comprehensively characterize each QA pair, the original intrinsic characteristics of different views are reserved in the common space. The effectiveness of Laplacian matrix has been proven in data representation and clustering [Feng et al., 2017], which captures the inherent relatedness among data points, following the traditional spectral graph embedding theory. We thus turn to penalize the disagreement of normalized Laplacian matrix between the common space and each view, to ensure that the quality characteristics of different views can be preserved as much as possible.

It is formulated as,

where L(k) denotes the normalized Laplacian matrix, and it is calculated as,

where I ∈ R(N+M)×(N+M) is an identity matrix; S(k) and D(k) ∈ R(N+M)×(N+M) denote the affinity matrix and the degree matrix, respectively. In this paper, we adopt cosine similarity to measure S(k) . To decrease the complexity of the formula, we normalize x(k) n via enforcing ∥∥∥x(k) i ∥∥∥2 F = 1. In this way, the (m, n)-th element of S(k) is calculated as S(k) m,n = x(k) m T x(k) n . Inspired by [Liu et al., 2013], we set S(k) n,n = 0 to eliminate the self-loop problem. The degree matrix D(k) is a diagonal matrix, and the (m,m)-th element D(k) m,m is calculated as,

It is notable that the common space learning follows the main idea of transductive learning. Particularly, in the Laplacian matrix, each data point is represented by its relations with the others, including both labeled and unlabeled ones. Therefore, both labeled and unlabeled cQA pairs are equally utilized. This is why we claim that the common space X(0) is capable of harvesting unlabeled cQA pairs to enforce its representation ability.

To incorporate the learned common space into the quality assessment task and strengthen the prediction performance, we co-regularize the empirical loss together with Eqn.(2). We then reach the final objective function O(w,X(0) ),

where µ and λ are both non-negative regularization parameters. Specifically, µ penalizes the disagreement between the common space and each view, and λ controls the complexity.

3.3 Optimization

We adopt the alternating strategy to minimize our objective function. We first fix w and optimize O w.r.t X(0) with gradient descent. It is worth noting that the matrix X(0) is normalized after updating each parameter to ensure ∥∥∥x(0) i ∥∥∥2 F = 1.

As to w, we first fix X(0) , set ∂O(w,X(0) ) w to zero, and we then have the closed-form solution,

where X(0) = [x(0) 1 , x(0) 2 , . . . , x(0) N ]T is the representation of labeled samples in the common space, and I is an N ×N identity matrix.

4 Experiments

4.1 Dataset Construction

We crawled data from two compact subsites of StackExchange, i.e., "English" and "Game". We collected 26, 752 questions, their corresponding 92, 397 answers, and 28, 271 users (askers and answerers) from "English" subsite. Meanwhile, we gathered 28, 023 questions, their corresponding 59, 423 answers, and 24, 079 users from "Game" subsite.

We utilized user vote information on StackExchange to construct quality labels. For a given cQA pair, the quality label considers both its answer quality sa and question quality sq. We employed the normalized answer vote to represent sa, which is calculated as sa = va maxa({vqa}) , where va is the number of votes received by the answer a, and maxa({vqa }) indicates the largest number of votes among the answers under the question q. We next utilized the number of votes vq received by question q to infer sq. The quality of question is normalized as sq = vq maxq({vq}) . We then used the average score of sa and sq to represent the quality of a cQA pair.

To justify the rationality of the auto-generated labels, we randomly selected 50 popular questions from "English" site with more than 5,000 viewers and 50 unpopular questions with less than 500 views. Based on these criteria, 456 and 111 cQA pairs were involved, respectively. Three volunteers were invited to annotate these cQA pairs on the scales from 1 (poor) to 5 (excellent) according to pre-determined guidelines. We calculated the inter-rater agreement among volunteers, and the pair-wise Kappa coefficients are all larger than 0.6. This demonstrates the significant alignment among them. The average score is hence regarded as the rating of the cQA pair. We observed a similar rating distribution between popular and unpopular questions, as shown in Figure 2(a), which reflects that the question popularity has less influence on the real quality of cQA pairs. Figure 2(b) and Figure 2(c) plot relations between volunteer ratings and auto-generated quality labels on popular and unpopular questions, respectively.

We found that the quality of popular cQA pairs can be better simulated, where the Pearson's correlation is 0.81, but it does not perform well on unpopular questions, where the Pearson's correlation is only 0.26. We hence selected the top 5% questions with the largest number of views to automatically construct quality labels. In total, we obtained 4, 704 and 3, 043 labeled cQA pairs from these two datasets, respectively.

4.2 EvaluationMetrics

In order to comprehensively justify our TMvL model, we evaluated it in two different ways, i.e., Auto-Evaluation and

Manual Evaluation.

In auto-evaluation, we utilized the automatically generated labels to evaluate the performance. We randomly selected 20% cQA pairs as unlabeled samples as well as testing samples. Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) [Cao et al., 2017] were used as metrics.

Since auto-evaluation is fit for popular questions only, we hence manually rated a number of cQA pairs, including both popular and unpopular questions. In manual evaluation, the aforementioned 4, 704 and 3, 043 automatically labeled cQA pairs were all treated as labeled data, and we randomly selected 1, 000 cQA pairs from each subsite as unlabeled ones. It is worth noting that there is no overlap between the labeled and unlabeled cQA pairs. From these unlabeled cQA pairs, we further randomly selected 100 cQA pairs and invited three volunteers to annotate their quality scores from 1 (poor) to 5 (excellent). The pair-wise Kappa coefficients are all larger than 0.6, so the average scores of the three volunteer ratings were used as the ground truth. Since volunteer ratings are discrete values between 1 and 5, while the predicted scores are continuous between 0 and 1, Pearson's Correlation (Corr.) [Shah and Pomerantz, 2010] are utilized.

Detailed description of the data used in these two evaluation methods are displayed in Table ??.

4.3 Feature Extraction

We intend to comprehensively assess the quality of cQA pairs from five distinct views: question, answer, cQA relevance, asker, and answerer. To accomplish this, we extracted a rich set of quality-oriented features from each view. It should be noted that the purpose of this study is not to explore the discrimination of features, so all features we used in this paper have been demonstrated effective in prior work [Li et al., 2012; Surdeanu et al., 2008; Dalip et al., 2013].

We next describe features in each view. 1) Question View. We extracted three kinds of features to characterize the question quality, i.e., User Generated Content (UGC) features, linguistic features, and embedding features. The number of tags, answers, user favourites, and the question age belong to UGC features. Linguistic features include the number of words, non-stop words, and sentences, the average and the maximum length sentences, the percentage of punctuation, pronouns, conjunctions, prepositions, spaces, as well as sentences starting with pronouns, articles, conjunctions, or prepositions. We also extracted a 50-dimensional sentence embedding with Para2Vec [Le and Mikolov, 2014]. 2) Answer View. Similar to question view, we extracted these three kinds of features. The UGC features contain the number of comments and editors. As to the linguistic features, all that described in the question view were also applicable. A 50dimensional embedding feature was also extracted. 3) QA Relevance View. We extracted BM25, cosine similarity, the number of common words and common sequences. These features describe the QA relevance. Besides, the number of new adjectives, new nouns, and new verbs in answers, as well as the ratio of answer length to question length are extracted. They verify the linguistic difference. The time span between question and answer posting time are also extracted. 4) Asker View and Answerer View. We treated askers and answerers as two different views, but they share the same features space. We extracted features based on user achievements, such as the number of badges, reputations, and flags. Users' behaviour preference signals their attitude. We hence extracted the number of questions asked, answered, edited, and votes casted.

4.4 Baselines

We compared TMvL with the following baselines: 1) LR [Shah and Pomerantz, 2010]. This is a single-view method. We concatenated all features extracted from the five views, and then directly fed them into a logistic regression model. 2) Comp. [Liu et al., 2011]. We trained five logistic regression models on five views separately, and fed these regression results into a final regressor to predict the quality score. This is a single-view method, as view agreement is ignored. 3) SVR [Drucker et al., 1996]. Support Vector Regression is another single-view method, which concatenates features from different views into a single feature vector. We chose the learning formulation with the kernel of radial-basis function, as it outperforms other kernels in our experiments. 4) MSNL [Song et al., 2015a]. Multiple Social Network Learning is a multi-view learning method, which takes both view agreement and view confidence into account. 5) MvDA+SVR [Kan et al., 2016]. Multi-view Discriminant Analysis is another multi-view learning method. It aims to learn a single unified discriminant common space from multiple views by jointly optimizing multiple viewspecific transfor-ms. We then fed feature representations of five distinct views in the common space into a SVR model to make further prediction. 6) SSR [Kim et al., 2009]. SemiSupervised Regression is a transductive learning method. It uses the information of unlabeled data to construct normal coordinates around each unlabeled point, and the normal coordinates are then employed to estimate the Hessian regularizer.

(a) MAE on English Dataset.

(b) RMSE on English Dataset.

(c) MAE on Game Dataset.

(d) RMSE on Game Dataset.

Figure 3: Results of auto-evaluation by varying the portion of unlabeled cQA pairs on "English" and "Game" datasets, respectively.

7) TMvL sep. This baseline is the variant of our proposed model by optimizing Γ1 and Γ2 separately. It first learns representations of cQA pairs with Γ2. Then the representations of labeled ones are then fed into Γ1 to train the regression model. This method belongs to transductive learning.

4.5 Overall Performance Comparison

The overall performance comparison is summarized in Table 2. We have the following observations: 1) As compared to single-view methods, i.e., LR, Comp., and SVR, multiview learning methods, i.e., MSNL, MvDA, TMvL sep, and TMvL, perform better on both MAE and RMSE. Because the latter explores the relatedness among views to strengthen their combination. 2) Transductive learning methods, i.e., SSR, TMvL sep, and TMvL surpass the inductive ones, i.e., LR, Comp., and SVR. The main reason is that transductive learning-based approaches are capable of harvesting information from the unlabeled data. 3) Regarding the comparison among multi-view learning methods, TMvL and TMVL sep outperform others. This is because that unlabeled cQA pairs are utilized to learn the common space by TMvL and TMvL sep. This indicates that the use of unlabeled data is helpful to learn a reliable common space. 4) When comparing three transductive learning methods, we observed that TMvL and TMVL sep perform better. Even though both labeled and unlabeled cQA pairs were used, TMvL and TMVL sep separately considered different views and modeled the view agreement. This justifies that TMvL and TMVL sep can well preserve the characteristics of different views. 5) Compared to TMvL sep, TMvL co-regularizes both Γ1 and Γ2, so the common space learned by TMvL is guided by the quality label. The outstanding performance of TMvL demonstrates that under the supervision of labeled cQA pairs, the common space can better assess the quality of cQA pairs. 6) No matter in auto-evaluation or manual evaluation, we got quite similar results, which means that for unpopular questions, our auto-generated quality labels are reliable enough to guide the training procedure. 7) Results of pair-wise sig-

4.6 Transductive Learning Analysis

To justify the impact of incorporating unlabeled data by a transductive model, we conducted experiments by varying the number of unlabeled cQA pairs.

In auto-evaluation, we gradually increased the portion of unlabeled cQA pairs from 20% to 80%; the portion of labeled pairs therefore decreased accordingly. As shown in Figure 3, the performance of all methods gradually drops. Moreover, the performance of transductive learning methods, i.e., SSR, TMvL sep and TMvL remain relatively stable as compared to others. This shows that transductive models can better alleviate the data deficiency problem.

We then performed manual evaluation to further analyze the influence of unlabeled data. We kept the labeled cQA pairs and gradually increased unlabeled ones from 100 to 1, 000. Only transductive learning methods were compared here because the change of unlabeled data has no effect on inductive learning methods. The result is shown in Figure 4. We have the following observations: 1) The performance of all methods is enhanced by increasing the number of unlabeled cQA pairs. This demonstrates that transductive learning methods are able to learn a more reliable common space from unlabeled data. 2) Among these three methods, TMvL consistently performs the best. This is because view agreement are considered in the common space by TMvL.

• (b) Two selected cQA pair examples and their corresponding quality scores calculated with different methods.

Table 3: Incremental multi-view integration.

4.7 Evaluation on Unpopular Questions

Our quality labels were constructed with popular questions, we hence manually evaluate the performance on both popular and unpopular questions to demonstrate its applicability. The cQA pairs from those 50 popular and 50 unpopular questions annotated in section 4.1 are used as unlabeled samples, and 4, 704 auto-labeled pairs are labeled ones. As the result shown in Figure 5(a), we found the followings: 1) As we mentioned in section 4.1, the Pearson's Correlation between our generated quality labels and volunteer ratings on unpopular questions is 0.26. And we found that all these comparative methods achieve higher performance. In addition, the performance on labeled and unlabeled questions is quite similar. This demonstrates that the constructed quality labels with popular questions are reasonable to supervise the model training. 2) TMvL stably outperformed other baselines in both two settings. It demonstrates that the proposed model is applicable to both popular and unpopular questions.

Figure 5(b) illustrates two examples and the quality assessment results of volunteers and all methods. The quality of the first one is bad, and volunteer rating is 2.33 on average, much lower than 3. Similarly, TMvL assesses it a low score of 0.42. While other methods tend to consider it as a high-quality one. The second example is of high quality. Even though all methods gave it high scores, the score rated by TMvL is much closer to human judgements.

4.8 Incremental Analysis of View Impact

To demonstrate the effectiveness of these five views in assessing cQA pair quality, we incrementally fed them into TMvL. Results are displayed in Table 3. Both auto-evaluation and manual evaluation were conducted, and they led to the same conclusion. It is observed that more views lead to better performance, and the best performance is obtained when all these five views are integrated. This reveals that every view contains useful information and can be effectively encoded in the common space by TMvL.

(a) Similarity visualization.

(b) Quality label distribution.

Figure 6: Visualization of the similarity matrix.

4.9 Common Space Visualization

To demonstrate the rationale of the common space learned by TMvL, we visualized similarities among cQA pairs in the common space. All cQA pairs were first sorted according to their label values, so that cQA pairs of high-quality and lowquality can be separated. We then calculated the similarity matrix, where each entry denotes the cosine similarity of two given cQA pairs in the common space X(0) . As the result shown in Figure 6(a), almost all cQA pairs are closer to the ones with similar quality. They can hence be roughly grouped into two clusters, i.e., high-quality ones and low-quality ones. After analyzing the distribution of label values, we found that these cQA pairs can also be divided into two categories, as shown in Figure 6(b). These two separations statically match, and the dot lines in these two sub-figures illustrate the same separation. This demonstrates that with the supervision of labels, the common space is more reliable and separable to assess the cQA pair quality.

5 Conclusion

This paper presents a novel transductive multi-view learning model to assess cQA pair quality. It learns an optimal common space by jointly leveraging both labeled and unlabeled training samples of the given cQA pairs with multi-facets. The common space enables each view to maintain the intrinsic properties. Comparative experiments with both automatic and manual evaluation on real-world datasets have demonstrated the promising performance of our model.

Acknowledgments

The work was mainly supported by the National High Technology Research and Development Program of China (863 Program, No. 2015AA015404) and the National Natural Science Foundation of China (61751201). This research is part of NExT++ project, supported by the National Research Foundation, Prime Minister's Office, Singapore under its IRC@Singapore Funding Initiative.