Aggregating Crowd Wisdoms with Label-aware Autoencoders

1 Introduction

Aggregating crowd wisdoms is also known as label aggregation for crowdsourcing or truth discovery [Li et al., 2016]. It is an increasingly important topic in machine learning. Many tasks of machine learning require large labeling datasets. Traditional label collection from domain experts is usually expensive and time-consuming, which may not match the increasing requirement for labels. Labeling by the crowd has become popular with the blooming of online crowdsourcing platforms such as Amazon Mechanical Turk [Ipeirotis, 2010] and CrowdFlower [De Winter et al., 2015]. Such a platform divides the whole labeling task into small parts and distributes them to ordinary web users (sources). Despite of low cost, crowdsourced labeling commonly suffers from (much) lower accuracy than that from experts. Therefore in many labeling tasks, for each object we need to aggregate multiple labels from different users to reduce the labeling noise [Tian and Zhu, 2015a].

Label aggregation takes multiple labels from various sources as input and infers true labels for objects. This is a typical unsupervised learning task as there is no ground truth provided for inferring labels. The most simple and widely used method is majority voting [Aydin et al., 2014]. It treats sources equally and picks the most voted label as the true label. Recent research work mainly models source credibility (or capability). The underlying assumption is that sources with high credibility assign labels more accurately than those with low credibility [Yin et al., 2008; Li et al., 2014]. There are roughly three kinds of modeling frameworks: weighted majority voting, trust propagation, and generative models. Weighted majority voting is the direct extension from traditional majority voting [Aydin et al., 2014; Li et al., 2014]. Trust propagation models both credibility of sources and reliability of provided labels [Yin et al., 2008; Pasternack and Roth, 2010; Galland et al., 2010]. More recent work can be categorized into the generative framework [Whitehill et al., 2009; Welinder et al., 2010; Bachrach et al., 2012; Qi et al., 2013; Simpson et al., 2013; Tian and Zhu, 2015a]. These methods generate source labels from underlying (unknown) true labels by probabilistic models and infer true labels by MAP or Bayesian estimation. Though these methods have superior inference performance to majority voting, they need to model sophisticated relationships between source labels and inferred labels (by experts). There are two weak points of such models. One is that they are usually designed for data with typical characteristics but may not generalize to the data with some other characteristics. The other one is that even experts may improperly model relationships between source labels and inferred labels which limits the inference performance (e.g. missing effective factors or adding too many constraints).

In this paper, we propose a novel framework named LabelAware Autoencoders (LAA) to aggregate crowd wisdoms. By vectorizing source labels, label aggregation is simplified as a classification problem to predict true labels from source labels. Since label aggregation is unsupervised and there is no ground truth for training a classifier, we combine a classifier and a reconstructor into a unified framework. The idea is motivated by classical autoencoders which encode input into latent features in the hidden layer and reconstruct the input from latent features in the output layer [Vincent et al., 2008]. We can regard the classifier in LAA as an encoder and inferred labels as latent features of the input. To the best of our knowledge, it is the first trial to combine label aggregation with autoencoders.

The framework is flexible for various implementations. In this paper we adopt networks to implement the classifier and the reconstructor. Instead ofmanuallymodeling sophisticated relationships between source labels and inferred labels, networks have the potential to automatically learn those underlying patterns. That property makes proposed model domainfree and easy to implement for different data. To further improve inference accuracy, we introduce object ambiguity and latent aspects into the classifier and the reconstructor.

Experiments on three real-world datasets show that even the basic version of LAA has competitive inference performance with the state-of-the-art. Modeling object ambiguity and latent aspects further improves the inference accuracy significantly over the state-of-the-art. We also examine learned patterns in networks to support the effectiveness of proposed models.

2 RelatedWork

The research of label aggregation can be traced back to 1979. Dawid and Skene proposed a probabilistic model to aggregate observations for patients [Dawid and Skene, 1979]. Recent research about this topic rises with the concept of truth discovery [Yin et al., 2008].

Models of label aggregation can be roughly categorized into three frameworks: weighted majority voting, trust propagation, and generative models. Weighted majority voting is the direct extension from traditional majority voting [Aydin et al., 2014; Li et al., 2014]. The key of these methods is estimating source weights or credibility. Despite of mediocre inference accuracy, weighted majority voting is intuitive and easy to implement. Trust propagation models [Yin et al., 2008; Pasternack and Roth, 2010; Galland et al., 2010] assume labels provided by trustworthy (high credible) sources are more reliable and sources providing reliable labels are more trustworthy. Without prior structures or parameters, these models need a sufficient number of labels and may suffer from sparse data. More recent work usually utilizes the generative framework [Whitehill et al., 2009; Welinder et al., 2010; Bachrach et al., 2012; Qi et al., 2013; Simpson et al., 2013; Tian and Zhu, 2015a]. These methods generate source labels from the underlying (unknown) true labels by probabilistic models and are trained via maximizing a posteriori (MAP) or Bayesian estimation. Besides modeling source credibility, various factors are introduced with the flexibility of probabilistic models, such as object difficulty [Bachrach et al., 2012] and confusion matrix [Simpson et al., 2013]. Other interesting work about label aggregation includes truth existence modeling [Zhi et al., 2015], minimax conditional entropy [Zhou et al., 2012], rank aggregating [Metrikov et al., 2015], crowd clustering [Gomes et al., 2011], etc.

With the prevalence of deep learning and learning representations, autoencoders have become a widely adopted unsupervised model during the last five years [Vincent et al., 2010]. Autoencoders perform unsupervised learning in a supervised learning fashion: trying to recover the input through a network with a small-sized hidden layer [Vincent et al., 2008]. Recently, variational autoencoders [Kingma and Welling, 2013], a marriage between Bayesian inference and autoencoders, attract much attention in building deep generative models to learn data distributions.

Despite of the wide usage of autoencoders, to the best of our knowledge, there is no previous work on label aggregation that leverages autoencoders to learn the latent data patterns amongst the source labels and infer true labels.

3 Methods

3.1 ProblemDefinition

Suppose there are M objects, N sources, and a set of labels the sources give to objects. We denote lmn the label object om received from source sn, m ∈ {1, ..., M} and n ∈ {1, ..., N}. A categorical label lmn ∈ {1, ...,K} where K is the number of categories. The goal of label aggregation is to infer a true label ˜ ym for each object om.

3.2 Label Vectorization

We represent an object by a source label vector. For object om, vector vm of N × K dimensionality is constructed to contain all labeling information of the object. The vector can be divided into N blocks where each block contains K consecutive elements. The n-th block corresponds to the label given by source sn. We use source-wise one-hot encoding to make the vector discriminative between categories. Figure 1 illustrates the construction of a source label vector. Let vm nk denote the k-th element of block n for object om. If source sn labels om as category k (i.e. lmn = k), vm nk is set to 1 while the other elements of the same block are 0. If source sn does not assign any label to om, then all K elements of block n are set to 0. An accompanying mask vector δm is constructed to conveniently indicate whether source sn gives the label or not. δm has the same dimensionality as vm. If lmn exists, then all corresponding K elements of δm nk are set to 1 (k ∈ {1, ..., K}), 0 otherwise.

It is obvious that all the label vectors of different objects have the same dimensionality. This property makes following learning methods feasible. A source label vector is also called an input vector when used in the label-aware autoencoder.

3.3 Label-aware Autoencoders

Since a source label vector contains all given information of one object, we can exploit a model or a classifier to take a source label vector vm as input and output the true label ˜ ym. However, training a classifier in traditional supervised machine learning problems needs partial ground-truth labels, but label aggregation is totally unsupervised [Li et al., 2016]. Therefore, we propose a novel framework named Label-Aware Autoencoders (LAA) to infer true labels in such an unsupervised scenario. LAA integrates a classifier and a reconstructor into a unified model. The classifier infers true labels from input and the reconstructor reconstructs input from inferred labels in an unsupervised manner. Analogizing classical autoencoders, we can regard the classifier as an encoder, the reconstructor as a decoder, and inferred labels as latent features of input [Vincent et al., 2008].

Formally, we describe the mechanism of LAA from the view of maximizing the log-likelihood of input. For a given input vector v (the superscript m is omitted for simplicity), denote the classifier as qθ(y|v) and the reconstructor as pφ(v|y), where θ and φ are model parameters, y is the inferred label. LAA maximizes the lower bound of loglikelihood log p(v), which is an analogy to variational autoencoders [Kingma and Welling, 2013].

On the right hand side in formula (1), the first termmeasures the expectation of reconstruction quality. It encourages the probability pφ(v|y) to be 1 to achieve good reconstruction. The second term is the negative KL divergence between the distribution of inferred label qθ(y|v) and the prior distribution p(y), which acts as the regularization term to constrain the inferred label distribution to the prior one.

3.4 Network Implementation

In this paper we adopt networks to implement the classifier and the reconstructor in LAA. We first construct a basic version LAA-B. It does not need extra knowledge about sources or objects. Figure 2 illustrates the architecture. The classifier qθ(y|v) is modeled by a network where we obtain a label vector ¯ym from input vm.

where weight matrix wq corresponds to classifier parameter θ (the bias term is omitted for simplicity). σ(·) is the softmax operator to make ¯ym a distribution. ¯ym is a K-dimensional vector where K is the number of categories. We then sample ym from the distribution ¯ym. For convenience in the network, ym is one-hot encoded. The reconstructor is also modeled by a network which takes ym as input and reconstructs ˜vm as

where weight matrix wp corresponds to reconstructor parameter φ. ˜σ(·) is the source-wise softmax operator. The operator applies a softmax operator only on nodes in same source block. By the treatment, reconstructed ˜vm has the same structure as input vm. Then the reconstruction term log pφ(v|y) in formula (1) is equivalent to the negative cross entropy between input vm and reconstructed ˜vm.

where element ˜ vm nk of reconstructed vector ˜vm corresponds to vm nk of input vector vm, and δm nk is the corresponding element of mask vector δm. The mask vector makes the calculation focus on observed labels only.

A potential problem of inferring labels by networks is that nodes are exchangeable. For a label vector ¯ym, its first element can either represent category 1 or category 2 if without any constraint. Therefore, we introduce a proper prior distribution for the KL divergence in formula (1) to constrain the representation of label vector ¯ym (i.e. to make the first element always represent category 1 while the second element always represent category 2). A simple and reasonable choice is to use voting results

where rm is a vector of voting distribution from vm where its k-th element rm k = ∑Nn=1 vm nk ∑Kk=1 ∑Nn=1 vm nk . rm has fixed positions for categories that constrain the representation of label vectors and solves the problem of node exchangeability.

Some real-world datasets are sparse, where one object receives labels from only a few sources or one source only labels a few objects. In such cases, we introduce l1-norm for weight matrices in the classifier and the reconstructor

Indistinctive elements in weight matrices are pushed to zero to reduce noise from sources which label only a few objects.

Taking formulas (4), (5), and (6) into (1) and summing over all objects, LAA-B has the loss function

where {vm} denotes the set of all input vectors. λkl and λs are constraint strength. Note that we regard the KL divergence term as a regularizer, and giving it a small λkl achieves good performance in practice. Model parameters are learned by minimizing the loss. When the model is well trained, true label ˜ ym can be simply predicted from ¯ym by choosing the category with maximum probability: ˜ ym = argmaxk ¯ ym k .

3.5 Relationship withWeightedMajority Voting

We demonstrate the intuition of inferring labels of LAA-B from the view of extended weighted majority voting. That also explains what is learned in the weight matrix of the classifier. Here we change the notation of weight matrix wq a little bit. Let wn ij denote the weight from the i-th element in source block n of an input vector to the j-th element of a label vector, i, j ∈ {1, ...,K} and n ∈ {1, ...,N}. Weights corresponding to source block n constitute a weight block.

We write out the expression of label vector ¯ym with the interaction between n-th blocks in the input vector and the weight matrix (the number of categories is set as 2)

The expression extends weighted majority voting [Li et al., 2014] which only assigns one weight for each source. A weight block for the corresponding source has K2 weights. wn ij is the weight from source labeled category i to inferred category j. A positively larger weight assigns more contribution from the corresponding source label to the inferred category. Therefore a weight block describes labeling credibility of the corresponding source. Note that a weight block with large diagonal weights represents a credible source which usually gives correct labels.

3.6 Object Ambiguity

Based on LAA-B, we can introduce more factors to further improve the inference performance. Here we introduce object ambiguity. An ambiguous object usually contains conflicting or little labeling information, that may produce large noise for learning. By contrast, an unambiguous object has clean and sufficient labeling information. One may have an easy understanding of object ambiguity by referring to object difficulty [Bachrach et al., 2012].

A model is expected to put more efforts to correctly label and reconstruct an unambiguous object than an ambiguous one. To achieve this goal, we introduce a scalar zm for each object om and combine it into the classifier and the reconstructor respectively. A large zm indicates an object is unambiguous while a small zm indicates an object is ambiguous.

We can see a larger zm results in more heterogeneous distribution after the softmax operator. Heterogeneous distribution leads to large loss if the object is not well reconstructed, that forces the model to improve the reconstruction quality. Since the input vector vm contains all object information, zm can be modeled based on it:

where wo is the network weight which is learned in the training process. τ is the softplus activation function to ensure that zm is positive. This model is called LAA-O (LAA with Object ambiguity).

3.7 Latent Aspects

Further extension for LAA-O is to introduce latent aspects. One object may have more than one latent aspects, such as colors and shapes of flowers. One source may be good at classifying flowers by shapes, but not that good by colors. The performance of a model can be improved by distinguishing source credibilities under different aspects.

Suppose one object om has I latent aspects which is denoted by an I-dimensional latent aspect vector zm. Its element zm i indicates the weight of the i-th latent aspect. For the i-th latent aspect, there are corresponding weight matrices wiq and wip for the classifier and the reconstructor respectively to represent source credibility under that aspect. Label vector ¯ym and reconstructed vector ˜vm are obtained by summing over all aspects.

The latent aspect vector zm is produced from vm.

The model is named LAA-L (LAA with Latent aspects). Figure 3 gives its architecture. We can see LAA-O is a special case of LAA-L where the number of latent aspects is 1.

4 Experiments

Three real-world datasets are used in experiments. Bluebirds [Welinder et al., 2010] consists of 108 bluebird pictures. There are 2 breeds among all the images, and each image is labeled by all 39 sources. Flowers [Tian and Zhu, 2015b] contains 200 flower pictures. Each source is asked whether the flower is a peach flower. 36 sources participate in the labeling task and contribute 2,366 binary labels in total. Web Search [Zhou et al., 2012] contains 2,665 query-URL pairs. 177 sources are asked to rate each pair by 5 relativity levels. In total 15,567 labels are collected.

Inference accuracy is used as the measurement

We implement proposed models by TensorFlow 1 which offers GPU acceleration . Gradient descent is exploited to minimize the loss. A dataset is split into training set and validation set. Training process stops when the loss on the validation set begins to increase. We grid-search proper hyperparameters by choosing the combination which achieves the lowest loss on the validation set. Hyperparameters include learning rate η ∈ [0.001, 0.1], constraint strength λkl ∈ [0.0001, 0.1], and λs ∈ [0.0001, 0.1]. After determining the optimal hyperparameters, we train the model by using all data with chosen hyperparameters. For LAA-L, we set the number of latent aspects as 2 (further discussion is in Section 4.4). In this paper, we implement networks with one layer for the classifier and the reconstructor respectively. Though deep networks can be easily exploited, we find they do not further improve inference accuracy on the datasets due to data size.

4.1 Accuracy Comparison

Representative label aggregation methods are used as baselines. They are MV (majority voting), CATD (a weighted majority voting model which estimates the confidence interval of source credibility [Li et al., 2014]), TruthFinder (the first trust propagation model [Yin et al., 2008]), DARE (a generative model which models source credibility and object difficulty [Bachrach et al., 2012]), DS (the first label aggregation model [Dawid and Skene, 1979]), BCC (a generative model using confusion matrix [Kim and Ghahramani, 2012]), CrowdSVM (a recent proposed method combining max margin majority voting and DS Model [Tian and Zhu, 2015a]). Proposed label-aware autoencoders LAA-B, LAAO, and LAA-L are compared as well. The results of inference accuracy are illustrated in Table 1. We can see even the basic model LAA-B is competitive with the state-of-the-art methods. Note that LAA-B does not use knowledge about sources or objects. By introducing object ambiguity, LAA-O improves the inference accuracy significantly. LAA-L further improves the inference accuracy by exploiting latent aspects. The results show that proposed LAA has advantages on label aggregation compared with other methods.

1 www.tensorflow.org

4.2 Source Credibility inWeight Matrix

Weight matrix of the classifier represents source credibility. To illustrate that, we take weight matrix wp after training LAA-B on the Flowers dataset. There are 36 blocks corresponding to 36 sources and each block has 2×2 weights. We use 20 gray levels to color weights according to their values. White indicates a large weight while black indicates a small weight. Figure 4b illustrates the weight blocks. For each block, its diagonal weights are relatively large. That means the inference accuracy of most sources are better than random guessing. LAA-B distinguishes sources with high credibility from others by giving large diagonal weights. To see that, we illustrate in Figure 4a labeling accuracy for sources which correspond to the first block column. Labeling accuracy of a source is: the ratio of correctly labeled object number to the total labeled object number by the source. We can observe that sources with high labeling accuracy correspond to blocks with large diagonal weights. The observation shows networks in LAA have the capability to capture source credibility by learning the weight matrix.

4.3 Effect of Object Ambiguity

In this subsection we show the effect of object ambiguity. After training LAA-O on the Web Search dataset, we sort objects by their ambiguity in ascending order and divide them into 10 buckets. The first bucket contains objects with the least ambiguity while the last bucket contains objects with the most ambiguity. For objects in each bucket, we calculate: 1. Average entropy of their inferred label vectors; 2. Average inference accuracy. The results are illustrated in Figure 5. Objects are given heterogeneous distribution of inferred labels (with small entropy) if the model treats them as unambiguous (blue bars). Those unambiguous objects usually lead to high inference accuracy (yellow bars). On the other hand, low accuracy is caused by objects with conflicting labeling information. LAA-O treats them as ambiguous and gives relatively balanced distribution of inferred labels (with large entropy) to reduce their effect in the learning process. That decreases the noise from those objects and improves the overall accuracy. To visualize object ambiguity, we show four images of peach flowers from the Flowers dataset and arrange them from unambiguous to ambiguous in Figure 6. The unambiguous peach flower is easy to recognize while the ambiguous one is not.

4.4 Effect of Latent Aspects

LAA-L is the extension for LAA-O by introducing more than one latent aspects. From Table 1, we observe that accuracy is significantly improved by LAA-L on the Bluebirds and the Flowers dataset, but not improved on the Web Search dataset. In Figure 7, we compare latent aspect vectors learned on Flowers and Web Search datasets to explain the reason. A latent aspect vector with 2 dimensionality can be illustrated as First aspect Second aspect a point in a 2-D figure. On the Web Search dataset, two latent aspects show strong positive correlation (Figure 7b). Therefore they can be merged into one aspect without decreasing the inference accuracy. On the Flowers dataset, however, latent aspects have slight negative correlation (Figure 7a) which means different objects have different dominant aspects. That supports the necessity of introducing latent aspects. We show two typical flower images for each aspect in Figure 8. Images dominated by the first aspect are pink flowers, while images dominated by the second aspect are white flowers with special petal shape. Figure 7 also shows a method to determine a proper number of latent aspects. We can try to add one latent aspect at a time, and train LAA-L to see whether there is positive correlation between aspects. If two aspects do not have positive correlation, then the added aspect is effective. If two aspects have strong positive correlation, then adding the extra aspect is not necessary. Through experiments, we find that the best numbers of latent aspects for Bluebirds, Flowers, and Web Search datasets are 2, 2, and 1 respectively.

Figure 8: Illustration of two typical flower images for each aspect. Images dominated by the first aspect (corresponding to red diamonds in 7a) are pink flowers, while images dominated by the second aspect (corresponding to black triangles in 7a) are white flowers with special petal shape.

5 Conclusion

In this paper, we propose Label-Aware Autoencoders (LAA) for aggregating crowd wisdoms. We exploit networks to implement the classifier and the reconstructor in LAA which have the potential to automatically learn underlying source labeling patterns. Object ambiguity and latent aspects are introduced to the basic model to further improve inference accuracy. Experiments on three real-world datasets show the advantages of proposed framework, where the basic model LAA-B is competitive with the state-of-the-art, LAA-O and LAA-L further improve inference accuracy significantly.