Aggregating Crowd Wisdom with Side Information via a Clustering-based Label-aware Autoencoder

1 Introduction

Aggregating crowd wisdom aims at inferring true labels from multiple noisy labels collected from sources, which is also known as truth inference or label aggregation for crowdsourcing [Li et al., 2016]. As a key technology for massive object labeling, aggregating crowd wisdom is usually considered in the domain of unsupervised learning since labeling tasks do not provide ground-truth labels. A traditional aggregation algorithm takes multiple noisy labels from different sources as input and infers true labels for objects. A simple aggregation algorithm, majority voting, has been widely used in many voting scenarios, which infers the most voted label as the true label. More sophisticated algorithms usually exploit source credibility to model the relationship between true labels and labels from sources (source labels), where high credible sources are supposed more likely to provide correct labels [Zheng et al., 2017]. Besides source labels, recent work introduces side information as object features to achieve higher inference accuracy [Raykar et al., 2010;

Rodrigues and Pereira, 2018]. Those algorithms usually adopt the learning-from-crowds framework where aggregated labels from source labels guide the classification of object features. The framework is analogized to supervised learning where aggregated labels are analogized to training labels.

However, label noise is still a crucial problem in previous algorithms imitating supervised learning. Traditional supervised learning uses perfect training labels, but in label aggregation, labels from real-world sources are noisy and may lead to noisy aggregated labels. Using aggregated labels to directly guide the classifier may cause imprecise decision boundaries (Figure 1b). Side information such as object features is expected to alleviate label noise by bringing objects with similar features together. However, previous algorithms utilizing object features consider each object in isolation and do not provide explicit solutions to the label noise problem.

To alleviate label noise and take advantage of object features, we propose a clustering-based label-aware autoencoder (CLA). CLA utilizes clusters to gather objects with similar features, where objects in the same cluster are supposed to have similar true labels. A simple and intuitive mechanism is illustrated in Figure 1c. Specifically, CLA assigns each cluster a centroid and a label distribution, and exploits a deep generative process, to simultaneously generate latent embedding from the cluster centroid and an intermediate label from the label distribution. The latent embedding is further used to generate object features, while the intermediate label is used to generate source labels. The deep generative process is inspired by deep Gaussian mixture models in the framework of variational autoencoders (VAE) [Dilokthanakul et al., 2016; Jiang et al., 2016] and label-aware autoencoders [Yin et al., 2017]. CLA combines generating source labels and object features from clusters and adopts cluster labels to infer true labels. Since a cluster label contains labeling information of all objects in the cluster rather than a single object, CLA alleviates the problem of label noise. For model inference, CLA introduces constraints for model parameters by extending the framework of VAE [Kingma and Welling, 2013] and utilizing maximizing a posteriori (MAP) estimation. The MAP estimation can be written into a likelihood term and a prior term over model parameters. The likelihood term is inferred by constructing VAE. The prior term acts as a regularizer that prevents model parameters from overfitting and trivial solutions. Experiments on real-world tasks show that CLA signif-

2

2

3

3

𝑙௢భ = 𝑙௢మ = 𝑙௢య = 1

𝑙∆భ = 𝑙∆మ = 𝑙∆య = 2

(a)

2

2

3

3

𝑙௢భ = 𝑙௢మ = 1

𝑙∆భ = 𝑙∆మ = 𝑙∆య = 𝑙௢య = 2

(b)

icantly improves inference accuracy compared with the stateof-the-art aggregation algorithms.

2 RelatedWork

Traditional aggregation algorithms use source labels to infer true labels. Among them, weighted majority voting [Aydin et al., 2014], trust propagation [Yin et al., 2008], and generative models [Bachrach et al., 2012; Liu et al., 2012; Simpson et al., 2013; Venanzi et al., 2014; Tian and Zhu, 2015] are extensively researched. Label-aware autoencoders (LAA) build a bridge between label aggregation and neural networks which makes aggregation modeling more flexible [Yin et al., 2017]. Features of objects are utilized to further achieve higher aggregation accuracy. Most work exploits the framework named learning from crowds [Raykar et al., 2010]. Yan et al. further model source credibility affected by features based on the framework [Yan et al., 2014]. The learning-from-crowds framework originally uses a linear classifier to classify object features. Recent work utilizes deep neural networks to enhance the learning capacity of the classifier [Albarqouni et al., 2016; Dizaji and Huang, 2018; Rodrigues and Pereira, 2018]. Besides learning from crowds, a generative framework generates both source labels and object features from a true label of the object [Felt et al., 2014]. Among them, some algorithms design a specific model structure for specific data types (e.g., latent Dirichlet allocation for textual data [Rodrigues et al., 2017] and convolutional

1

1

1

1

neural networks for image data [Albarqouni et al., 2016; Cao et al., 2019]). Our model CLA is a general approach and does not assume a specific data type.

The other topic related to CLA is clustering, especially in the framework of VAE. VAE providing effective model inference for generative models, has advantages on unsupervised and semi-supervised learning tasks [Kingma and Welling, 2013; Kingma et al., 2014]. Label aggregation as an unsupervised learning task can be modeled by LAA, which is a variant of VAE. Deep clustering methods, especially deep Gaussian mixture models benefit from VAE by simplifying model inference [Dilokthanakul et al., 2016; Jiang et al., 2016]. Though label aggregation and clustering problems are respectively modeled in the framework of VAE and achieve attractive performance, to the best of our knowledge, there is no pioneering work combining both of them in a unified framework to utilize clustering to infer true labels.

3 Methods

3.1 ProblemDefinition

Given a label set {lmn}, m ∈ {1, ..., M} and n ∈ {1, ..., N}, where M is the number of objects and N is the number of sources. lmn denotes a label object m receives from source n. lmn ∈ {1, ...,K} is a categorical label, where K is the number of categories. For the convenience of representation, all labels belonging to object m are collected to form a label vector lm. lm has N source blocks and contains all labeling information of the object [Yin et al., 2017]. Traditional label aggregation aims to infer a true label ˜ tm for each object m given the label vector. In this paper, feature vector xm ∈ RI of each object is also utilized to make high-quality inference. I is the dimensionality of the feature vector.

3.2 Clustering-based Label-aware Autoencoder

Given label set {lm} and feature set {xm}, a clustering-based label-aware autoencoder (CLA) optimizes model parameters Θ via maximizing a posteriori (MAP) estimation.

The first term on the right hand is the log-likelihood, and the second term is the log-prior of model parameters. The equation assumes each object has a label vector and a feature vector independent with other objects given model parameters. We also use p(l,x) to denote p(lm,xm|Θ), for simplicity.

To model the log-likelihood, we introduce a deep generative process to generate label vectors and feature vectors from clusters (Figure 2a). Suppose there are C clusters. For an object, we first draw a cluster index z ∈ {1, ...,C}. Given cluster index z, we draw intermediate label y from the chosen cluster, and then draw label vector l from y. Meanwhile, we draw latent embedding h from the cluster, and then draw feature vector x from h. A formal description is given below.

For each object:

𝒍࣊࢝

௭ ௬࣊

𝐶

𝒍

ݕ

࢞ሻ 𝒍, 𝒉, ,ݕ ,ݖ𝑝ሺ

ݖ

௭࣊

𝑀

(a) Generative process (Decoder)

1. Draw cluster index z with

• πz represents the probability of each cluster being chosen in the categorical distribution. Subscript z in πz is only a name indicating the distribution is used to generate variable z.
• Draw intermediate label y given cluster index z with

where y ∈ {1, ...,K}. The superscript z in π zy indicates each cluster has its own distribution π zy , z ∈ {1, ..., C}.

1. Draw label vector l from y with

Since label vector l is a combination of N source blocks, we draw a label for each source block from a categorical distribution independently. Each source n has its own distribution parameter π nl (y), which is calculated by using y as input, via a neural network with weight matrix w nπl . To make a brief representation, we concatenate all N weight matrices into a weight matrix wπl to form a big network, which computes labels for all source blocks and generates label vector l from the intermediate label y.

1. Draw latent embedding h from cluster z with

࢞

𝒉

࢞࣋࢝

௭

𝝁௛

ሺ𝝈 𝟐 ሻ௛

௭

𝐶

h ∈ RJ where J is the dimensionality of the embedding feature space. Each cluster has a multivariate normal distribution with mean µzh and a simplified covariance matrix diag((σ 2 )zh ). σ 2 is a vector where each element measures the variance of the corresponding element in h.

1. Draw feature vector x from h with

if each element in x is Bernoulli distributed. Ber(ρx(h)) is a multivariate Bernoulli distribution. Vector ρx(h) ∈ [0, 1]I is calculated via a multi-layer neural network with weights wρx . An alternative is a multivariate normal distribution if x is normally distributed.

By introducing the generative process, we have the joint distribution for each object

The log-likelihood in Eqn. (1) is then deduced into an ELBO and solved by constructing VAE [Kingma and Welling, 2013]

where p(z, y,h, l,x) can be regarded as the decoder, and q(z, y,h|l,x) can be regarded as the encoder in CLA.

With the mean-field method, q(z, y,h|l,x) is decoupled

y is assumed to be conditionally independent with x given l. Similar assumption applies for h. In the equation, q(z|l,x) is directly computed (see Eqn. (15)).

q(y|l) is a categorical distribution

where probability ˜π(l) is computed via a neural network with weight matrixw˜π, given l as input.

q(h|x) is a multivariate normal distribution

where ˜µ(x) and ˜σ 2 (x) are modeled by two multi-layer neural networks with weights w˜µ and w˜σ respectively.

By introducing the decoder and the encoder, the framework of CLA is illustrated in Figure 2, with model parameters

According to Eqn. (1) and (8), CLA is optimized by maximizing the ELBO and the log-prior term.

After model optimization, cluster labels are utilized to infer a true label for object m. Cluster labels contain labeling information of all objects belonging to the cluster and are effective to alleviate the problem of label noise.

An inferred true label is then chosen by the category which achieves the maximum probability. We use posterior probability q(z|l,x) instead of p(z|l,x) since p(z|l,x) is not directly modeled in CLA.

3.3 Model Inference: ELBO

We write out the ELBO by using Eqn. (7) and (9).

ELBO(l , x)

All terms in the equation can be derived by directly using distribution definitions (we do not show detailed solutions), except for q(z|l,x). Here we solve q(z|l,x) via

where v|k denotes the k-th element in the vector. The integral over q(h|x) is approximated by sampling ht from distribution q(h|x) T times. p(z|y = k,ht ) is a calculable probability by exploiting the generative process

where p(y = k|z = c), p(ht |z = c), and p(z = c) can be directly computed from corresponding distributions (3), (5), and (2), respectively. Eqn. (16) is an analogy to the 'Estep' in Expectation-Maximization methods which estimates the distribution of latent variables by using current model parameters. We calculate probability p(z|y,h) rather than directly modeling q(z|y,h), since there is no simple distribution describing q(z|y,h), to the best of our knowledge.

3.4 Model Inference: Regularizer

The log-prior term in Eqn. (1) plays an assistant but important role: prevent parameters from overfitting and set prior distributions to avoid trivial solutions. We use the term as a regularizer and assign coefficient λ to control the strength.

For neural network weights, namely, wπl , wρx , w˜π, w˜µ, and w˜σ, a prior is normal distribution N(0, 1) for each element in weights to prevent weights from overfitting.

For parameters related with clusters, namely πz, {π zy }, {µzh }, and {(σ 2 )zh }, following prior distributions are used.

Dir(αm) assigns each object a prior Dirichlet distribution. Dir(β z ) assigns each cluster a prior Dirichlet distribution. Mean µzh of each cluster is assigned a multivariate normal distribution. The variance vector σ 2 has each element generated from an Inverse-Gamma distribution. We set a = b = 2 to make E[σ 2 |j ] = 1, j ∈ {1, ..., J}. We find these prior distributions work well in practice, while a shared distribution (no regularizer or a uniform regularizer) has a force to push clusters together which leads to fewer clusters than expected and inferior results. The coefficient λ of the regularizer is set as the reciprocal of the size of model parameters. This is a balance between ELBO and regularizer, which prevents the optimizing process from favoring the regularizer term even if a model has a million parameters.

We exploit pre-training to find appropriate prior distributions. Specifically, we pre-train q(h|x) by constructing a mirror structure to form an autoencoder, which learns embedding h in the middle layer by reconstructing x. Then we concatenate h and majority voting label ymv from l and apply the K-means clustering on the concatenated vectors to obtain initial C clusters. The cluster index for each object is encoded into a smoothed one-hot vector to avoid zeros and used as αm in Eqn. (17). The centroid of each cluster is used as µz in Eqn. (19). Source labels in each cluster are collected by majority voting, which forms vector β z in Eqn. (18).

4 Experiments

4.1 Implementation Details

CLA has only a few hyperparameters and is easy to train. wρx has a fully connected multi-layer structure with node numbers {40, 100, I/2, I} from the bottom up. I is the number in the output layer, while 40 is the dimensionality of embedding feature space. w˜µ, w˜σ have similar structures with VAE and we use node numbers {I, I/2, 100, 40} from the bottom up. Sampling time T = 5. Cluster number C is discussed in the subsequent section. We implement CLA by TensorFlow and use gradient ascent for training 1 . Hyperparameter search adopts a similar manner with LAA by splitting a dataset into a training set and a validation set [Yin et al., 2017]. Here the learning rate is 0.001. Training is stable and usually achieves desirable inference accuracy after 1,500 epochs.

1 Demo code is at https://github.com/coverdark/cla demo

Table 1: Overall Inference Accuracy Comparison

parison. Since there is randomness in the clustering of CLA, we run the algorithm 20 times and report the average. Results are illustrated in Table 1.

From Table 1, TF, DS, IBCC, DARE, and LAA have only a slight or no advantage compared with MV, which indicates the difficulty of modeling source credibility from sparse and conflicting source labels. On the other hand, algorithms utilizing object features usually achieve higher performance. In these algorithms, DLC and CrowdDeepAE are better than LC on some tasks, which indicates the advantage of exploiting deep neural networks. However, on the other tasks, DLC or CrowdDeepAE do not show advantages because of label noise. Deep neural networks enhance learning capability but are easier to be trapped in imprecise decision boundaries in unsupervised learning. ML has a good performance on CUB200-2010 tasks, but fails on the document task, due to the difficulty of modeling source credibility from noisy features. Among the compared methods, CLA achieves the highest accuracy. The result is significant by conducting a t-test between CLA and the other algorithm achieving the highest accuracy. CLA uses clusters to alleviate label noise and is more effective than the state-of-the-art algorithms. 2

4.4 Effect of Cluster Number

A key factor making CLA effective is that CLA usually exploits more clusters than label categories. Since CLA assumes objects in the same cluster have similar true labels, the assumption is satisfied when the number of clusters is sufficient. Though ideally, the deep generative process can perfectly cluster objects into corresponding label categories, the ideal case generally does not happen on real-world data. More clusters make it easier to form small clusters that contain similar objects in a relatively short distance in the embedding feature space. In Figure 3, we illustrate the inference

2 Here we remind the limit of CLA when object features are too noisy to form effective clusters. We conduct experiments on a sentiment polarity dataset [Rodrigues et al., 2013], but CLA (Acc. 0.9078) and other algorithms utilizing object features cannot outperform IBCC (0.9150).

4.2 Datasets

Reuters contains a document categorization task. 1,786 documents from 8 categories receive labels from online users. 38 users contribute labels giving an average of 3 answers per document [Rodrigues et al., 2017]. We apply latent Dirichlet allocation to bag-of-words feature vectors to obtain 200 topics as object features [Blei et al., 2003].

CUB-200-2010 dataset contains tasks to label local characteristics for 6,033 bird images [Welinder et al., 2010]. We use 7 binary labeling tasks, namely bill (bill shape is all-purpose or not), head (head pattern is plain or not), shape (shape is perching-like or not), forehead (forehead is black or not), throat (throat is black or not), underpart (underpart is yellow or not), and breast (breast pattern is solid or not). For each task, there are about 500 users contributing labels and each image receives 5 labels. We collect ground truth from whatbird.com for evaluation. For an image, we use 287 local attributes collected from online users as object features.

4.3 Overall Inference Accuracy

We compare inference accuracy among representative aggregation algorithms. Overall inference accuracy is the ratio of the number of correct inferred objects to the overall number of objects. Compared algorithms are: MV (majority voting), TF (TruthFinder [Yin et al., 2008]), DS (Dawid & Skene's model [Dawid and Skene, 1979]), IBCC (a bayesian version of the DS model [Kim and Ghahramani, 2012]), DARE (a generative model with source credibility and object difficulty [Bachrach et al., 2012]), LAA (label-aware autoendcoders [Yin et al., 2017]), MomResp (a model generating both object features and source labels from true labels [Felt et al., 2014]), LC (learning from crowds [Raykar et al., 2010]), DLC (deep learning from crowds [Rodrigues and Pereira, 2018], ML (a multiple labelers method [Yan et al., 2014]), CrowdDeepAE ([Dizaji and Huang, 2018]), CLA (proposed model, a clustering-based label-aware autoencoder). A Kcoin model [Raykar et al., 2010] (K is the number of label categories) is exploited to model source credibility for MomResp, LC, DLC, ML, CrowdDeepAE, and CLA for fair comaccuracy along with different numbers of clusters. The accuracy increases when the number of clusters increases, and achieves the highest when the number ranges from 100 to 400. Then the accuracy slightly goes down when the number of clusters is very large. When the number is too small, each cluster contains too many objects even they are not very similar. That breaks the assumption of CLA and results in inaccurate cluster labels. When the number of clusters is too large, a cluster contains too few objects, which is insufficient to infer a reliable cluster label.

Interestingly, the CUB-200-2010 dataset contains images from 200 real-world bird categories and clusters from CLA correspond to these underlying bird categories. In Figure 4, we show the clustering result on the bill task when cluster number C = 200. For a clear illustration, we use tSNE [Maaten and Hinton, 2008] to map embedding h into 2-dimension and plot 20 bird categories with different colors. This figure intuitively shows the learned embedding and clusters, where bird categories roughly accord with learned clusters. Images of the same bird category usually share the same characteristics, that supports the effectiveness of CLA.

4.5 Effect of Regularizer

To show the effect of the regularizer in CLA, we compare results with no regularizer, with a uniform regularizer, and with an appropriate regularizer as described in the Regularizer section. The uniform regularizer uses a uniform distribution in Eqn. (17), (18), and a normal distribution N(0,I) in Eqn. (19). We experiment on the forehead task, the other tasks show similar trends. In Figure 5, we illustrate the number of objects in each cluster with cluster number C = 200. Clusters are sorted by the number of objects in descending order.

A model collapses to a few clusters with no regularizer, or with a uniform regularizer. The collapse leads to a similar inferior result of insufficient cluster numbers, as shown in Figure 3. The collapse does not happen with a CLA regularizer, and the model can give full play to the advantages of CLA.

4.6 Effect of Embedding Size

CLA is robust with different settings of embedding size. We set dimensionality of the embedding feature space ranging from 10 to 100 for CLA and illustrate the corresponding results in Figure 6. The accuracy does not show a significant change with different embedding sizes.

5 Conclusion

This paper proposes a clustering-based label-aware autoencoder (CLA) to utilize object features and alleviate label noise for crowd wisdom aggregation. CLA exploits a deep generative process to generate both source labels and object features from clusters. CLA extends the framework of VAE and utilizes MAP estimation for model inference. Experimental results on real-world tasks show that CLA significantly improves inference accuracy compared with the state-of-the-art aggregation algorithms. Besides high performance and robustness, experiments show the intuition of learned embedding and clusters to support the effectiveness of CLA.

Acknowledgments

The corresponding authors Yong Yu and Weinan Zhang thank the support of NSFC (61702327, 61772333).