Label Enhancement for Label Distribution Learning via Prior Knowledge

1 Introduction

Multi-label learning (MLL) [Grigorios and Ioannis, 2006] can deal with the ambiguous case where one instance belongs to more than one classes [Zhou and Zhang, 2006; Zhang and Zhou, 2014; Gibaja and Eva, 2015], but it does not fit many real-world learning problems well where the importance of relevant labels is often different. Label distribution learning (LDL) is proposed to cover a certain number of labels, representing the degree to which each label describes the instance [Geng, 2016]. During the past decade, LDL techniques have been widely employed to learn age estimation [Xin et al., 2013; Gao et al., 2018], video parsing [Ling and Geng, 2017], emotion classification [Yang et al., 2017b; Zhou et al., 2016; Yang et al., 2017a], facial landmark detection [Su and Geng, 2019], etc. However, most of the existing datasets only contain single logical labels or multilabels rather than label distributions. It is difficult to obtain the label distribution directly. A popular method to obtain the label distribution is the vote strategy [Yang et al., 2017b;

∗Corresponding Author

Liu et al., 2017]. The vote strategy is not only costly, but also easily influenced by the annotators' individual subjective opinions, age, sex, occupation, social background, etc. This fact encourages researchers to explore new methods to recover label distributions.

Label enhancement (LE) [Xu et al., 2018] is the way to recover the label distributions from the logical labels in the training set leveraging the topological information in the feature space and the correlation among the labels. Some label enhancement approaches, like fuzzy clustering [Gayar et al., 2006], label propagation [Li et al., 2016] or manifold learning [Hou et al., 2016] are proposed to recover label distributions. But the pioneer works ignore the importance of the prior knowledge, which is widely existing in many applications. The prior knowledge indicates some natural properties of the label distribution. For example, the people usually predict another person's age in a way such as "around 25 years old", which indicates using not only 25, but also the neighboring age to describe the appearance of the face. Among the ages, 26 and 24 are closer to 25, 27 and 23 are farther away 25. The farther away from the chronological age, the less the importance it is. In addition, psychology theories, like Plutchik's wheel [Plutchik, 2001] and Mikels' wheel [Mikels et al., 2005] can also be used as the prior knowledge. We propose a reinforcement learning based method for label enhancement (RLLE) via the prior knowledge. Reinforcement learning is much more focused on goal-directed learning [Sutton and Barto, 2018]. Given a goal, an agent can automatically learn a policy to reach it. Coincidently, properties implied in the prior knowledge can be defined as the goal. And the target label distribution may not be obtained in one step. It is a more natural way to obtain the label distribution by sequential adjustment. Therefore, in this paper, label enhancement is formulated as a dynamic decision process that a set of actions conducted on the label distribution to adjust the description degrees automatically until it reaches the terminal state, which is defined by the prior knowledge. RLLE can automatically learn a policy to recover the label distribution from the single logical label directly based only on the prior knowledge. No additional information is needed anymore.

The proposed model defines instance as the environment. The features of an instance and the current label distribution are defined as the state. The action space includes {bigger, smaller, constant}. The multi-dueling Q-learning algorithm is designed to output several actions at every step. The model starts by analyzing the initial label distribution whose description degrees are all zero and then sequentially to adjust the label distribution by the actions. Figure 1 illustrates several steps of the dynamic decision process to generate a label distribution. To evaluate the proposed method, we conduct a comprehensive experimental evaluation on both facial age estimation and image emotion classification datasets, obtaining competitive results in terms of MAE and accuracy. Our contributions are summarized as follows.

• We model the label enhancement as a dynamic decision process and propose a reinforcement learning based algorithm for label enhancement (RLLE). To the best of our knowledge, this paper is the first to apply reinforcement learning to label enhancement.
• We address the label enhancement for label distribution learning problem by using the prior knowledge and design the multi-dueling Q-learning algorithm to train the RLLE network.
• We conduct comprehensive experiments on two applications. The results show that our proposed approach not only outperforms the other state-of-the-art methods, but also achieves superior performance over the vote strategy.

2 RelatedWork

Label distribution learning. Label distribution learning is successfully applied to ambiguous tasks. According to the methodology in [Geng, 2016], the nature way to label an instance x is to assign a real number dyx , to each possible label y, representing the degree to which y describes x. Without of generality, assume that dyx ∈ [0, 1]. Further suppose that the label set is complete, i.e. using all the labels in the set can always fully describe the instance. Then, ∑y dyx = 1. Such dyx is called the description degree of y to x. The learning process on the instances labeled by label distributions is therefore called label distribution learning (LDL). Problem transfer (PT), algorithm adaption (AA), and specialized algorithms (SA) strategies are contained in [Geng, 2016]. LDL is successfully applied to age estimation [Xin et al., 2013; Gao et al., 2018], crowd counting [Zhang et al., 2015], etc. LDSVR [Geng and Hou, 2015] is proposed to predict the crowd opinions about a movie. [Yi and Wu, 2019] proposes an end-to-end framework called PENCIL, which can update both network parameters and label estimations as label distribution to solve the problem of the noisy labels. DLDL [Bin-Bin et al., 2017] is proposed to learn the label distribution using deep neural network for tasks with discrete and continuous labels. [Yang et al., 2017b] develops binary conditional probability neural network (BCPNN) and augmented conditional probability neural network (ACPNN) for predicting sentiment distributions. [Yang et al., 2017a] demonstrates a multi-task deep framework by jointly optimizing classification and label distribution prediction. As shown above, LDL applications are very broad. However, due to the difficulty of obtaining the label distributions directly, many training sets only contain simple logical labels rather than label distributions.

Label enhancement. Label enhancement (LE) [Xu et al., 2018] is defined as recovering the label distributions from the logical labels in the training set via leveraging the topological information of the feature space and the correlation among the labels. The logical label vector of instance xi is denoted by li = (ly1 xi , ly2 xi , ..., lyc xi )T , where c is the number of possible labels. The description degree of y to x is denoted by dyx , and the label distribution of xi is denoted by di = (dy1 xi , dy2 xi , ..., dyc xi )T . Given a training set, S = {(xi, li)|l ≤ i ≤ n}, where li is the logical labels, LE recovers the label distribution di of xi from the logical labels, and thus transforms S into a LDL train set ε = {(xi, di)|l ≤ i ≤ n}. Fuzzy clustering (FCM) [Gayar et al., 2006] is one of the label enhancement algorithms, which attempts to cluster feature vectors by iteratively minimizing an objective function. [Li et al., 2016] recovers the label distributions from logical labels by using iterative label propagation. The topological structure of the feature space is considered for label enhancement in [Hou et al., 2016], which is based on manifold learning. [Jiang et al., 2006] demonstrates that a fuzzy support vector machine can effectively reduce the effects of outliers when solving the classification problem and uses the kernel function to calculate the center and radius. Generating the distribution with a univariate Gaussian function is widely used in multiple applications [Xin et al., 2013; Gao et al., 2018; Yang et al., 2017a]. More recently, the Graph Laplacian Label Enhancement (GLLE) [Xu et al., 2018] is proposed to generate label distributions. However, the related works ignore recovering the label distribution by using the prior knowledge, which always implies the important properties of the label distribution. In this paper, we consider using prior knowledge combining reinforcement learning algorithm to solve the label enhancement problem.

3 Methodology

We define the properties implied in prior knowledge as the target state. The agent can learn the policy to sequentially adjust the description degrees of the label distribution automatically to the target state.

3.1 The Definition of Prior Knowledge

The prior knowledge always implies important properties of the label distribution. Figure 2 shows two examples of prior knowledge for facial age and image emotion respectively. In the age estimation application, due to the appearance similarity of the neighboring ages, both the chronological age 25 and the neighboring ages 24 and 26 can be used to describe the appearance of a 25-year-old face [Xin et al., 2013]. This is consistent with the real-life experience that people usually predict another person's facial age in a way such as "around 25 years old". The description importance of neighboring ages is lower than the chronological age. Therefore, as shown in Figure 2(a), the prior knowledge for facial age estimation satisfies the following two properties: 1) The description degree of α is the highest in the label distribution, which ensures the leading position of the chronological age; 2) the description degree of other age decreases with the increase of the distance away from α, which makes the age closer to the chronological age contribute more to the class description. In the adjustment process, if the generated label distribution meets these two properties, it means that the terminal state has been reached.

Psychological theories are widely adopted [Zhou et al., 2016; Yang et al., 2017a]. The distance among emotions can be defined by counting the number of steps from one emotion to another according to Mikels' wheel [Mikels et al., 2005]. In this paper, we use the label from the dominant votes as the single emotion label ydominant label. The description degree of the single emotion label is the highest. dyi x > dyj x if the distance between the i-th emotion and the dominant votes label is smaller than the distance between the j-th emotion and the dominant votes label. Inspired by [Zhou et al., 2016], we reproduce a wheel of eight emotions' relationship, which is shown in Figure 2(b). In the reproduced emotion wheel, emotions sat at the opposite end have an opposite relationship, while emotions next to each other are more closely related. The correlation between the two emotions comes from psychological research [Mikels et al., 2005]. We quantify the relations between each pair of emotions based on the angle between them in the wheel. For example, emotion pairs with 45 degrees have a positive relationship, while emotion pairs with 90 degrees have no relationship. Emotions pairs with 180 degrees and 135 degrees have negative relationship. To reduce the influence of the negative relationship, it would like to enforce the description degree dyi x = 0, if the angle between the label yi and the dominant votes label ydominant label more than 90 degrees.

From the above analysis, the prior knowledge can be defined as the properties that the descriptive degree of the logical label is the highest, the farther the other labels are from the logical label, the lower the descriptive degree. The prior knowledge can be applied to any applications that meet these two properties. It is not limited to facial age estimation and image emotion classification.

3.2 Label Enhancement as a Dynamic Decision Process

We cast the problem of label enhancement as a dynamic decision process since this setting provides a formal framework to model an agent that makes a sequence of decisions to adjust the single logical labels into label distributions. The properties implied in the prior knowledge can be formulated as the terminal state. A reinforcement learning agent can learn a policy to reach the terminal state. Formally, the dynamic decision process has a set of actions A, a set of states S, and a reward function R. This section presents details of these three components.

Actions. The set of actions A is composed of three actions that can be applied to the description degrees of the label distribution respectively. These actions are illustrated in Figure 1, action to make bigger, smaller, constant. In this way, the agent has three degrees of freedom to adjust the distribution during any interaction with the environment. A label distribution is defined as dx = {dy1 x , dy2 x , ..., dyc x }, where c is the number of the possible labels. Any of the adjustment actions making a discrete change to the description degree by a factor relative to its current value in the following way:

In early exploration experiments, we noticed that smaller values make the agent slower to reach the target distribution. We set δ = 1 in our experiments. Since this value gives a good trade-off between speed and adjustment accuracy. According to the definition of label distribution learning, ∑y dyx = 1. Every adjustment step is followed by a normalization operation on the label distribution to satisfy the sum of description degrees is 1, i.e

We do not set a trigger action like [Zhao et al., 2018], because we found that the trigger action introduced extra errors. Unlike previous works, only one action is mapped from the state. In our work, a set of actions is selected at every step. The number of selected actions equals to the number of labels.

State. The state st is defined as a tuple (f, dt), where f ∈ R is the features of an instance, and dt ∈ Rc is the label distribution at time step t. The feature vector f is extracted from a pre-processing stage. We will illustrate the details in Section 4.1 and 4.2. The label distribution dt is defined as the result after the last adjustment step. c is the number of labels. So dt is a c-dimension vector. In our earlier experiments, inspired by [Zhao et al., 2018], the history actions were included in the state space. But the results were poor, so we removed the history actions from the state space. After the decision of the action at in the state st, the next state st+1 is obtained by the state transition: st+1 = (f, dt+1). When the agent reaches the terminal state, which is defined by the prior knowledge, we finish the distribution adjustment process and obtain the target label distribution.

Reward. The reward function r is proportional to the improvement that the agent makes to adjust the label distribution after selecting a particular set of actions. The reward function is defined as r(st) since the agent obtains the reward by the state s regardless of the action a. Similar to [Silver et al., 2016], at the terminal step T , that is, the agent reaches the terminal state, r(sT ) is assigned by 3. We give a larger award to encourage RLLE to produce the label distribution with quick convergence, which was shown effective in experiments.

3.3 Training withMulti-dueling Q-learning

The input of RLLE is the features of the instance and the logical label. The output is the required label distribution dx = {dy1 x , dy2 x , ..., dyc x }. A set of actions is needed to recover the label distribution. We propose multi-dueling Q-learning named as MQ-learning that takes as input the state representation and gives as output the value of a set of actions based on dueling Q-learning [Wang et al., 2015]. The multi-dueling Q-learning outputs c actions. And every action is selected by argmaxa∈AQ∗ (s, a). So c dueling Q functions are included in MQ-learning. Every dueling Q function represents two separate estimators: one for the state value function and one for the state dependent action advantage function, while sharing a common feature learning module. The multi-dueling Q-learning structure is combined by the all dueling Q functions jointly.

where c indicates the number of labels. s represents the state, a denotes the action, and θ, φ, β mean the parameters in the feature learning module, advantage function and value function respectively.

Our network has three fully connected layers (fc1, fc2, fc3) combined with the ReLU layers, fc1 and fc2 have 64 nodes, fc3 has 32 nodes, as shown in Figure 3. In our task, the network outputs c actions at every step, where c equals to the number of labels. And every branch has two separate estimators. One for the value function V (s; θ, β) and the other for the advantage function A(s, a; θ, φ). The c actions will be applied to the description degrees respectively. The update for the network weights at the i-th iteration θi given transition sample (s, a, r, s, ) is as follows:

where a, represents the actions that can be taken at state s, , α is the learning rate and γ is the discount factor.

4 Experiments

We evaluate our method on two applications, e.g. age estimation and image emotion classification, with four datasets. For both models, the learning rate is 0.001, the batch size is 64, the discount factor γ is 0.9. And the size of the prioritized replay is 5000. We use the ϵ-greedy method to select the action for exploration. For every instance, the number of episodes is 5. Though label enhancement is different from the standard supervised learning. The setting of episodes can make the agent reach the terminal state with a quick speed.

4.1 Age Estimation

Implementation Details. Two datasets are used in this application. The first one is the FG-NET Aging dataset [Lanitis et al., 2002]. There are 1,002 face images from 82 subjects in this dataset. Each image is labeled by its chronological age. The ages are distributed in a wide range from 0 to 69. The second dataset is the much larger MORPH dataset [Ricanek and Tesafaye, 2006]. There are 55,132 face images from more than 13,000 subjects in this dataset. The ages of the face images range from 16 to 77.

An agent selects an action according to the state representation. As is illustrated in Section 3.2, the state st is defined as a tuple (f, dt), where f ∈ R is the features of the instance. For fair comparison, following [Xin et al., 2013], the feature extractor used for the FG-NET dataset is the appearance model [Edwards et al., 1998]. The first 200 model parameters are used as the extracted features. The features used for the MORPH dataset are the BIF [Guo et al., 2009]. The dimensionality of the BIF vector is further reduced to 200 using marginal Fisher analysis [Yan et al., 2005]. In our earlier experiment, we try to output actions for all ages at every step, but we find that the actions farther from the chronological age are noise. Therefore, 5 actions are output left and right respectively, centered on the chronological age. The features vector concatenates with the label distribution, resulting in a state vector s ∈ R211 in the FG-NET and MORPH datasets. We compare the proposed method RLLE against the Gaussian and Triangle functions used in [Xin et al., 2013; Gao et al., 2018]. The standard deviation of Gaussian function and the bottom length of Triangle function are recommended by [Xin et al., 2013]. We use the label distributions generated by RLLE as the ground-truth to train the age estimation model with three label distribution learning algorithms, e.g. IIS, BFGS, and CPNN [Xin et al., 2013]. According to [Xin et al., 2013; Gao et al., 2018], mean absolute error (MAE) is used to evaluate the performance.

Results. Table 1 shows the MAE of the all compared algorithms. The standard deviations on the MORPH dataset are also given in the table. We randomly select 80% for training and the remaining 20% for testing. The results are reported by running this process 10 times. As can be seen, the overall performance of the RLLE method is better than that of the Single, Gaussian and Triangle functions [Geng, 2016; Xin et al., 2013; Gao et al., 2018]. Specially, RLLE performs remarkably better than the baseline with the BFGS algorithm on the MORPH dataset, which reaches 3.94±0.07. There are mainly two reasons for the good performance of RLLE. First, since Gaussian and Triangle functions are just simple distribution hypothesis. And there are many restrictions by using Gaussian and Triangle functions. The prior knowledge always implies natural properties. The using of the prior knowledge makes it possible that reduces the restrictions and learns more natural label distributions. Second, given the target properties, an agent of reinforcement learning can learn to achieve the implied properties automatically. Sequential sets of actions applied to adjust the description degrees of the label distribution by multi-dueling Q-learning is a more natural way for label enhancement. The experimental results demonstrate that the label distributions generated by RLLE are closer to the natural truth label distributions.

4.2 Image Emotion Classification

Implementation Details. We execute our experiments on two image emotion distribution datasets, Flickr LDL and Twitter LDL [Yang et al., 2017b], which have multiple annotations. Flickr LDL and Twitter LDL contain 11,150 and 10,045 images respectively, whose labels fall in the typical eight emotions space [Mikels et al., 2005] (i.e. amusement, anger, contentment, awe, disgust, excitement, fear and sadness). 11 viewers are hired to label the Flickr LDL dataset with the eight commonly used emotions and 8 viewers are hired to label Twitter LDL dataset. The votes from the workers are integrated to generate the label distribution for each image. We use the label from dominant votes as the single emotion label to generate label distributions. We compare the label distributions generated from our method with those from vote strategy and the other label enhancement methods to evaluate which result is more suitable for image emotion classification task.

As suggested by [Yang et al., 2017b], deep features are extracted with VGGNET [Simonyan and Zisserman, 2014]. For each image, we extract the last fully connected layer output as the emotion representation and reduce it to 280 dimensions using principle component analysis (PCA). Because the label space contains eight labels, so the state vectors in Flickr LDL and Twitter LDL result in s ∈ R288 . Figure 2 shows the relationship among emotions, emotion pairs with 45 degrees have a positive relationship, while emotion pairs with 90 degrees have no relationship. Emotions pairs with 135 degrees and 180 degrees have a negative relationship. To reduce the influence of the negative relationship, we enforce the description degrees of the label distribution generated by RLLE be zero if the angle between one emotion and the dominant votes emotion is more than 45 degrees. We use the processed results as the ground-truth to train the image emotion classification model via nine label distribution learning approaches, including PT-bayes, PT-SVM, AA-KNN, AA-BP, IIS, BFGS, CPNN [Xin et al., 2013], BCPNN, and ACPNN [Yang et al., 2017b]. What needs to be pointed out in particular is that BCPNN and ACPNN are specially designed for image emotion classification with Flickr LDL and Twitter LDL datasets. Besides, we compare the proposed method against the state-of-the-art label enhancement approaches used in [Yang et al., 2017a]. [Yang et al., 2017a] considers two constraints to generate the label distributions. The first one is using the Gaussian function to assign the description degrees of all the emotion labels based on the distance to the dominant votes label according to Mikels' wheel. The description degrees of the label distribution can be written as follows:

where µ represents the label from the dominant votes, and the σconf denotes the level of influence of each emotion label determined by the confidence in the label annotations. And the fixed parameters ε and C ensure that the sum of description degrees is normalized to 1. The second constraint assumes that each affective image only evokes sentiment with the same valence in the label distribution, either positive or negative. So the description degrees of the label distribution are changed to:

where Yµ denotes all the emotion labels of the same valence, either positive or negative. There exists label ambiguous cases among the emotions. Most works adopt vote strategy to get label distributions [Yang et al., 2017b; Geng and Hou, 2015]. Vote strategy is always considered as the most objective method to solve the uncertainty of the "ground-truth label" problem. We also compare our method against the results from the vote strategy.

Results. The performance of ourmethod and the other three methods is shown in Table 2. For Flickr LDL and Twitter LDL datasets, the accuracy is examined for evaluating the performance of the results. For testing, the accuracy is measured by the label with the maximum description degree in the label distribution. As can been seen, for Fliker LDL dataset, RLLE ranks 1st in 66.6% cases and 2nd in 22.2% cases across all the evaluation accuracy measures. And vote strategy ranks 1st in 33.3% cases and 2nd in 44.4% cases. The other two Gaussian constraints have lower ranks. The average rank of RLLE is 1.44, which achieves superior performance over the vote strategy and the other label enhancement methods. For Twitter LDL dataset, RLLE ranks 1st in 66.6% cases and 2nd in 33.3% cases. And the vote strategy ranks 1st in 33.3% and 2nd in 44.4% cases. For the two image emotion distribution datasets, the average rank of our method is both 1st. We can conclude that our proposed method shows superiority in accuracy across all the two Gaussian constraints and the vote strategy. The experimental results present that using psychology theories as the prior knowledge is more suitable for generating the label distributions than the vote strategy and the other methods. There are mainly two reasons for the good performance of RLLE. First, using prior knowledge to recover the label distributions can highlight the essential properties, which is implied in the prior knowledge. Second, similar to the age estimation application, given the target properties, sequential decision making process to adjust the description degrees of the label distribution is a more suitable way for label enhancement.

5 Conclusion

This paper proposes the RLLE approach for label enhancement via the prior knowledge. We cast the problem of label enhancement as a dynamic decision process and design a multi-dueling Q-learning algorithm to train the proposed model. Given the target state defined by the prior knowledge, the agent can conduct a set of actions on the description degrees to adjust the initial label distribution into the target one. We conduct experiments on two scenarios. Comparing with several state-of-the-art methods, our proposed approach achieves the best performance. In the future, we will explore more prior knowledge to achieve better performance for different applications.

Acknowledgements

This research was supported by the National Key Research and Development Plan of China (No. 2017YFB1002801), the National Science Foundation of China (61622203), the Jiangsu Natural Science Funds for Distinguished Young Scholar (BK20140022), the National Science Foundation of China (61702095), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Collaborative Innovation Center of Wireless Communications Technology.