Joint Learning on Relevant User Attributes in Micro-blog

1 Introduction

Social media, such as Twitter and Facebook, enables the users to post messages and share information in social networks, producing an unprecedented amount of user generated content (UGC) with rich facts about the users, including their personal attributes. Since then, UGC has been applied to various user attribute classification tasks, which recognizes user attributes, such as gender [Wang et al., 2015a; Zhu et al., 2015], age [Marquardt et al., 2014] and profession [Tu et al., 2015]. During the last few years, user attribute classification has drawn more and more attention due to its great potential influence to various applications, such as personality analysis, intelligent marketing and online advertising [O'Connor et al., 2010; Volkova et al., 2013; Preotiuc-Pietro et al., 2015].

However, previous studies mainly focus on single attribute classification involving only one attribute, which ignores the relationship among various user attributes. Intuitively, the relationship among various user attributes may benefit different attribute classification tasks and should be considered. For Aux-LSTM, which first learns an auxiliary representation from the auxiliary task with an auxiliary Long Short-Term Memory (LSTM) layer and then integrate the auxiliary representation into the main task for joint learning.

∗Corresponding author

2 RelatedWork

In the last decade, many researches have devoted their efforts on user attribute classification (e.g., profession, gender and age classification) in several research communities, such as natural language processing and social network analysis. Related studies differ primarily in focusing on different styles of texts and extracting different types of features for one user attribute.

For the gender classification task, Schler et al., [2006] exploit the differences in writing style and content between male and female bloggers to determine an unknown author's gender on the basis of a blog vocabulary. Mohammad and Yang [2013] show that there are marked differences across genders in how they use emotion words in work-place email. Ciot et al., [2013] conduct the first assessment of latent attribute inference in various languages beyond English, focusing on gender inference of Twitter users. Li et al., [2015] aim to identify the genders of two interactive users on the basis of micro-blog text. Some other studies, such as [Mukherjee and Liu, 2010], [Peersman et al., 2011] and [Gianfortoni et al., 2011], focus on exploring more effective features to improve the performance.

For the age classification task, most studies are devoted to explore efficient features in blog and social media. Schler et al., [2006] focus on textual features extracted from the blog text, such as word context features and POS stylistic features. Peersman et al., [2011] apply a text categorization approach to age classification with textual features extracted from the text in social media. More recently, Marquardt et al., [2014] propose a multi-label classification approach to predict both the gender and age of authors from texts adopting some sentiment and emotion features.

For the profession classification task, there is less related studies. Tu et al., [2015] extract the features from the verification descriptions in the users' personal information to achieve a significant performance improvement due to the fact that all the users in their collected data set are verified by officials of Sina Wiebo. However, most of users in social media are unverified and without any verification descriptions about them.

Different from all above studies, we focus on the classification task involving two or more user attributes. To the best of our knowledge, this is the first attempt to use the classification task of one user attribute to help the classification task of another user attribute.

3 Data Collection

We collect our data set from Tencent Micro-blog 1 , which is one of the most popular SNS websites in China. From this website, we crawl each user's homepage containing user information (e.g. name, profession, age, gender) and the posted messages. The data collection process starts from some randomly selected users, and then crawl the data of their followers and friends. To get a more reliable data, we remove these users if they publish less than 50 messages. In total, we collect the user information and messages of 25000 users as our data set.

1 http://t.qq.com

Table 1 shows the number of male and female users who have public gender information in the data set collected by us. We select these users as the data set for the gender classification task.

Table 2 shows the number of the users, whose profession information is available, in top-seven available profession categories accounting for the most proportions. We select the users from these seven profession categories as the data set for the profession classification task. We select the users from these seven profession categories as the data set for the profession classification task.

Figure 2 shows the age distribution of the users whose age attribute information can be obtained from their homepages in our data set. From this figure, we can see that most users are young and their ages distribute in the range of 18-28 years old. To enforce the task difficulty, we select the users whose ages range from 18 to 28 as the data set for the age classification task, totally 11 age categories.

4 Joint Learning on Relevant Attributes via Aux-LSTM

In this section, we describe our Aux-LSTM approach to joint learning on two user attributes in detail. The first subsection introduces the basic LSTM network. The second subsection introduces the LSTM model for user classification. Finally, we present the joint learning approach to user classification.

4.1 Basic LSTMNetwork

Long short-term memory network (LSTM) is proposed by [Hochreiter and Schmidhuber, 1997] to deal with gradient explosion or disappearance. LSTM maintains a separate memory cell inside it that updates and exposes its content only when deemed necessary. In order to map the input sequence of main task to a fixed-sized vector, we adopt the standard LSTM layer used by [Graves, 2013], which consists of four components formalized through Equation (1) - (6), i.e., an input gate it, an output gate ot, a forget gate ft, and a memory cell ct:

Where σ denotes a sigmoid function, wt is the current input at time step t and ⊗ denotes the elementwise multiplication. The candidate memory cell gt is computed by Equations (4). ct computed by Equations (5) defines the calculation in each memory cell at each time step t , and the output ht computed by Equations (6) is the hidden state of LSTM units.

4.2 LSTMModel for User Classification

LSTM takes word embeddings as input. Each word in a text is represented by a real-valued row vector. Given a text with n words T = {w1, w2...wn}, all the word vectors are stacked in a word embedding matrix L ∈ Rd×|V | , where d is the dimension of word embeddings, V is the vocabulary. Then, we feed L into the LSTM layer. Through the LSTM layer, the input of word embedding matrix is converted into a new representation h, i.e.,

After calculating the hidden vector of each step, we feed the representation h, which obtained by averaging the outputs of LSTM over all T time steps, into a fully-connected layer to obtain the representation h′ as follows:

Where φ is the "ReLU" activation function and dense(·) denotes the output of the fully-connected layer.

Subsequently, a dropout layer is used to obtain less interdependent network units and achieve better performance. The computing function is given as follows:

Where D denotes the dropout operation and p∗ denotes the dropout probability.

Finally, in user classification task, we feed h∗ to a softmax layer to get the prediction probabilities p, i.e.,

Where w is the weight vector to be learned and b is the bias term.

4.3 Joint Learning for User Classification

Figure 3 delineates the overall architecture of Aux-LSTM model which contains a main LSTM layer and an auxiliary LSTM layer. In our study, we consider one user attribute classification task as the main task and another user attribute classification classification task as the auxiliary task. The goal of the approach is to employ the auxiliary representation to assist the performance of the main task. The main idea of our Aux-LSTM approach lies in that the auxiliary LSTM layer is shared by both the main and auxiliary tasks so as to leverage the learning knowledge from two user attribute classification tasks.

(1) The Main Task:

Formally, the representation of the main task is generated from both the main LSTM layer and the auxiliary LSTM layer respectively:

Where hmain1 means the representation for the classification model via the main LSTM layer, while hmain2 means the representation for the classification model via the auxiliary LSTM layer. Note that, the auxiliary LSTM layer is a shared LSTM layer bridging the two user attribute classification tasks.

Subsequently, we feed hmain2 into a fully-connected layer followed by a dropout layer to get the auxiliary representation Raux, i.e.,Raux = dense(hmain2) · D(p∗ ) (13)

We can obtain a novel representation after concatenating above two representation hmain1 and Raux and use them as the input of a fully-connected layer followed by a dropout layer in the main task:

Where h∗main denotes the output of the dropout layer in the main task and ⊕ denotes the concatenate operator.

(2) The Auxiliary Task:

The representation of the auxiliary task is generated from the auxiliary LSTM layer:

Subsequently, a fully-connected layer followed by a dropout layer is utilized to obtain a feature vector for classification:

Where h∗aux denotes the output of the dropout layer in the auxiliary task.

(3) Joint Learning:

Once we obtain the representation of the main task and the auxiliary task, i.e., h∗main and h∗aux , we feed them into the softmax layers to predict the probability of label y given the inputs L as follows:

Where ̂ pθ(ymain |Lmain ) is the output of the main task and ̂ pθ(yaux |Laux ) is the output of the auxiliary task.

To learn the parameters of our Aux-LSTM model, we define our joint cost function as a weighted linear combination of the cost functions of both the main and auxiliary tasks, i.e.,

Where the ymain j and yaux j are the ground-truth label; N is the number of training samples; C is the category number and l is a L2 regularization to bias parameters.

In the above equation, λ is the weight parameter between the main task and auxiliary task. In our Aux-LSTM model, λ is set to be 0.75 in order to reduce the influence of noisy information from auxiliary task. Besides, the model parameters are optimized by using Adagrad [Duchi et al., 2011]. All the matrix and vector parameters are initialized with uniform distribution in [-√6/(r + c), √6/(r + c)], where r and c are the numbers of rows and columns in the matrices [Glorot and Bengio, 2010]. In order to avoid over-fitting, the dropout strategy is used in both the LSTM layer and auxiliary LSTM layer.

5 Experimentation

5.1 Experimental Settings

• Data Settings: The users are collected from Tencent Micro-blog, one of the most popular SNS websites in China. For each kind of user attribute (i.e., profession, gender and age) classification task, we randomly split the users into a training set (80% users) and a test set (20% users). We also set aside 10% users from the training as the validation data which is used to tune learning algorithm parameters.
• Word Embeddings: We use word2vec 2 (Skip-gram model is used) to pre-train word embeddings using our crawled data set containing 25000 users and word embedding matrices are not updated during model training. The dimensionality of word vector is set to be 200. The window size is set as 5.
• Hyper-parameters: The hyper-parameter values in the LSTMmodel are tuned according to performances in the development data.
• Evaluation Metrics and Significance test: The performance is evaluated using Macro-F1 (F ), which is calculates as F = 2PR P+R , where the overall precision P and recall R are averaged on the precision/recall scores from all categories. Furthermore, t-test is used to evaluate the significance of the performance difference between two approaches [Yang and Liu, 1999].

2 http://word2vec.googlecode.com/

5.2 Experimental Results on Individual Classification

For thorough comparison, we implement several kinds of classifiers for individual classification for each user attribute:

• SVM: SVM classifier with four types of textual features including word unigram and two kinds of complex features, i.e., F-measure, POS-pattern. These kinds of textual features yield the state-of-the-art performance for user classification proposed by [Mukherjee and Liu, 2010].
• ME: ME classifier with the same features settings as SVM. This is exactly the approach proposed by [Li et al., 2015].
• CNN: Basic bow-CNN 3 proposed in [Johnson and Zhang, 2015]. Each user is represented by a bag of features consisting of above four types of textual features.
• PCNN: CNN with two or more convolution layers in parallel so as to complement each other to improve model performance (i.e., the extension of bow-CNN), proposed in [Johnson and Zhang, 2015].
• LSTM: The standard LSTM model including a LSTM layer, a fully connected layer and a dropout layer. The representation model is Word Embeddings described in section 5.1.

Table 3 shows the results of five approaches to three user attribute classification tasks respectively. From the table, we can see that, SVM and ME yield similar results, while CNN and PCNN achieve better performances than SVM and ME. This result implies that deep learning approaches are more appropriate for the task of user attribute classification. Among the five approaches, LSTM performs best in all three tasks. The success might due to the fact that LSTMmodel can capture sequence information in the context and be more effective in learning representations with a flexible compositional structure [Wang et al., 2015b]. Therefore, it is a good choice to pick LSTM as the basic classification algorithm in our joint learning approach. Experimental results show that our LSTMmethod outperforms the other four approaches.

5.3 Experimental Results on Joint Learning

For thorough comparison, we implement three approaches to joint learning on two user attributes:

• Meta-Learning: This approach is a two-stage classification algorithm. In the first stage, we train a SVM classifier on the auxiliary task and obtain the posterior probabilities of the training samples in the main task. The

3 http://riejohnson.com/cnn download.html

posterior probabilities are prepared as meta-features for the classifier in the second stage. In the second stage, we train another SVM classifier on the main task with the extra meta-features obtained in the first stage. This approach is a straightforward strategy to use the information of the other user attribute.

• Multi-task LSTM: This baseline is inspired by the DNN-based multi-task learning framework with shared word representations proposed by [Collobert et al., 2011]. Two LSTM-based sequence embedding model are jointly trained with their specific word embeddings and a set of shared word embeddings. All the embeddings are initialized with pre-trained word2vec embeddings, and are dynamically updated during model training. Besides, unlike the input of LSTM, the specific word embeddings are pre-trained separately on the data set of each user attribute classification task, while the shared embeddings are pre-trained on the entire data set containing 25000 users.
• Aux-LSTM: This is our approach which performs joint learning on two user attribute classification tasks. The representation model is Word Embeddings.

(a) Main Task: Gender Classification

In this section, we report the classification results of different approaches where the main task is gender classification and the auxiliary task is age or profession classification. Table 4 shows the results of different approaches to gender classification where LSTM is an individual classification approach while the other approaches are all joint learning approaches.

From Table 4, we can see that:

(1): Joint learning on gender and age classification is not helpful, no matter what joint learning approaches are applied. The failure of the joint learning approaches is due to the fact that there might be not much correlation between user's age and gender information. Fortunately, our approach yields a performance loss by a very small margin (from 0.869 to 0.867).

(2): Joint learning on gender and profession classification is beneficial for performance improvement, no matter what joint learning approaches are applied. Specifically, our AuxLSTM (gender+prof ) approach achieves a 3.4% promotion on Macro-F1 (F) compared to individual LSTM approach. Our approach also performs better than Meta-Learning (gender+prof) and Multi-task LSTM (gender+prof). Significance test shows that the improvement of our approach over the other two joint learning approaches is significant (p - value < 0.05).

(b) Main Task: Age Classification

In this section, we report the classification results of different approaches where the main task is age classification and the auxiliary task is gender or profession classification. Table 5 shows the results of different approaches to age classification.

From Table 5, we can see that:

(1): Similar to the results in the previous experiments, joint learning on age and gender is also not helpful, no matter what joint learning approaches are applied. Fortunately, our approach still yields a performance loss by a very small margin (from 0.280 to 0.279).

(2): Joint learning on age and profession classification is beneficial for performance improvement, no matter what joint learning approaches are applied. Specifically, our Aux-LSTM (age+prof) approach achieves a 3.4% promotion on Macro-F1 (F) compared to individual LSTM approach. Our approach also performs better than MetaLearning (age+prof) and Multi-task LSTM (age+prof). Significance test shows that the improvement of our approach over the other two joint learning approaches is significant (p - value < 0.05).

(c) Main Task: Profession Classification

In this section, we report the classification results of different approaches where the main task is profession classification and the auxiliary task is age or gender classification. Table 6 shows the results of different approaches to profession classification.

From Table 6, we can see that:

(1): Joint learning on profession and age classification is beneficial for performance improvement, no matter what joint learning approaches are applied. Specifically, our Aux-LSTM (prof+age) approach achieves a 3.4% promotion on Macro-F1 (F) compared to individual LSTM approach. Our approach also performs better than MetaLearning (prof+age) and Multi-task LSTM (prof+age). Significance test shows that the improvement of our approach over the other two joint learning approaches is significant (p - value < 0.05).

(2): Joint learning on profession and gender classification is beneficial for performance improvement, no matter what joint learning approaches are applied. Specifically, our Aux-LSTM (prof+gender) approach achieves a 4.1% promotion on Macro-F1 (F) respectively compared to individual LSTM approach. Our approach still performs better than Meta-Learning (prof+gender) and Multi-task LSTM (prof+gender), which certifies the stronger capability of our Aux-LSTM model on leveraging sharing information across two relevant tasks. Significance test shows that the improvement of our approach over the other two joint learning approaches is significant (p - value < 0.05).

Overall, our experimental results show that the two user attributes of gender and age are not related and joint learning on them is not helpful. The user attribute profession is related to both age and gender, and joint learning on them is consistently effective for performance improvement.

6 Conclusion

In this paper, we propose a joint learning approach, namely Aux-LSTM, to perform the task of user attribute classification when relevant user attributes exist. Our approach well incorporates the relationship among the relevant user attributes. Specifically, we employ an auxiliary LSTM layer to learn the auxiliary representation for the main user attribute classification task. Experiments on three different user attribute classification tasks demonstrate that our proposed method significantly boosts the performance of the main task with the help of the auxiliary representation when the two user attributes are related.

In our future work, we would like to extend our proposed Aux-LSTMmodel to make it be capable of learning the auxiliary representation across three or more tasks. Moreover, we will make our efforts to incorporate more user attributes, e.g., locations, to perform joint learning.

Acknowledgments

This research work has been partially supported by three NSFC grants, No.61331011, No.61672366 and No.61375073.