Hashtag Recommendation for Multimodal Microblog Using Co-Attention Network

1 Introduction

In recent years, microblogging, like Twitter and Sina Weibo, has become one of the most popular services for information generation and diffusion, as well as social interaction among the various social media outlets. According to the quarterly report released by Twitter 1 , there are 317 million monthly active users. The users write texts with limited number of characters to record life or express their emotion. Therefore, microblogs have been widely used as sources for public opinion analysis [Bermingham and Smeaton, 2010], prediction [Bollen et al., 2011], and many other applications [Sakaki et al., 2010; Becker et al., 2010; Guy et al., 2013]. Microblogs contain a form of metadata tag (hashtag), which is a string of characters prefixed with the symbol (#). Hashtags are used to mark keywords or topics within a microblog and have proven to be useful for many applications, including microblog retrieval [Efron, 2010], query expansion [Bandyopadhyay et al., 2012], and sentiment analysis [Wang et al., 2011].

1 Twitter Q3 report 2016: https://investor.twitterinc.com/results.cfm

However, despite their proven success, relatively few microblogs include hashtags labeled by their users. Hence, the task of automatically recommending hashtags has become an important research topic and has received considerable attention in recent years. Various models have been adopted for this task using kinds of features [Ohkura et al., 2006; Heymann et al., 2008], collaborative filtering [Kywe et al., 2012], generative models [Krestel et al., 2009; Ding et al., 2013] and deep neural networks [Gong and Zhang, 2016]. Although some research has been done on this topic, most of the studies have focused only on the use of textual information. However, according to the data retrieved from Twitter, we observe that more than 30% of tweets contain not only text but also images. Hence, it is not easy to correctly recommend hashtags through approaches designed to use only textual information. Figure 1 illustrates a multimodal microblog with the hashtag #dog. There is no information about dog in this tweet. With only textual information, we may extract the hashtag #birthday. However, the hashtag #dog is hardly to be identified.

To address this issue, we present a multimodal model to combine textual and visual information together. Some previous works simply combine the image feature vector and the text feature vector [Antol et al., 2015]. However, the correct hashtags are often only related to a small part of images or texts. Hence, using a global vector to represent the image or text may lead to suboptimal results due to the noises made by the irrelevant part of images or texts. Motivated by the work [Yang et al., 2015; Ma et al., 2015] on image QA task and [Vinyals et al., 2015] on image captioning, we introduce an attention mechanism to conduct the task of hashtag recommendation. Attention mechanism allows the model to focus on specific parts of the input. Since image captioning and image QA are mainly focused on image processing, these works mentioned above only intend to model image features by taking image attention into consideration. However, textual information is an essential part of hashtag recommendation tasks for multimodal microblogs. In addition to the textual information, the image also can guide feature extraction of the text. In this work, we introduce a co-attention network that takes the mutual influence of both text and image into consideration.

To demonstrate the effectiveness of our model, we carry out experiments on a large data set collected from Twitter. Experimental results illustrate that the proposed method can achieve better performance than state-of-the-art methods using textual information only. The main contributions of our work can be summarized as follows.

• We introduce an integrated framework of visual and textual information for hashtag recommendation tasks.
• We propose a co-attention network that incorporates tweet-guided visual attention and image-guided textual attention.
• Experimental results using a dataset collected from Twitter demonstrate that the proposed method can achieve significantly better performance than current state-of-the-art methods.

2 RelatedWorks

2.1 Hashtag Recommendation

Because of the usefulness of hashtag recommendation, many methods have been proposed from different perspectives. [Krestel et al., 2009] introduced Latent Dirichlet Allocation to elicit a shared topical structure from the collaborative tagging efforts of multiple users for recommending hashtags. Based on the observation that similar webpages tend to have the same tags, [Lu et al., 2009] proposed a method taking both tag information and page content into account to achieve the task. [Ding et al., 2013] proposed the use of a translation process to model this task. They extended this translation-based method and introduced a topic-specific translation model to process various meanings of words in different topics. In [Tariq et al., 2013], discriminativeterm-weights were used to establish topic-term relationships, of which users perception were learned to suggest suitable hashtags for users. [Gong and Zhang, 2016] adopted CNNs with an attention mechanism to perform this task. They added an extra channel to take trigger words into consideration.

The task of hashtag suggestion/recommendation for images is also related to this work and has been studied from various aspects. Most of these works give much attention to the tags annotated by users through social media services such as Flickr, Zooomr, and so on. The tags annotated in these services are like labels that are added to a photo to make it easier to find later. Previous works mainly focused on recommending tags that are good descriptors of the photo itself, whereas hashtags are usually referred to more abstract concepts. [Sigurbj¨ornsson and Van Zwol, 2008] studied the tag recommendation task for images. Their approach was based on the statistics of Flickr annotation patterns and tag co-occurrence statics. When a user submits a photo and enters some tags, an ordered list of candidate tags is derived for each of those entered tags. Hence, it cannot be directly applied to images only. In [Garg and Weber, 2008], the problem of personalized, interactive tag recommendation was also studied based on the statistics of the tags co-occurrence. In [Li and Snoek, 2013], ensembles of Support Vector Machines per tag was used to classify tag relevance.

From the brief descriptions given above, we can observe that most of the previous works focused on either textual information or visual features. In this work, the proposed method incorporates both textual and visual information.

2.2 Multimodal Model

There is a large quantity of literature on multimodal models. Early works usually focused on modeling the relation between an image and text. Recently, the association between an image and text has been studied for automatic image captioning and image QA tasks. [Chen and Zitnick, 2014] used a recurrent visual memory to aid in both sentence generation and visual feature reconstruction. [Vinyals et al., 2015] first extracted high-level image features and then fed them into an LSTM to generate captions. [Antol et al., 2015] used an LSTM to encode a question and then combined the question with an image using elementwise multiplication. [Yang et al., 2015] used an attention mechanism to query an image multiple times to infer the answer progressively. However, as previously mentioned in the Introduction section, these works did not consider the guiding significance of an image on textual feature extraction.

Attention mechanism allows models to focus on specific parts of a visual or textual input and has been successfully used in various multimodal models. In this work, we adopt the mechanism to select important information from the input tweets and images.

3 The ProposedModels

Given a tweet with corresponding images, our task is to automatically generate proper hashtags for the tweet. To conduct this task on multimodal tweets, we formulate the task as a multi-class classification problem. The overall architecture of the model is shown in Figure 2. The input to our network is an image and a tweet comprised of a variable-length sequence of words. The output is a vector, and each dimension of the vector represents the probability of a hashtag. To ease understanding, we describe our model in three parts. The feature extraction is described in Sec 3.1. The co-attention network and hashtag prediction are described in Sec 3.2 and Sec 3.3, respectively.

3.1 Feature Extraction

Image feature extraction

We use a pretrained 16-layer VGGNet [Simonyan and Zisserman, 2014] to extract an image feature map. We first rescale images to 224×224. Unlike previous works that use a global vector as the image feature, we want spatial features of different regions which contain more information of the orginal image. We divide an image into an N × N grid, and then use VGGNet to extract a D-dimensional feature vector for each region of grids. Therefore, an image could be represented as vI = {vi|vi ∈ RD , i = 1, 2, · · · ,m}, where m = N ×N is the number of regions in the image, which is equaled 49 in our case and vi is a 512-dimensional feature vector corresponding to each region i. For convenience of calculation, we use a single layer perceptron to convert each image vector into a new vector that has the same dimension as the tweet feature vector.

Text feature extraction

Each word of a given tweet t is first represented as a onehot vector in the size of the vocabulary. Then, each onehot vector is embedded into a real-valued word vector xi distributed in a continuous space. We sum up the embedded vectors to obtain a sentence-level tweet representation: t = x1, x2, · · · , xT , where T is the maximum length of the tweets. In our work, sentences with length less than T are padded with zeros.

Because the LSTM has shown good performance in understanding text and has been widely used in recent years, we employ it to generate text feature. The process of text feature extraction is shown in Figure 3. At each time step, the LSTM unit takes an input vector (word embedding vector in our case) xt and outputs a hidden state ht, using input gate it, memory cell ct, forget gate ft, and output gate ot. The details are illustrated as follows:

We construct the tweet feature matrix vT by combining the ht states at every time step: vT = {hi|hi ∈ Rd , i = 1, 2, · · · , T}, where d is the LSTM's dimension and T is the maximum length of a tweet. In particular, the semantic information have used in Sec 3.2 is obtained by averaging the LSTM states ht over all of the time steps .

3.2 Co-Attention Network

Based on the notations provided in Sec 3.1, we have the image feature matrix vI and the text feature matrix vT . Because tweets and images are not equally important in the hashtag recommendation task, we propose a co-attention network that generates tweet attention and image attention sequentially. Because text is the main part of the hashtag recommendation task for tweets, we first attend to images based on tweets.

Tweet-guided visual attention

In most cases, a hashtag is only related to a specific region of the input image. For example, in Figure 1, there are many objects in the image: balloons, a dog, a blackboard, and grass, but the hashtag #dog is only related to the little dog in the image. Hence, instead of using a global vector as an image feature, we divide an image into grids and extract the feature vector of each grid to obtain a feature matrix vI. We then use a tweet-guided attention layer to filter out noises and find regions that are relevant to the corresponding hashtags.

As mentioned in Sec 3.1, we first use an average pooling operation to summarize the tweet features into a single vector have. Next, we propose an image attention based on the tweet summary have. We use a single-layer neural network to combine the tweet and image features and then use a softmax layer to generate an image attention distribution. The equation for these operations is as follows:

where vI ∈ Rd×m and vT ∈ Rd , m is the number of regions in an image, and d is the dimension of the representation. WvI , WvT , WpI are parameters, and WvI ,WvT ∈ Rk×d , WpI ∈ R1×k . Therefore, pI ∈ Rm, which corresponds to the attention probability of each region, is an m-dimensional vector. In addition, we use ⊙ to denote the combination of the image feature matrix and tweet feature vector, which is obtained by concating each column of the matrix by the vector.

Based on the attention probability pi of each image region i, the new representation of the image is constructed as the weighted sum of the image vector,

We then use the new image representation ˜ vI to guide the textual attention.

Image-guided textual attention

Compared to models that only use text to guide image attention, the co-attention network model constructs a more informative representation by considering the mutual influence of both text and image. The process of image guided textual attention is similar to visual attention. In order to obtain the textual attention distribution, we use the new representation ˜ vI of image to query the original text feature matrix vT . We then generate a new representation ˜ vT for text based on the probability distributions. The detail is illustrated as follows:

where vT ∈ Rd×T and ˜ vI ∈ Rd , T is the maximum length of tweets, and d is the dimension of the representation. W˜ vI , WT , WpT , bpT are parameters, and W˜ vI , WT ∈ Rk×d , WpT ∈ R1×k . Therefore, pT ∈ RT , which corresponds to the attention probability of each word of a tweet.

Stacked co-attention network

For more complex tweets, we can try to explore some subtle relationships among text and images by iteratively querying the original feature matrixes of images and texts using the newly generated representations. Formally, the formula can be summarized as follows: for the k-th (where k is greater than or equal to 2) co-attention layer, we respectively compute the distribution of visual and textual attention and generate a new representation for the input image and text based on the attention probability. The new query vector is formed by adding the new feature vector to the previous vector, the equation for image attention is as follows:

where q1 I is initialized to be ˜ vI. The query vector qkT for textual attention is as same.

3.3 Prediction

Finally, the hashtags are predicted using a multi-class classification. We adopted a single-layer softmax classifier with cross-entropy loss, and the input is a combination of the features generated from both attention operations. The final vector f = ˜ vT +˜ vI , and we achieve the scores of the hashtags for the dth tweet using:

where Wf , bf , θ are parameters, A is the set of candidate hashtags.

According to the scores output from the last softmax layer, we can get a ranked list of hashtags for each tweet and recommend the top-ranked hashtags to users.

3.4 Training

In our work, the training objective function is:

where hp is the hashtag for tweet tp, and S is the training set. The parameters in out model is:

where W are words embeddings; Mi and Mt are the parameters of the feature extraction of images and texts respectively; Miatt and Mtatt are the parameters of textual and visual attetion layers; the rest parameters belong to the fully connected layer.

The parameters were trained using stochastic gradient descent with the adam [Kingma and Ba, 2014] update rule. Dropout regularization [Srivastava et al., 2014] has proved to be an effective method and is used in our work.

4 Experiments

4.1 Dataset and Setup

We started by using Twitters API 2 to collect public tweets from randomly selected users. The collection contained 282.2 million microblogs published by 1.1 million users. From these microblogs, we extracted those that contained both images and hashtags. In this step, 2.27 million microblogs were extracted. Since some hashtags rarely occur, we filtered out the hashtags whose frequencies were very low in our corpus. Finally, the collection we constructed contained 402,782 tweets with corresponding images and hashtags of high frequency. The detailed statistics are shown in Table 1. The unique number of hashtags in the corpus was 3,292, and the average number of hashtags per tweet was 1.17. We split the dataset into a training set and a test set, with a ratio of 8:2, and randomly selected 10% of the training set as the development set.

For text words, we filtered out the stop words and lowfrequency words. The constructed word vocabulary contained 278,000 distinct words. For images, we downloaded images from the retrieved urls and rescaled them to 224×224. Then we fed them into a pre-trained VGG-16 network . The outputs of the last pooling layer of VGGnet were extracted as the image feature.

We used precision (P), recall (R), and the F1-score (F1) to evaluate the performance. The number of recommended hashtags for each tweet is denoted as k, where k = {1, 2, 3, 4, 5}, and the P, R, and F1 at the k result are denoted as Pk, Rk, and F1k, respectively.

4.2 Baseline

For comparison with the proposed model, we evaluated the following methods on the constructed corpus:

• Naive Bayes (NB): The hashtag recommendation task is formalized as a classification task. We applied NB to model the posterior probability of each hashtag given the textual and visual information of the microblogs.
• Support Vector Machine (SVM): We also followed the method proposed in [Chen et al., 2008], which used an SVM to solve the tag recommendation problem.
• TTM: TTM was proposed by [Ding et al., 2013] for hashtag recommendation using only textual information. The topical translation model was used to recommend hashtags.
• CNN-Attention: CNN-Attention was proposed by [Gong and Zhang, 2016] and is a convolutional neural network architecture that uses the attention

2 https://dev.twitter.com/

mechanism to incorporate trigger words, and it was the state-of- the-art method for this task.

• Image-Att: Image-Att is a variant of our proposed model, which only uses text information to generate visual attention distribution. Similar work was done by [Yang et al., 2015].
• Co-Att-2layer: This is also a variant of our proposed model. We applied a stacked two-layer co-attention network to model the images and tweets.

4.3 Results and Discussion

We evaluated the proposed method from the following perspectives: 1) comparing the proposed method to state-ofthe-art methods using a real world dataset and 2) identifying the impacts of the parameters.

Table 2 shows a comparison of the proposed method to the state-of-the-art discriminative and generative methods on the constructed evaluation collection. The three metric results listed were obtained when we recommended the top one hashtag for each tweet. Based on the results, we observed that the proposed method is better than the other methods. Discriminative methods achieved worse results than generative methods. The results of the proposed methods with and without visual information demonstrated that visual information can benefit both P and R.

Observing the comparisons of the "CNN-attention" and our proposed model, it is clear that images can significantly improve the performance of hashtag recommendation task. Since there are more and more users prefer to publish a tweet with corresponding images, finding an effective way to incorporate both textual and visual information posted is very meaningful.

For the lower performances, we believe there are several factors that led to this result, including the large number of labels. The dataset itself may be very confusing and difficult to distinguish, since all methods resulted in a low score in the evaluation dataset. In addition, the diversity of the image types and sources may impact methods using visual information. The VGG network we used for image feature extraction was pre-trained on an ILSVRC dataset, which includes images of 1,000 classes, most of which were outdoor scenes. However, the images crawled from Twitter were more irregular and some of them were graffiti, selfies, or even just screenshots of smart phones. Therefore, simply adopting a pre-trained weight may have caused deviations.

Figure 4 shows the P, R, and F1 of models with different numbers of recommended hashtags on the evaluation dataset. Each point of the curve represents the number of hashtags recommended ranging from 1 to 5. Obviously, P decreases and R increases as the number of recommended hashtags increases, and we obtained the highest F1 when recommending the top one hashtag. The curve that is the highest on the graph indicates the best performance. From the results, we can see that the performance of our proposed method is the highest of all the methods. This also indicates that the proposed method was significantly better than the state-of-the-art methods.

We also compare the precision, recall and F1-score of models with variants of our proposed method. Image-att only use text information to generate visual attention distribution. The results listed in Table 3 show that co-attention model achieve better results for the model that only use tweet-guided visual attention. The result of the model using a stacked twolayer co-attention work is slightly lower than the one layer coattention network. We believe the short length of tweets limit the extent of layers. Since increasing number of layers of the network will make the model more complex and require more training time, and the result is not significant improved, we do not try more co-attention layers model.

4.4 Parameter Influence

The result listed in Table 3 also shows the contribution of embedding dimension to the performance. Higher embedding dimension leads to a higher performance. We can see that when the dimension equals to 100, the result is very poor. Our proposed model performed well with a high embedding dimension. The performance improved significantly when the dimension changed from 100 to 300 and improve slightly from 300 to 500. The size of the embedding dimension represents the expression ability of each word, and a higher dimension can enhance the text feature expression ability.

5 Conclusion

In this paper, we propose and study a novel task, recommending hashtags for multimodal microblogs. We convert the hashtag suggestion task into a multi-way classification and introduce a co-attention network for this task. The proposed model combines textual and visual information. And we adopt attention mechanism to obtain more informative representation for both text and image. Since tweets and images are not equally important in hashtag recommendation task, we propose the co-attention network which generates textual attention and visual attention sequentially. We also constructed a large data collection retrieved from live microblog services to evaluate the effectiveness of our model. Experimental results showed that the proposed method is capable to achieve better performance than state-of-the-art methods using textual information only.

Acknowledgments

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No. 61532011, 61473092, and 61472088) and STCSM (No.16JC1420401).