Attentional Image Retweet Modeling via Multi-Faceted Ranking Network Learning

1 Introduction

Microblog services like Twitter have become important social platforms for users to share their media contents. Retweet function is usually considered to be key mechanism that enables users to repost someone else's tweets [Zhang et al., 2015b]. In social media sites, users who follows other users are termed as "followers" and users who are followed are termed as "followees". Central problem of retweet prediction is to model tweet sharing behavior that users repost tweets along followee-follower links so that more users are informed in SMS, which has attracted considerable attention recently in [Chen et al., 2016; Firdaus et al., 2016; Zhang et al., 2015b; 2016; Feng and Wang, 2013].

∗Corresponding author.

Existing approaches for retweet prediction [Firdaus et al., 2016; Zhang et al., 2015b; 2016; Feng and Wang, 2013; Zhang et al., 2015a] learn user preference model from their past retweeted textual tweets, and predict users' tweet sharing behavior in SMS. With the popularity of mobile devices, the amount of user-generated image tweets grows tremendously. For example, there are about 17.2% of tweets associated with images in Twitter [Chen et al., 2016]. So it is important to study problem of image retweet prediction in SMS. We give a simple example of image retweet prediction in Figure 1. As there is not discriminative feature representation for tweets with image [Chen et al., 2016] and SMS data is sparse [Firdaus et al., 2016] , existing proposed retweet prediction methods are ineffective to image retweet prediction problem.

Currently, most of existing retweet prediction methods [Firdaus et al., 2016; Zhang et al., 2015b; 2016; Feng and Wang, 2013; Zhang et al., 2015a] learn semantic representation of tweet based on hand-crafted feature (e.g., bagof-words). Recently, high-level visual features for image representation with pre-trained CNNs have shown success in various visual recognition tasks [Szegedy et al., 2013;

Zhao et al., 2018]. Since image tweets are always visual data, it is natural to employ deep convolutional neural networks [Simonyan and Zisserman, 2014] to learn visual representation of image tweets. On the other hand, image tweets are often associated with textual context information such as users' comments and captions [Chen et al., 2016]. Contextual image tweet information usually convey important messages and can gain better understanding of tweets. Since textual contextual information is always sequential data with variant length, we employ deep recurrent neural networks [Hochreiter and Schmidhuber, 1997] to learn its semantic representation. We employ multi-modal neural network learning method [Atrey et al., 2010] to learn joint image tweet representation from their multi-modal contents, which provides complementary information with different modalities.

Sparsity of SMS data is also a challenging issue for image retweet prediction. In SMS sites, network between image tweets and users is constructed through users' retweet relations on image tweets. Usually, each user only retweets a few image tweets and thus SMS network is sparse. Inspired by homophily hypothesis [Yuan et al., 2014], it is possible and reasonable to assume that collective information from users' followees and users' retweeted tweets can be jointly considered for tackling the sparsity problem of image retweet prediction. It is observed that social impact for retweet behavior varies between user and his/her different followees. We thus employ attention mechanism [Luong et al., 2015] to adaptively incorporate users' followee preference for jointly predicting targeted user's image retweet behavior.

In this paper, we study image retweet prediction problem from viewpoint of attentional multi-faceted ranking network learning. We first propose heterogeneous image retweet modeling (IRM) network that exploits multi-modal image tweets, users' retweet behaviors and their following relations for image retweet prediction. We introduce multi-modal neural networks with two sub-networks, where recurrent neural networks learn semantic representations of image tweets' contextual information, and convolutional neural networks learn visual representations. Multi-modal fusion layer is added to learn joint image tweet representation frommulti-modal neural networks. We develop attentional multi-faceted ranking method with introduced multi-modal neural networks, such that multi-faceted ranking metric is implicitly embedded in user preference representation for image retweet prediction. Main contributions of this paper are summarized as follows:

• Unlike previous studies, we present image retweet prediction problem from viewpoint of attentional multifaceted ranking network learning. We propose heterogeneous IRM network to model the problem, which exploits multi-modal image tweets, users' retweet behaviors and their following relations.
• We develop attentional multi-faceted ranking method with multi-modal neural networks to learn user preference representation based on retweeted tweets and following relations for image tweet prediction.
• We evaluate our method's performance using dataset collected from Twitter. Extensive experiments show that our method outperforms several state-of-the-art so-

lutions to the problem.

2 Image Retweet Prediction via Attentional Ranking Network Learning

2.1 The Problem

Before presenting the problem, we first introduce some basic notions and terminologies. Since image tweets are always visual data, it is natural to employ deep convolutional neural networks [Simonyan and Zisserman, 2014] to learn visual representation of image tweets. Given a set of image tweets I = {i1, i2, . . . , in}, we take convolutional neural networks' last hidden layer as visual representation of image tweets by X = {x1, x2, . . . , xn}. On the other hand, textual context information of image tweets such as users' comments and captions also gain better understanding of image tweets. We thus employ deep recurrent neural networks [Hochreiter and Schmidhuber, 1997] to learn its semantic representation. Given a set of textual contexts D = {d1, d2, . . . , dn}, we take recurrent neural networks' last hidden layer as semantic embedding of textual contexts by Y = {y1, y2, . . . , yn}. We denote the joint image tweet representations by Z = {z1, z2, . . . , zn}, where zi is joint representation of the i-th image tweet based on its visual representation xi and contextual semantic representation yi. We denote the set of ranking models for user preference representation by U = {u1, u2, . . . , um}, where uj is preference representation embedding of the j-th user.

Recently, existing approaches for retweet prediction [Firdaus et al., 2016; Zhang et al., 2015b; 2016; Feng and Wang, 2013; Zhang et al., 2015a] learn user preference model from their past retweeted textual tweets, and then predict users' tweet sharing behavior. Unlike previous studies, we propose attentional multi-faceted ranking metric heterogeneous IRM (i.e., image retweet modeling) network that exploits multimodal image tweets, users' past retweet behaviors and their following relations for image retweet prediction. We denote proposed heterogeneous IRM network by G = (V, E), where the set of nodes V is composed of the joint image tweet representations Z and user preference representations U, the set of edges E consists of users' past retweeted behaviors H and their following relations S. We denote the retweeted behaviors between image tweets and users by matrix H ∈ Rn×m, where the entry hij = 1 if the i-th image tweet is retweeted by the j-th user, otherwise, hij = 0. We then consider the following relations between users by matrix S ∈ Rm×m, where sij = 1 if the i-th user follows the j-th user. We next denote the set of the i-th user's followees by Ni (i.e., uj ∈ Ni if sij = 1), and the total set of users' followees by N = {N1,N2, . . . ,Nm}. We illustrate a simple example of the heterogeneous IRM network in Figure 2(a).

We then derive the heterogeneous triplet constraints from the IRM network as the users' relative preference for training the attentional multi-faceted ranking networks. We consider that the users express the explicit positive interest on the image tweets when he/she retweeted them in the IRM networks. On the other hand, following the existing Twitter analysis works [Chen et al., 2012], we consider that the users may show the implicit negative interest on the non-retweeted image tweets of their followees. This is because the nonretweeted image tweets by the followees are more likely to be seen but disliked by the user.

Given retweeted behavior between the i-th image tweet zi and the j-th user uj (i.e., hij = 1), we sample a nonretweeted image tweet of uj's followees as zk. Following popular homophily hypothesis [Yuan et al., 2014], we also incorporate users' followee preference for image tweet modeling. We then model users' relative preference by ordered tuple (j, i, k, Nj), meaning that "the j-th user prefers the i-th image tweet to the k-th one". Let T = {(j, i, k,Nj)} denote set of ordered tuples obtained from IRM network for a set of n image tweets and m users. We then consider ordered heterogeneous tuples as the constraints for learning user preference representations. More formally, we aim to learn the multifaceted ranking metric function for image retweet prediction. For any (j, i, k, Nj) ∈ T , the inequality holds:

where Fuj (·) = fuj (·)hNj (·) is the multi-faceted ranking model of the j-th user for image retweet prediction. The function fuj (·) is the personalized ranking model of the j-th user and hNj (·) models the social impact of the relative followee preference on the j-th user. We then define the personalized ranking function by fuj (zi) = uTj zi, where uj is the relative preference of the j-th user and zi is the joint representation of the i-th image tweet. We will present the details of the function hNj (·) in the next section.

Using the notations above, we define the problem of image retweet prediction from the viewpoint of attentional multifaceted ranking network learning as follows. Given the input image tweets I with their associated contexts D, the set of ordered tuples for users' relative preference T, and the heterogeneous IRM network G, our goal is to learn the multifaceted ranking metric representations for all user preferences U and the multimodal image tweet contents Z, and then rank the image tweets for the targeted users for image retweet prediction. The image tweets to user u are then ranked according to the multi-faceted user preference function Fu(·).

2.2 Attentional Multi-faceted Ranking Network Learning

In this section, we propose the attentional multi-faceted ranking network for image retweet prediction. We present the learning process in Figures 2(a), 2(b) and 2(c).

We first choose proper multi-modal neural networks for image tweet representation in IRM networks, which consists of two sub-networks: a deep convolutional neural network for visual representation of image data, and a deep recurrent neural network for semantic representation of textual contextual data. These two sub-networks interact with each other in a multi-modal fusion layer to form the joint representation, illustrated in Figures 2(b) and 2(c). For the visual representation of the image data, we use the activation of the 15-th layer of the proposed convolutional neural network VGGNet [Simonyan and Zisserman, 2014], which has been widely used in many visual representation tasks [Zhang et al., 2017; Zhao et al., 2017a; 2017b]. Meanwhile, we train the LSTM networks [Hochreiter and Schmidhuber, 1997] for the associated contexts of image tweet, and then take the output the last LSTM cell as its semantic representation. Considering the fact that the associated context of image tweets may be in the paragraph of several sentences with user comments and captions, we split them into sentences for learn the semantic representations by LSTM networks, and then fuse them by an additional max-pooling layer, shown in Figure 2(c).

In order to learn the joint representation of image tweets with different modalities, we set up the multi-modal fusion layer that connects the textual representation oriented from recurrent neural network part and visual representation oriented from convolutional neural network part, illustrated in Figure 2(c). We then map the activation of the two layers (i.e., the visual representation of image tweets and the semantic representation of textual contexts) into the same multi-modal feature fusion space and add them together to obtain the activation of the multi-modal fusion layer, given by

where + denotes the element-wise addition for the image tweet representation with different modalities. The vector zi is the joint representation of the i-th image tweet, which is obtained from the multi-modal fusion layer. The matrix W(i) and W(d) are weights for the fusion of multi-modal representations which can be learned in the training phrase of the attentional multi-faceted ranking network learning. The g(·) is the element-wise scaled hyperbolic tangent function, which forces the gradients into the most non-linear value range and leads to a faster training process, proposed in [LeCun et al., 2012].

We then present the attentional multi-faceted ranking function learning for image retweet prediction. Inspired by the attention mechanism [Luong et al., 2015; Zhao et al., 2017c], we design the social impact function hNj (·) based on the ordered tuple constraints T = {(j, i, k, Nj)} as follows. Given the user preference representations U = {u1, u2, . . . , um}, the social preference attention score for the p-th user and his/her q-th followee user in Np is given by

where W(s) and W(n) are parameter matrices to model the preference correlation between the user and his/her followee. The b is the bias vector and p is the parameter vector for computing the social preference attention score. For each followee uq in Np, its preference activation is given by αq = exp(spq) ∑q∈Np exp(spq) . Thus, the the social impact of the relative followee preference on the j-th user is given by hNj (zi) = ∑uq∈Nj αqfuq (zi).

Given the formulation of personalized ranking function fuj (·) and social impact function hNj (·), we now design the attentional multi-faceted ranking loss function as follows:

where the ranking function Fuj (zi) = fuj (zi)hNj (zi), the superscript F + uj (·) indicates the positive preference and F - uj (·) denotes the negative preference. We denote the hyperparameter c (0 < c < 1) controls the margin in the loss function.

We next introduce the details of our proposed attention multi-faceted ranking network learning. We denote all the model coefficients including neural network parameter, the joint image tweet representations and user preference representation by Ψ. Therefore, the objective function in our learning process is given by

where β is the trade-off parameter between the training loss and regularization term. To optimize the objective, we employ the stochastic gradient descent (SGD) with diagonal variant of AdaGrad [Kingma and Ba, 2014].

3 Experiments

3.1 Data Preparation

We collect data from Twitter, which is a popular microblog services for Web users to share their media contents [Java et al., 2007]. Users usually show their positive preference on image tweets by retweeting them in social media sites. We crawl profile of the users including their past retweeted image tweets and their following relations. In total, we collect 9,900 users, 7,193 image tweets and 29,501 following relations. We report that the average time that an image tweet retweeted by some collected users is 12.2, and the average number of image tweets that some collected user retweets is 9.1. Average number of followees among the collected users is 6.2, and maximum number of followees is 162. Average number of words in the context of image tweets is 9.1, and its standard variance is 5.4. For each retweet behavior (i.e., hij = 1) of the user, we sample two negative image tweets from his/her followees. We sort users' retweet behaviors based on their timestamp and use the first 60%, 70% and 80% of data as training set and the remaining for testing, so the training and testing data do not have overlap. The validation data is obtained separately from the training and testing data. The dataset will be released later for further study.

3.2 Evaluation Criteria

Retweet prediction task usually aims at providing top K image tweets to a user in most online media services. To evaluate the effectiveness of our method in terms of top-K ranked image tweets, we adopt two ranking-based evaluation criteria, Precision@K [Chen et al., 2016] and AUC [He and McAuley, 2015; Rendle et al., 2009; Li et al., 2016] to evaluate the performance of image retweet prediction. Given test set of users U t and image tweets it , we denote predicted ranking of the top K image tweets from test set for a certain user ui by Rui , where size of ranking list |Rui | is K.

3.3 Performance Comparison

We evaluate performance of our method AMNL with five other state-of-the-art solutions to problem of image retweet prediction: CITING [Chen et al., 2016], VBPR [He and McAuley, 2015], FAMF [Rendle et al., 2009], ADABPR [Rendle and Freudenthaler, 2014], RRFM [Li et al., 2016].

Existing retweet prediction methods are mainly based on low-rank factorized ranking model. Methods FAMF, ADABPR and RRFM learn factorized ranking metric based on pairwise preference constraints. Methods CITING and VBPR are feature-aware factorized ranking algorithms based on pairwise preference constraints and feature of item contents.

We extract feature of item contents as follows. Input words of all textual information are initialized by pre-calculated word embeddings and input visual representation of image tweets are initialized by VGG-Net [Simonyan and Zisserman, 2014]. Parameters of the neural networks used to get the representations of visual content and textual context are updated during training process. The weights of deep neural networks are randomly initialized by a Gaussian distribution with zero mean in our experiments. Following experimental setting in [Chen et al., 2016; He and McAuley, 2015], we consider the associated textual contexts as the side information of the method CITING and the visual representation of image tweets as the side information of the method VBPR. The hyper-parameters and parameters which achieve the best performance on the validation set are chosen to conduct the testing evaluation. We set the learning rate to 0.01 for the gradient method. We think the top 3 tweets that users want to retweet can reveal the discriminative characteristics of the tweets that users want to retweet. So we evaluate the ranking performance of all methods on the quality of the top 3 ranked image tweets. To exploit the effect of the visual representation of image tweets and the semantic representation of the associated contexts to the performance of our method, we denote that our method with visual representation of image tweets only by AMNLi, and our method with semantic representation of the associated contexts only by AMNLd.

Tables 1, 2 and 3 show evaluation results of all methods on ranking criteria Precision@1, Precision@3 and AUC, respectively. Evaluation were conducted with different ratio of data as training set from 60%, 70% to 80%. We report result value of all methods using three ranking evaluation criteria. We then report performance of our model with different modalities, where dimension of user preference representation is set to 400, and 80% of data is used for training. We illustrate the effect of visual representation of image tweets, semantic representation of the associated contexts and the joint image tweet representation to our model in Table 4. These experimental results reveal a number of interesting points:

• The methods with content feature as the side information for learning the ranking metric, CITING and VBPR, outperform the low-rank factorized ranking metric methods

FAMF, ADABPR and RRFM, which suggests that the deep neural networks with both image tweets and the associated context information is critical for the problem of image retweet prediction.

• Compared with other ranking methods with the side information, our method AMNLi achieves better performance than the method VBPR, and our method AMNLd achieves better performance than the method CITING, respectively. This suggests that the multi-faceted ranking metric is important for the problem.
• Compared with our methods AMNLi and AMNLd, our method AMNL achieves better performance. This suggests that the attentional multi-faceted ranking network learning framework which exploits the joint image tweet representation of multi-modal image tweets and their associated context can get better performance than the attentional multi-faceted ranking network learning framework which only exploits the representation of tweets' images or the representation of tweets' contexts.
• In all cases, our AMNL method achieves the best performance. This shows that the attentional multi-faceted ranking network learning framework that exploits both the joint image tweet representation of multi-modal image tweets and their associated contexts, and multifaceted ranking metric can further improve the performance of image retweet prediction.

In our approach, there is one essential parameter, which is the dimension of user preference representation. We vary the dimension of user preference representation from 100, 200, to 400. We show the effect of the dimension of user preference representation using 60% of the data for training on Precision@1, Precision@3 and AUC in Figures 3(a), 3(b) and 3(c). We find out that our method's performance trend becomes stable after the dimension of user performance presentation larger than 400 with different proportions of data for training.

The updating rule for training our proposed attentional multi-faceted ranking network learning method is essentially iterative. Here we investigate how our AMNL method converges. Figures 4(a) and 4(b) show the convergence and running time curves of AMNL method, respectively. The x-axis denotes the iteration number in both figures. The y-axis in Figure 4(a) denotes the objective value and the y-axis in Figure 4(b) shows the running time of our proposed method. Each epoch contains 22,881 iterative updates. We set the dimension of user preference representation to 400, and use 80% of the data for training. We show that our method converges after 9-th epoch and the computation cost is less than 50 minutes. This study validates the efficiency of our method.

4 RelatedWork

Central problem of retweet prediction is to model tweet sharing behavior that users repost tweets along followee-follower links so that more users are informed in SMS, which has attracted considerable attention recently in [Chen et al., 2016; Firdaus et al., 2016; Zhang et al., 2015b; 2016; Wang et al., 2013; Feng and Wang, 2013]. Chen et. al. [Chen et al., 2016] exploit various contexts for image understanding and retweet prediction. Firdaus et. al. [Firdaus et al., 2016] propose a retweet prediction model by considering user's author and retweet behaviors. Zhang et. al. [Zhang et al., 2015b] propose non-parametric models to combine structural, textual, and temporal information together to predict retweet behavior. Zhang et. al. [Zhang et al., 2016] propose deep neural networks to incorporate contextual and social information. Wang et. al. [Wang et al., 2013] present a recommendation model to solve the problem of whom to mention in a tweet. Feng et. al. [Feng and Wang, 2013] propose the featureaware factorization model to re-rank the tweets, which unifies the linear discriminative model and the low-rank factorization model. Peng et. al. [Peng et al., 2011] model the retweet behavior using conditional random fields. Zhang et. al. [Zhang et al., 2015a] employ the social influence locality for modeling the retweet behaviors. Unlike previous studies, we formulate the problem of image retweet prediction from the viewpoint of attentional multi-faceted ranking network learning, which can be solved by the negative sample based ranking metric learning with multi-modal neural networks.

5 Conclusion

In this paper, we introduced problem of image retweet prediction from viewpoint of attentional multi-faceted ranking network learning. We propose heterogeneous IRM network that exploits both users' past retweeted image tweets, associated textual context and users' following relations. We present a novel attentional multi-faceted ranking network learning method with introduced multi-modal neural networks to learn joint image tweet representations and user preference representations, such that multi-faceted ranking metric is embedded in representations for prediction. We evaluate performance of our method using dataset from Twitter. Extensive experiments demonstrate that our method can achieve better performance than several state-of-the-art solutions.

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grant No.61602405 and No.61572431, Alibaba Innovative Research, Zhejiang province key project (2015C01027) and China Knowledge Centre for Engineering Sciences and Technology. This work is also Supported by Zhejiang Natural Science Foundation (LZ17F020001) and Key R&D Program of Zhejiang Province (2018C01006).