Sequential Prediction of Social Media Popularity with Deep Temporal Context Networks

1 Introduction

Social media is now globally ubiquitous and prevalent. Consequently, understanding and predicting popularity in social media (e.g., Twitter, Facebook, Youtube and Flickr) has attracted great attention [Wu et al., 2016b; 2016a; Li et al., 2015; He et al., 2014; Khosla et al., 2014; Pinto et al., 2013], since it offers opportunities to reveal individual preference and public attention from evolutionary social systems. Accurate popularity prediction can help improve user experience, service effectiveness, and benefit a broad range of applications, such as content recommendation [Khosla et al., 2014], online advertising [Li et al., 2015] and information retrieval [Roy et al., 2013; Gan et al., 2016].

Previous research, although achieves promising results, neglects one distinctive characteristic of social data, i.e., temporal sequentiality. For example, the popularity generated by photo sharing in Instagram on a weekday is observed to be at the peak during hours at 2 a.m., 8-9 a.m. and 5 p.m. [Beese, 2016]. Existing predictive algorithms on social media popularity are not considering the temporal order of data, making them have limited success to sequential data scenarios (e.g. news feed, tweet timeline, photo stream, etc.). Particularly, most of the existing works on popularity prediction are based on the content of a post or the person who published the post [Shamma et al., 2011; Gelli et al., 2015; Cappallo et al., 2015]. Recently, although some researchers have analyzed temporal characteristics of social popularity (e.g., temporal fluctuations), the temporal interrelationship of popularity data is not explicitly exploited [Shen et al., 2014; Zhao et al., 2015; Wu et al., 2016a; 2016b; Martin et al., 2016].

In this research, we take one step further to investigate the problem as a temporal prediction task with sequentiality. Unlike previous work on using time information as latent factors or variables [Shen et al., 2014; He et al., 2014; Zhao et al., 2015], our purpose is to predict popularity from both sequential and temporal views based on time series data. Specifically, we incorporate a time-centered perspective called 'temporal context' into popularity prediction, which was inspired by social psychology for human behavioral processes [McGrath and Kelly, 1992]. Utilizing temporal context of popularity as a novel prior knowledge, we attempt to predict new popularity by exploiting consecutive temporal coherence of popularity at multiple time-scales.

Based on the above idea, we propose two types of sequential temporal contexts for learning two different types of temporal coherence of popularity: Neighboring Temporal Context (NTC) and Periodic Temporal Context (PTC). On one hand, we model NTC to learn the continuous trend or patterns in short-term time series. For instance, before and after Thanksgiving Day, the popularity of pumpkin picture rises and decays in only a few days. The NTC helps us reveal the trend and variances. Such temporal coherence has also been successfully applied to the citation estimation of scientific articles [Shen et al., 2014]. On the other hand, we model PTC to learn discontinuous temporal coherence in long-term time series. Temporal coherence is often influenced by periodic events or human activities. For example, according to the survey in 2016 [Ellering, 2016], the best time periods to make a post on Facebook are Saturday and Sunday around 12 a.m. to 1 p.m., with periodic peaking time from 9 a.m. to 3 p.m. And weekly peak time for Pinterest is on Saturday from 8 p.m. to 11 p.m. These findings motivate us to consider both of the temporal and sequential characteristics for predicting popularity more precisely.

In this paper, therefore, we propose a novel deep prediction framework called Deep Temporal Context Networks (DTCN) by exploring both temporal contexts and temporal attention at different time-scales jointly (such as days of a week, hours of a day). Figure 1 shows the overview of our DTCN framework, which is a sequential prediction architecture containing three main components: from embedding, learning to predicting, along with the model trained as an entire network by optimization learning. With a joint embedding network, we map multi-modal data onto the same embedding space to obtain a unified deep representation for user-post sharing activities. Then, based on the embedded data sequence, we design temporal context learning to recurrently learn the dynamic popularity from adaptive temporal contexts (NTC and PTC) for popularity over time. Finally, we provide a multiple time-scale attention for computing multi-scale temporal coherence in predicting the popularity of a new user-post pair. Unlike previous work on temporal modeling for popularity prediction [Wu et al., 2016b; 2016a; Li et al., 2015], our study attempts to provide a novel view on temporal context modeling, which considers both of the temporal and sequential coherence during prediction.

The main contributions of this study are: (i) To our best knowledge, we are the first to consider both temporal and sequential characteristics into sequential prediction of social media popularity over time; (ii) we address the problem by incorporating a temporal context perspective, and proposing two types of novel temporal contexts including NTC and PTC; (iii) we propose a novel deep prediction model DTCN architecture jointly integrating embedding, temporal context learning and predicting with temporal attention to optimize the entire network, which outperforms state-of-the-art predictive algorithms in sequential popularity prediction.

2 RelatedWork

Recently, time-aware popularity prediction receives much attention. Both academia and industry have paid more effort on this research topic. Existing works about prediction with temporal information can be concluded into two main paradigms. The first paradigm focuses on predicting the popularity growth of a published post by analyzing its temporal trend and pattern at early stage. Szabo and Huberman proposed to predict popularity based on growth pattern characteristics of online popularity at early stage [Szabo and Huberman, 2010]. Yang and Leskovec found the temporal patterns reveal how the contents popularity grows and fades during the post propagation [Yang and Leskovec, 2011]. Roy et al. proposed to grasp sudden bursts of a post popularity with crossdomain knowledge [Roy et al., 2013]. Kong et al. explored the problem of detecting which hashtag would be bursting. Most of these works provided dynamic comprehension on popularity accumulation of online content, but their prediction needs to rely on early stage popularity patterns of a published post. Our method takes one step further on popularity prediction before that the corresponding post was published.

The other paradigm is predicting popularity based on temporal features or dynamic signals. Zhao et al. applied human reaction time as temporal variables in self-exciting point processes [Zhao et al., 2015]. Shen et al. proposed a reinforced Poisson process to model the dynamic popularity based on the arrival time of attention [Shen et al., 2014]. He et al. designed a time-aware bipartite graph for popularity estimation [He et al., 2014]. Wu et al. proposed to predict popularity by context factorization and tensor decomposition algorithms to unfold popularity dynamics [Wu et al., 2016b; 2016a]. These models all neglect the sequentiality of popularity during prediction.

Summary. We focus on the investigation of sequential temporal coherence of popularity over time before the corresponding sharing behavior happened. Being differed from previous works on the predicting popularity with temporal information e.g., [Shen et al., 2014; He et al., 2014; Zhao et al., 2015; Wu et al., 2016a; 2016b], we explore the temporal context of popularity from sequential data and consider both of sequential and temporal characteristics of popularity during prediction. It is worth mentioning that our algorithm is generic and applicable to the prediction of other sequential data scenarios in social media, e.g. news feed, tweet timeline, photo stream, and so forth.

3 Social Media Popularity Prediction

Taking photo popularity prediction on Flickr as an exemplary case, we define our problem and introduce the definitions of several core concepts, including user-post sequence, temporal context, and multiple time-scales.

Problem Definition: Given a new photo v of a user u, the problem of predicting its popularity s is to estimate how many attentions would be obtained after the post was published on social media (e.g. views, likes or clicks etc.).

On Flickr, when browsing a personal photo stream or image search results, users can view details of a photo content with its metadata through clicking photo thumbnails. In our prediction, since "viewing count" is a significant indicator of how popular a photo is, we use it to describe the photo popularity as follows:

Popularity Normalization. To suppress the large variations among different photos (e.g. view count of different photos vary from zero to millions), we implement a log function to normalize the value of popularity, based on the previous work [Wu et al., 2016a; Khosla et al., 2014]. In brief, the log-normalization function for popularity can be defined as:

where s is the normalized value, r is the view count of a photo, and d is the number of days since the photo was posted.

Several core concepts in the paper are also defined as follows:

Definition 1: User-Post Sequence. A user-photo pair p ⇔ 〈u, v〉 is derived from the photo sharing behavior (e.g., post publishing, photo sharing and video uploading), which made by user u on photo v. Suppose we have n user-photo pairs and the sharing time of each pair. Then the user-post sequence can be denoted by S = {〈u1, v1〉 , 〈u2, v2〉 , ..., 〈un, vn〉} with its sharing time order t1 ≤ t2 ≤ ... ≤ tn.

Definition 2: Temporal Context. Given the prediction target p and its previous user-post sequence Pu before time t, temporal context (as a prior knowledge) is a time series, which was built on Pu and expressed by a triple sequence: Cp = {〈u1, v1, tp1 〉 , 〈u2, v2, tp2 〉 , ..., 〈uk, vk, tpk 〉}, tp1 ≤ tp2 ≤ ... ≤ tpk ≤ t where k is the item count in the temporal context.

Definition 3: Multiple Time-scales. For time order sequence, time-scale determines a time unit of the sequential data. In our paper, without loss of generality, there are four levels of time-scales, Tunit = {t1M, t1P , t1D, t1W}, which means that minute of a hour, period of a day, day of a week and week of a month. With regard to the period of a day, we segment one day into six periods [Wu et al., 2016a], i.e. "morning (8:00am-12:00am)", "lunch time (12:00am14:00pm)", "afternoon (14:00am-17:00pm)", "dinner time (17:00am-20:00pm)", "evening (20:00am-24:00pm)" and "night (0:00am-8:00am)".

4 Deep Temporal Context Network

How can the temporal context at multiple time-scales be utilized for popularity prediction? We address the problem as a sequential prediction task, where the input is a user-photo sequence (with time order) while the output is the popularity of a "future" photo (a photo before its publication on social media). We propose a deep prediction framework called DTCN containing three components: (1) Multi-modal Joint Embedding (2) Temporal Context Learning and (3) Multiple Time-scale Temporal Attention. With all the components, the model is trained as an entire network by optimization algorithm RMSprop [Tieleman and Hinton, 2012].

4.1 Multi-modal Joint Embedding

In the first stage of DTCN, Multi-modal Joint Embedding (MJE) is to generate a unified deep representation of multimodal sequence data. While an individual popularity is highly correlated with user-photo interactions [Cappallo et al., 2015; Khosla et al., 2014; Qian et al., 2014], our embedding model is not only designed to correctly understand what appears in photos but also incorporate the knowledge of who posts it.

Suppose that a variety of photo sharing behaviors over time can be naturally viewed as a user-photo sequence. The bimodal sequence data are the input data of our embedding network. In order to parametrize the visual content and user influence from the data sequence, we design a two-stream Feedforward Neural Network (FNN), which has both of the user analysis pipeline and the photo analysis pipeline respectively. On one hand, the photo analysis pipeline starts with a pretrained ResNet [He et al., 2015] in 152 layers for generating high-level visual representations of 2048 dimensions. On the other hand, we adopted a group of features to measure user influence in the user analysis pipeline, such as the average value of views, photo count, the number of contacts, mean number of members in a user's groups, and having a Pro Flickr account or not. In our embedding network, each of the pipelines contains four layers with two hidden layers and shares the same dimension number of hidden neural nodes of 256 and 32. In order to perform non-linear mapping of features from the original space to a new latent space, the tanh activation function is applied in each layer of the embedding networks. Meanwhile, we apply a random dropout mechanism with parameter 0.5 in each hidden layer of the two pipelines, since dropout strategies have been adopted to prevent over fitting in neural network training phrase [Srivastava et al., 2014]. In the end of the architecture, user and photo information xu and xv generated from the last layers of the two-stream FNN are embedded together by minimizing on embedding loss:

where U and V are the collections of all users and photos respectively, and || · ||2 means L2 norm regularization for loss function. The final output of the embedding network is a 64dimensional vector. By joint training with all the other components of the DTCN framework, we obtain an embedding representation of the user-photo sequence.

4.2 Temporal Context Learning

In the learning stage of DTCN, Temporal Context Learning (TCL) is to learn sequential and temporal coherence from the temporal context for prediction. Different from traditional context learning, our temporal context is adaptive time series and correlated with the time information of a prediction target and its previous user-photo sequence.

The first step of TCL is constructing temporal contexts as prior knowledge for each "future" popularity. To model short or long-term fluctuations of popularity over time, we propose two variable-length temporal contexts: Neighboring Temporal Context (NTC) and Periodic Temporal Context (PTC). Suppose we already have the embedding data of previous user-photo sequence Pu for a prediction target p, the data items are corresponding with the user-photo pairs in Pu. Specifically, the temporal context Cp is a time series generated from the time information of Pu. Therefore, NTC is consist of neighboring items of the prediction target from its previous data sequence, which is applied to describe rapidlyvarying popularity fluctuations in a short-term time range; PTC is built with discontinuous items from the previous data sequence, which is applied to represent periodic popularity patterns in a long-term time range. Here we design two discrimination functions of NTC and PTC to determine whether a certain item s ∈ Pu would be a possible element of the corresponding temporal context. The NTC discrimination function is defined as follows:

where tp and ts is the sharing time of p and s, and ∆tunit is the duration of a time unit. Dirac delta function δ(·) is to compute a discrimination signal, and we use l in discrimination function to control the signal condition lengths of time span for temporal context computation. In addition to shortterm fluctuations, popularity also has periodic variances in long-term. Therefore the discrimination function for PTC is:

where the modulo function mod(·) is to detect whether p and s are in the same time blocks in different time-scales (e.g. the same period in different days or the same day in different weeks). In order to learn consecutive variance from both of NTC and PTC, we maintain the time-scale tunit ∈ Tunit of them to be same in context learning.

To learn consecutive coherence from temporal contexts, we provide a two-stream recurrent neural network based on LSTM [Hochreiter and Schmidhuber, 1997] with Mean Squared Error (MSE) loss function as the optimization objective. Our network not only considers contextual information but also utilizes temporal information to learn the temporal context for both short-term and long-term temporal coherence. Taking the embedding data sequence as the input of LSTM, TCL is able to learn and read the surrounding temporal context for each prediction target by controlling recurrent state updates of the network.

4.3 Multiple Time-scale Temporal Attention

Multiple Time-scale Temporal Attention (MTTA) is proposed to consider the dynamic impact of each contextual item into prediction. Intuitively, we incorporate the attention mechanism to be infer the temporal attention across multiple timescales between previous instances and new instances for new popularity prediction. Different from traditional attention, multiple timescale coherent among data need to be learnt from both of weights of relative hidden states and multiple scales of time-series.

Considering all data items of temporal context, the temporal context vector ci is the input of LSTM at the step i. It can be computed by a weighted sum of hidden state hi:

where the αij is attention weight. It is calculated by comparing current prediction target p and the data item of previous temporal context at position j:

where ¯ t denotes multi-scale time vector. Note that unlike in the traditional attention mechanism, we use attention score eij to leverage the temporal consistency between ¯ tp and ¯ ts. Its calculation relies on time vector ¯ t = (t1M, t1P , t1D, t1W ) with multiple time-scale information instead of hidden state vector h for temporal attention. To compute the temporal consistency, we apply cosine distance function as a simple metric, while it can be alternated by other distance functions.

5 Experiments

In this section, we demonstrate the effectiveness of the proposed framework on Flickr dataset as follows: (1) we compare performances between our proposed method and current state-of-the-art algorithms for popularity prediction on different data scales respectively. (2) we provide the experiment results of using single temporal context on DTCN and demonstrate the performance results with different context types, considering temporal unit and time-scales.

5.1 Experimental Setup

Temporal Popularity Image Collection (TPIC17) 1 .

TPIC17 is an image popularity dataset with multi-faceted information, such as user profile, photo metadata, and visual content. To construct the sequential prediction dataset and protect the privacy of photo sharing behaviors, we extracted time information of the adopted time-scales from the original timestamps of photo sharing. It contains 680K photos in total, and the sharing time of photos are over three years from Flickr. In order to use the temporal information from the dataset, we extracted time information from the metadata. To obtain multiple data settings with different dataset size, we individually sampled three sub datasets (100K, 200K and 400K) from the 680K dataset to evaluate algorithms.

Moving Partition Validation. In order to evaluate sequential prediction task, we have a 5-round moving partition strategy on the segment of training and testing data for validation. We organized the entire data in time order and divided it into 14 parts in total. The moving partition strategy is using data from a moving time window recurrently. In our experiments, we use the data in time window as input for each round. In our experiments, the length of time window are 10, which means the partition ratio of training versus testing data is 9:1.

Evaluation Metrics. We evaluated the prediction performance of our approach and the baselines on a correlation metric Spearman Ranking Correlation (SRC) and a precision metric Mean Absolute Error (MAE). SRC is to measure the ranking correlation between ground-truth popularity set P and predicted popularity set ˆ P, varying from 0 to 1. If there are k samples, the SRC can be expressed as:

where ¯ P and σP are mean and variance of the corresponding popularity set. Furthermore, we also use Mean Absolute Error (MAE) to calculate the averaged prediction error:

5.2 ComparedMethods

In order to compare with state-of-the-art models, we implemented the following approaches which can be applied into popularity prediction task as baselines.

Baseline 1 & 2: Convolutional Neural Networks (CNNAlexNet [Krizhevsky et al., 2012] and CNN-VGG [Russakovsky et al., 2015]). CNNs have been proved a powerful tool in the field of image understanding for photo popularity prediction [Cappallo et al., 2015]. Consequently, we apply CNN-AlexNet and CNN-VGG as the baselines for CNNs, and fine tuned the 8-layer AlexNet and 19-layer VGG as a regression task that is optimized with our image dataset to predict photo popularity. Different from other methods, the input of CNNs baseline is original image file only.

1 We released the data set in https://github.com/social-mediaprediction/flickr-data-prediction-2017

Baseline 3: Support Vector Regression (SVR) [Khosla et al., 2014]. Khosla et.al. implement SVR algorithm in popularity prediction task, which utilized user information and visual content together as feature vectors by using linear kernel. Simultaneously, we add temporal information into SVR as another baseline method that denotes as SVR(T).

Baseline 4: Multiple Layer Perceptron (MLP) [Zhang et al., 1998]. MLP is a typical feedforward neural network trained with back propagation, which has the general ability and does not require any assumption about the distribution of training data to solve real-world problems. Meanwhile, we also implement a variant MLP(T) by adding temporal information to the input vector of MLP.

Baseline 5: Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997]. LSTM is a recurrent neural network (RNN) architecture, which is capable of dealing with sequential information for popularity prediction.

Baseline 6: Contextual LSTM (CLSTM) [Ghosh et al., 2016]. Since the contextual deep learning is a relatively new research direction, there are only few existing research works to compare with. To the best of our knowledge, the most closely related work is CLSTM, which also considered contextual information in training and predicting phrase. Therefore, we used Contextual LSTM as a representative of Contextual Recurrent Neural Network (CRNN). Different from traditional LSTM, CLSTM takes the contextual information into account during the training and predicting process, and it has been proved feasible in NLP tasks. The parameters of Recurrent Layers (e.g. LSTM) and hidden neural nodes shared the same settings over proposed prediction framework (DTCN), LSTM and CLSTM. The numbers of output dimensions are 64 with the "hard sigmoid" activation function.

5.3 Prediction Performance

The prediction performances of our proposed method and the compared models are shown in Table 1 and Figure 2. Overall, our approach DTCN achieves the best prediction performance on three data size settings with highest SRC of 0.6692

and minimum MAE 1.2341. DTCN outperforms state-of-theart deep prediction algorithms (MLP, MLP(T), LSTM and CLSTM), with an average of 21.51% relative performance improvement on SRC. Using raw data from images directly without user metadata and temporal information, both CNNAlexNet and CNN-VGG receive the worse results compared to other deep learning baseline algorithms. As multiple layer networks, MLPs obtains higher effectiveness than SVRs on prediction, while both of them are using user and visual features. From SVR and MLP to SVR(T) and MLP(T), there are neck-to-neck results across different sizes of datasets. Thus involving temporal information as non-sequential feature only has limited power for prediction, the behind reason might be ignorance of the sequential coherence of temporal information. With consideration of sequential coherence in the modeling of prediction, LSTM and CLSTM achieve good performances on SRC, i.e., 0.4629 and 0.6072, which are better than MLP and lower than DTCN. Overall, the results of sequential prediction (i.e. RNNs) are much superior than the results of non-sequential prediction (e.g. SVR, MLP or CNNs) on predicting social media popularity. The prediction accuracies of different methods are shown in Figure 2. From the histogram of MAE, our model also achieves the minimal prediction error 1.2341. From CNNs to DCTN, it can be seen that the error drops from about 2.1 to 1.2, and DTCN has stable improvements on TPIC17-100K, TPIC17200K, and TPIC17-400K dataset. That means our approach provides a more accurate model for popularity prediction.

5.4 Temporal Context Analysis

Here we analyze different types of temporal context in terms of using only one of them in our prediction model on TPIC17100K. There can be different variants of distinct temporal context type settings TCtunit -tr, based on the type of time unit tunit and the temporal range of context tr chosen. Generally, short-term settings are applied for NTC types, which ranges from 0.5 day to 1 week in our experiment. Simultaneously, PTC types are applied to explore long-term patterns for popularity evolving, which ranges from 3 days to 1 month. The time unit tunit control the time-scale of NTC and PTC. For instance, TC1P -1D of NTC means to construct NTC from the data in previous 1 day range using the timescale on period of day. Another case is TC1D-3W of PTC, which means the range of PTC for prediction target is in the same weekday of previous 3 weeks.

The results of using different types of NTC and PTC in our model are shown in Table 2, where the highest SRC is 0.5875 when using TC1P -1D of NTC. Overall, the results using NTC settings are better than PTC settings, which reveals that there are obvious short-term patterns in the 100K dataset of Flickr. Next, we analyze the improvements of performance in terms of NTC and PTC, respectively. Compared with the performance of DTCN on same data in Table 1, the performances of using a single temporal context (NTC or PTC) in Table 2 drop down from 0.5990 to 0.5875 and 0.5745. This finding illustrates that prediction using both of NTC and PTC on same data performs more accurately than using one of them only, and our method can incorporate them together effectively.

6 Conclusions and FutureWork

This paper proposes to learn consecutive temporal coherence of temporal context for predicting social media popularity over time, which comprises of three components. Firstly, we design a joint embedding network for multi-modal feature representation. Then, we propose and utilize two types of temporal context alignment to learn sequential popularity in short-term and long-term popularity fluctuations. Furthermore, we provide a temporal attention mechanism for predicting popularity at multiple time-scales. To evaluate our approach, we constructed a publicly available dataset with user-photo sequence data. Experimental results show that our prediction network achieves promising performances and outperforms the state-of-the-art deep prediction algorithms by 5.79%-44.86% relative improvements on TPIC17. The technique serves a deep prediction framework for sequential popularity prediction, and our paper along with the released dataset further helps promote the research.

There are several possible directions for future investigations on social popularity prediction. One is considering the social network structure (and inherent social network analysis techniques) to improve the popularity prediction. Another open question is to exploit impact of most influential users in the social network.

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China under Grant 2016YFB0800403 and the National Nature Science Foundation of China (61525206, 61571424).