HyperNews: Simultaneous News Recommendation and Active-Time Prediction via a Double-Task Deep Neural Network

1 Introduction

Nowadays, with massive news aggregated from various channels every minute by online news platforms such as Google News and Toutiao, it is impossible for users to read through all of them [Phelan et al., 2011; Morales et al., 2012; Okura et al., 2017; Lian et al., 2018; Wu et al., 2019c]. Therefore, personalized news recommendation, which can help users cope with such information overload, has recently become a hot research topic [Cheng et al., 2016; Gulla et al., 2017; An et al., 2019; Zhu et al., 2019].

Basically, the core problem in news recommendation is to learn news and user representations. The early studies in this area typically encode news data with static features such as the title, content and categories of each news article. For example, to employ the acclaimed LibFM [Rendle, 2012] that combines the generality of feature engineering with the superiority of factorization models for news recommendation, we could simply concatenate the news features and the user features as inputs, and then get the corresponding click probabilities as outputs. Specifically, the news features could consist of the TF-IDF vectors representing the given news article's title and content as well the one-hot vector of its categories, while the user features could be extracted from the past news articles read by that particular user in the same fashion. Moreover, both DeepFM [Guo et al., 2017] and Wide&Deep [Cheng et al., 2016] try to further capture the higher-order interactions in addition to the linear and pairwise ones between features. Thus they share the same inputs as LibFM and also generate click probabilities. Similarly, DSSM [Huang et al., 2013] projects queries and documents into a common low-dimensional space where the relevance of a document to the given query could be computed by their cosine similarity score. It could also be utilized to generate news and user representations for news recommendations, as in LibFM. To sum up, the above methods are general-purpose recommender systems mainly using static features; they probably would not be able to handle the dynamics between news and users very well.

∗Corresponding author, supported in part by the China Postdoctoral Science Foundation (Grant No. 8206300295).

To deal with the possible drift of users' interests, the latest solutions for personalized news recommendation utilize not only the above-mentioned static features but also the features extracted from each user's recently read news articles to approximate his/her current reading interests. For example, DKN [Wang et al., 2018] utilizes a multi-channel and wordentity-aligned knowledge-aware CNN which fuses semanticand knowledge-level representations to construct a news encoder, and designs an attention module to dynamically aggregate a user's history w.r.t. current candidate news articles. NAML [Wu et al., 2019a] adaptively selects informative representations for news and users by designing a multi-view attentive mechanism. NPA [Wu et al., 2019b] proposes a personalized attention network which exploits user-ID embeddings as queries for the word- and news-level attention networks to realize the differentiated dynamics. LSTUR [An et al., 2019] aims to learn long-term user representations from the embeddings of their IDs and short-term user representa-

(b) News Encoder.

Figure 1: Our proposed HyperNews model.

tions from their recently browsed news articles via a GRU network to combine the long-term preferences and short-term interests for better personalized recommendation. In summary, the above relatively new approaches to news recommendation attempt to capture users' current interests implicitly through their recently read news articles. However, as we will show later, it would be even better to exploit the temporal attributes in the news dataset and model the timeliness of news explicitly.

Furthermore, we reckon that the so-called 'active-time' (i.e., the time interval from the user's click opening the news article page to the user's click closing it) attribute measures the time spent by the user to read that particular news article, and thus can partially reflect that user's reading interests: ceteris paribus, the longer the 'active-time' (reading time), the more interested the user. The problem of active-time prediction is itself meaningful for some practical applications like display advertising and network traffic control. It also needs to model the interaction between news articles and news consumers. Due to the apparent correlation between the active-time prediction task and the news recommendation task, we conjecture that it would be promising to address them together.

Contributions. Motivated by the above two considerations, we put forward a new approach to personalized news recommendation, named HyperNews, which introduces an explicit timeliness module to refine news representation, and also an auxiliary task of active-time prediction to reinforce news recommendation in the multi-task learning framework. Our experiments on real-world news datasets show that HyperNews outperforms state-of-the-art news recommendation techniques. Our analysis of the experimental results confirm that the timeliness module and the active-time prediction task both contribute to the performance improvements.

2 Problem Statement

In the problem of personalized news recommendation, we try to predict how likely a user will click on a previously unseen news article based on his or her clicking history. The recently available event-based news data (describing which user clicked on which news article at what time and closed it at what time) provide us with opportunities to carry out news recommendation in a richer context. In particular, we would argue that a user's active-time on a news article could be, and probably should be, predicted at the same time. Here, we propose a unified model to undertake these two tasks simultaneously.

Suppose a news dataset for training consists of n instances 〈x, yp, yt〉, where x is the composite feature vector representing a pair of user and news article, yp is the binary label indicating whether the user clicked on that news article (yp = 1) or not (yp = 0), and yt records the corresponding active-time. Intuitively, the news recommendation task (i.e., estimating yp based on x) and the active-time prediction task (i.e., estimating yt based on x) should be correlated with each other. Therefore, it would make perfect sense to exploit the commonalities across these two tasks by utilizing the multitask learning framework: 〈 ̂ yp, ̂ yt〉 = DoubleT askModel(x) where ̂ yp is the predicted click-probability and ̂ yt is the predicted active-time. Obviously, the click-probability output can be used to determine how highly the given news article should be recommended to the respective user. To make the active-time prediction task easier, we discretize the continuous active-time values into discrete time interval groups (e.g., "15-25 seconds"), and thus convert the regression problem into a classification problem. It turns out that such an aggressive simplification worked really well in our experiments, suggesting that a crude estimate of active-time would be good enough for our purposes.

3 The Method: HyperNews

Fig. 1a shows the overall framework of our proposed method, HyperNews, which can simultaneously conduct news recommendation and active-time prediction in the manner of multitask learning. These two tasks would share the same inputs, i.e., the attributes of news and users including 'news-title', 'news-content', 'news-category', 'publish-time', 'click-time', and 'active-time'. However, the news encoder for these two tasks would use two different sets of learnable weights to combine explicit embedding and implicit embedding as they are likely to have different impacts upon different tasks (see Section 3.1). It is worth mentioning that for news recommendation, we have designed and included a timeliness module (see Section 3.3) in order to emphasize the impact of news freshness (i.e., the elapsed time between the 'publish-time' of the news article and the 'click-time' when the user clicked to open it for reading).

3.1 News Encoder

The news encoder (Fig. 1b) is used to learn news embedding from various news attributes (i.e., title, content and categories). We divide these attributes into two types: explicit information and implicit information. The explicit information includes the news article's title and categories that users can see before making their decisions to click, while the implicit information refers to the news article's content that a users can see only after clicking. The reason why we differentiate them is that they should have different effects on click-probability and active-time. Intuitively, the explicit information would have a greater impact on click-probability than the implicit information. Once the embeddings of these two parts have been obtained, an attention unit would be utilized to adaptively learn their combinations for different tasks.

Explicit Embedding

The explicit embedding unit contains two components.

The first is to embed news titles using a typical CNN-based model with a sequence of words as inputs. Specifically, a news title is represented as a word embedding based matrix w1:n = [w1, w2, ..., wn] ∈ Rd×n , where wi ∈ Rd×1 is the i-th word of the news title, d = 100 denotes the dimension of word embedding, and n represents the number of words. Then the matrix w1:n would be applied with a convolution operator via a filter H ∈ Rd×l , where l = 3 (l < n) means the window size. For each sub-matrix wi:i+l-1, we can get a feature ci by:

where ∗ is the convolution operator, b is a bias term, and f is the ReLU activation function. With all the possible positions of the matrix w1:n crossed, we get a feature map:

where ci marks the i-th result of the filter H. Then we utilize a max-pooling operation to identify its most significant feature:

Next, we take m = 200 filters with the same window sizes (i.e., l = 3) to conduct the same operation as above, and each filter will generate a feature. As a result, the m features are concatenated together to finally represent the news title as:

The second component is to embed news categories. The reason why we take them into consideration is that users usually only care about the news articles within those categories of their interests and would barely click on any news article outside. Regarding this part, the input for each category is a unique id, which is randomly initialized with a low-dimensional (e.g., 100D) vector ec. After that, a dense layer is employed to learn the category's hidden representation:

whereWc and bc are learnable parameters. Since each news article may have a different number of categories and usually different categories have different influences, we choose the top three most important categories and then use an attention unit to calculate a weight for each one. The final representation ˜ec for news categories is the weighted sum of their hidden representations. Note that if a news article has fewer than three categories, we would pad the input with zero vectors. Denote the weight of the i-th category as αc i , then there holds:

and

where qc, Vc and b′c are the parameters of the attention unit.

In the end, the representations of news-title and newscategories would be concatenated together, and then fed into a one-layer fully connected neural network (FNN) to get a lowerdimensional embedding as the representation of the explicit information:

Implicit Embedding

The implicit embedding unit is to embed news content.

First, we train the LDA [Blei et al., 2003] topic model on the whole Adressa 1 news dataset, based on which we could infer a probabilistic representation for any news article's content. Let D denote the news article's content, then the topic representation (#dimensions=100), regarded as part of the input, is given by:

Second, we use doc2vec [Le and Mikolov, 2014], also trained on Adressa, to construct another document representation (#dimensions=100) as:

Through the utilization of doc2vec, the semantic representation of news content would be further enriched. Note that the above two steps can both be processed in advance and therefore would not incur extra computational costs for our model. Third, the length of news content is obviously a big factor affecting users' active-time on that piece of news, so we include it as useful feature for our model. Specifically, the raw lengths of news content would be divided into a series of bins each with an interval of 50 words, and then similar to word embedding, we represent each news-length-bin feature as a low-dimensional (e.g., 100D) vector el. It would be randomly initialized and then go through a fully-connected layer to become the final representation:

where el, Wl and bl are to-be-learned parameters.

Finally, we concatenate these three vectors and feed them to a one-layer FNN to represent the implicit information:

Attention Unit

Intuitively, the explicit information and the implicit information would exert different influences on click-probability and active-time. Therefore we introduce an attention unit to automatically learn the weights for these two different parts. Denote the two weights as αn 1 and αn 2 , we have:

and

where qn, Vn and bn are learnable parameters. Afterwards, the news embedding is calculated as:

Note that in the HyperNews model, we actually would have three different sets of values for those parameters (qn, Vn, bn) which would lead to three separate news encodes ep , et and eu for click-probability, active-time, and user representation respectively (see Fig. 1a).

3.2 User Encoder

The user encoder is to learn a user's feature representation from that user's click history. Today, there are two ways which have been proved effective. To be specific, recurrent networks (like GRU and LSTM) are skilled at encoding sequential data, while attention networks may be better for distinguishing important clicked news articles from the others. Here, we choose the second manner, because the series of recently read articles are not a strictly continuous sequences (there could be a very long interval between two clicks). Denote the weight of the i-th clicked news as αu i , then we could arrive at:

and

where qu, Vu and bu are the parameters of the attention unit to be learned, and eU is the final user embedding. Note that we collect a user's 30 most recently clicked news articles to model that user's current interests.

3.3 Click-Probability Prediction

The freshness of a news article for a user, defined as the elapsed time from that news article's 'publish-time' to its 'click-time' by that user, is pretty important for personalized news recommendation. Therefore we design a timeliness module (Fig. 1a) to refine the embedding from the news encoder. Specifically, we divide the continuous time into variable-length time interval bins [1h, 2h, · · · , 1d, 2d, · · · , 1m, 2m, · · · ], where h, d and m represent hour, day and month respectively. For example, if it is 75-minutes, we would assign it to the 2h bin and set its id as 2. For each interval id, a low-dimensional (e.g., 200D) vector ef could be randomly initialized and then forwarded to a dense layer:

where ef , Wf and bf are parameters to be learned. In what follows, a new time-enriched news embedding for news recommendation can be reached via:

where ⊙ is the Hadamard (element-wise) multiplication operator. Finally, the click-probability (how likely a user clicks on a news article) is calculated by the sigmoid function of the dot product between user embedding and news embedding, i.e.,

based on which our news recommender system can recommend to each user the corresponding top-k (e.g., k=20) news articles with the highest scores.

3.4 Active-Time Prediction

Making very precise estimations of active-time is challenging, but actually it may not be really necessary. For example, whether a user has spent 50 seconds or 60 seconds on a news article probably would not make much difference. Therefore we convert the continuous time regression task into a discrete time interval classification task. In what follows, we plot "the number of events (#events) vs. active-time" of the whole Adressa dataset in Fig. 2a and find that it is a long-tailed distribution; then we partition the active-time range [5, 205] equally into 20 time interval bins (Fig. 2b). The reasons why we conduct this division are twofold: (1) the events of active-time in the range of [5, 205] account for about 90% of the cases; (2) the active-time values fewer than 5 seconds are probably anomalies. Note that the 20 time intervals are numbered 1, 2, · · · , 20 as class labels.

In the end, we combine the news embedding and user embedding as eT = [et; eU ], and then utilize a softmax classifier to calculate the probabilities over the 20 time intervals:

whereWt and bt are the classifier's parameters. Afterwards, we conduct active-time prediction by selecting the most likely time interval corresponding to the largest element of ̂yt.

3.5 Model Training

Similar to Ref. [Zhu et al., 2019], we also regard the observed click events as positive samples and unobserved ones as negative samples. In what follows, we denote a training sample as X = ({x1, · · · , xr}, xr+1, yp, yt), where {x1, · · · , xr} represents the r news articles recently read by the user, xr+1 is the candidate news article, yp is the click label (1 for clicked and 0 otherwise), yt is the active-time label (a one-hot vector for the time interval id; invalid for negative samples). The output of our double-task HyperNews model would include both ̂yp and ̂yt: their respective loss functions are as follows:

and

where S+ and S- are the set of positive samples and the set of negative samples respectively, while S∗ is the set of positive samples with the active-time attribute.

As Fig. 2b clearly says, the active-time prediction task is probably susceptible to severe class imbalance. Therefore we adjust each class's weight based on its effective size [Cui et al., 2019]. Specifically, the effective size of a class is defined as:

where m is the actual number of samples in the class, and β is a hyperparameter whose value is usually close to 1 (e.g., 0.99, 0.999, etc.). Accordingly, the class-balanced variant of Lt can be written as:

where myt is the actual number of samples in class yt.

Taking both Lp (Eq. (25)) and L′t (Eq. (28)) into consideration, the overall loss function to minimize is:

where λ is a non-negative parameter to balance the importance of news recommendation and active-time prediction.

During training, we apply two regularization techniques, dropout (rate=0.5) and batch normalization, on the fullyconnected layers to avoid overfitting. The popular Adam optimizer (with learning rate=1e-6) is employed. The batch size has been set to 400, while the number of epochs for convergence has been set to 20.

4 Experiments

In this section, we examine the effectiveness of HyperNews by comparing it with several competitive baselines.

4.1 Datasets

Adressa 1 [Gulla et al., 2017] is an event-based real-life news data collection that contains a large number of Norwegian

1 http://reclab.idi.ntnu.no/dataset/

,

news articles in conjunction with their readers. In our experiments, we represent each news reading event by its most important attributes: 'user-id', 'news-id', 'news-title', 'newscontent', 'news-category', 'publish-time', 'click-time', and 'active-time'. Among them, the three temporal attributes - 'publish-time', 'click-time', and 'active-time' - have not ever been considered in previous news recommendation studies, but we have found that they could be exploited for more accurate modeling of the interaction between users and news articles.

Two editions of Adressa dataset, dubbed Adressa-1week and Adressa-4week, have been constructed, which correspond to the first 1 week's and the first 4 weeks' news consumption event-logs, respectively. For Adressa-1week, we take the first six days' events to train predictive models and the last day's events to test their performance; while for Adressa-4week, we take the first three weeks' events as the training set and the last week's events as the test set. Table 1 shows some characteristics of these two datasets.

4.2 Competitors andMetrics

To evaluate the performance of our HyperNews, several stateof-the-art competitors have been invited for comparison, including LibFM 2 [Rendle, 2012], DeepFM 3 [Guo et al., 2017], Wide&Deep 4 [Cheng et al., 2016], DSSM 5 [Huang et al., 2013], LSTUR 6 [An et al., 2019], and DAN 7 [Zhu et al., 2019]. Moreover, to verify the usefulness of multi-task learning in our application, we construct two single-task editions of HyperNews by removing some parts of the model: (1) HyperNews(NoPred) which is a single-task news recommendation model without the active-time prediction component, and (2) HyperNews(NoRecom) which is a single-task activetime prediction model without the news recommendation component.

We have made the source code of HyperNews and also the processed datasets for our experiments available online 8 .

Regarding the performance measures, we employ the popular AUC [Fawcett, 2006] and F1 [Li et al., 2008] metrics for news recommendation, and just F1 for active-time prediction.

2 https://github.com/srendle/libfm

3 https://github.com/ChenglongChen/tensorflow-DeepFM

4 https://github.com/kaitolucifer/wide-and-deep-learning-keras

5 https://github.com/InsaneLife/dssm

6 The source code is kindly provided by the authors.

7 https://github.com/zhuqiannan/dan-for-news-recommendation

8 https://github.com/penghll/Hypernews

4.3 Settings

There are two groups of competitors: general-purpose recommender systems (LibFM, DeepFM, DSSM, Wide&Deep) and news-oriented recommender systems (LSTUR, DAN). In accordance with the widely-adopted settings in Refs. [An et al., 2019; Zhu et al., 2019], for the former group, we just take the concatenation of news title, content and categories as input features; whereas for the latter group, we adopt the configurations as described in each method's corresponding paper. The baseline methods' parameters have been tuned on Adressa-1week to get competitive performances and then applied to both datasets. In the same way, the parameters of HyperNews are configured as λ = 0.8 and β = 0.99999 (see Section 4.5).

4.4 Results

Table 2 shows the AUC and F1 scores of different news recommendation methods. Clearly, our multi-task learning approach - the complete HyperNews system - defeats all the other competitors, which confirms its high effectiveness. It can be observed that the news-oriented models (LSTUR, DAN, and HyperNews) can usually achieve higher scores than the general-purpose models (LibFM, DeepFM, Wide&Deep, and DSSM), probably because the former only resort to basic features (like title, content, categories) but ignore task-specific information utilized in the latter . Moreover, the double-task HyperNews model exhibits a substantial performance gain over the single-task HyperNews(NoPred) model, which tells us that the active-time prediction task does help the news recommendation task through their joint learning of the neural network.

In addition, the results of active-time prediction are shown in Table 3. The F1 scores of the double-task HyperNews model are significantly higher than those of the single-task HyperNews(NoRecom) model, which suggests that the news recommendation task could reinforce the active-time prediction task as well.

4.5 Hyperparameter Analysis

HyperNews contains two hyperparameters: λ that controls the balance between news recommendation and active-time prediction, and β that is used to calculate each class's effective size. Keeping β = 0.99999 fixed, we vary λ from 0 to 5 and plot the experimental results in Fig. 3a. It can be seen that on Adressa-1week, no matter what task it is and which metric is adopted, λ = 0.8 always leads to the best performance. Similarly, keeping λ = 0.8 fixed, Fig. 3b reveals that when β = 0.99999, HyperNews yields the best results for both tasks in terms of both metrics. Note that when β = 0, Eq. (28) would be equal to Eq. (26).

4.6 Ablation Study

Fig. 4 compares the complete HyperNews system with the ablated version of HyperNews without the timeliness module. It is clear that on both datasets, for both tasks, under both performance measures, the complete HyperNews outperforms the ablated HyperNews consistently and substantially. This confirms the critical contribution of the timeliness module.

5 Conclusion

This paper investigates the usefulness of temporal attributes to news recommendation, which has been neglected before. Specifically, we propose a novel deep neural network model named HyperNews which includes a timeliness module and utilizes a multi-task learning framework (to conduct news recommendation and active-time prediction simultaneously). Our extensive experiments on real-life news datasets have confirmed that the explicit timeliness module and the auxiliary active-time prediction task do benefit news recommendation greatly and thus enable HyperNews to beat a number of stateof-the-art news recommendation techniques.