Personalized Multimedia Item and Key Frame Recommendation

1 Introduction

Living in a digital world with overwhelming information, the visual based content, e.g., pictures, and images, is usually the most eye-catching for users and convey specific views to users [Zhang et al., 2018; Yin et al., 2018; Chen et al., 2019]. Therefore, when recommending or advertising multimedia items to users, an emerging trend is to present each multimedia item with a display image, which we call the key frame in this paper. E.g., as shown in Fig. 1, the movie recommendation page usually displays each movie with a poster to attract users' attention. Similarly, for (short) video recommendation, the title cover is directly presented to users to quickly spot the visual content. Besides, image-based advertising promotes the advertising item with an attractive image, with 80% of marketers use visual assets in the social media marketing [Stelzner, 2018].

As a key frame distills the visual essence of a multimedia item, key frame extraction and recommendation for multimedia items is widely investigated in the past [Mundur et al., 2006; Zhang et al., 2013; Lee et al., 2011]. Some researchers focused on how to summarize representative video content from videos, or apply image retrieval models from text descriptions for universal key frame extraction [Mundur et al., 2006; Wan et al., 2014]. In the real-world, users' visual preferences are not the same but vary from person to person [Matz et al., 2017; Wu et al., 2019]. By collecting users' behaviors to the images of the multimedia items, many recommendation models could be applied for personalized key frame recommendation [Adomavicius and Tuzhilin, 2005; Rendle et al., 2009; Covington et al., 2016]. Recently, researchers have made one of the first few attempts to design a computational KFR model that is tailored for personalized key frame recommendation in videos [Chen et al., 2017b]. By exploring the time-synchronized comment behavior from video sharing platforms, the authors proposed to encode both users' interests and the visual and textual features of the frame in a unified multi-modal space to enhance key frame recommendation performance.

Despite the preliminary attempts for personalized key frame recommendation, we argue that these models not well designed due to the following two reasons. First, as the key frame is a visual essence of a multimedia item, recommending a key frame is always associated with the corresponding recommended multimedia item. Therefore, is it possible to design a model that simultaneously recommends both items and personalized key frames? Second, most of the current key frame recommendation models relied on the finegrained frame-level user preference in the modeling process, i.e., users' behaviors to the frames of the multimedia item, which is not available for most platforms. Specifically, without any users' frame behavior, the classical collaborative filtering based models fail as these models rely on the userframe behavior for recommendation [Rendle et al., 2009; Koren et al., 2009]. The content based models also could not work as these models need users' actions to learn the visual dimensions of users [Hu et al., 2018; Lei et al., 2016]. Therefore, the problem of how to build the user visual profile for key frame recommendation when the fine-grained user-image behavior data is not available remains challenging.

Figure 1: Three application scenarios with the key frame presentation of a multimedia item.

In this paper, we study the general problem of personalized multimedia item and key frame recommendation without fine-grained user-image interaction data. The key idea of our proposed model is to distinguish each user's latent collaborative preference and the visual preference from her multimedia item interaction history, such that each user's visual dimensions could be transferred for visual based key frame recommendation. We design a Joint Item and key Frame Recommendation (JIFR) model to discern both the collaborative and visual dimensions of users, and model how users make item preference decisions from these two spaces. Finally, extensive experimental results on a dataset clearly show the effectiveness of our proposed model for the two personalized recommendation tasks.

2 ProblemDefinition

In a multimedia item recommender system, there are a userset U (|U| =M), and an itemset V (|V| =N). Each multimedia item v is composed of many frames, where each frame is an image that shows a part of the visual content of this item. Without confusion, we would use the term of frame and image interchangeably in this paper. E.g., in a movie recommender system, each movie is a multimedia item. There are many images that could describe the visual content of this movie, e.g., the official posters in different countries, the trailer posters, and the frames contained in this video. For video recommendation, it is hard to directly analyze the video content from frame to frame, a natural practice is to extract several typical frames that summarize the content of this video [Mundur et al., 2006]. Besides, for image-based advertising, for each advertising content with text descriptions, many related advertising images (frames) could be retrieved with text based image retrieval techniques [Wan et al., 2014].

Therefore, besides users and items, all the frames of the multimedia items in the itemset compose a frameset C (|C|= L). The relationships between items and frames are represented as a frame-item correlation matrix S ∈ RL×N . We use Sv = [j|sjv = 1] to denote the frames of a multimedia item v. For each item v, the key frame is the display image of this multimedia itemwhen it is presented or recommended to users. Therefore, the key frame belongs to v's frameset Sv. Besides, in the multimedia system, users usually show implicit feedbacks (e.g., watching a movie, clicking an advertisement) to items, which could be represented as a user-item rating matrix R ∈ RM×N . In this matrix, rai equals 1 if a shows preferences for item i, otherwise it equals 0.

Definition 1 [Multimedia Itemand Key Frame Recommendation] In a multimedia recommender system, there are three kinds of entities: a userset U, an itemset V, and a frameset C. The item multimedia content is represented as a frame-item correlation matrix S. Users show item preference with useritem implicit rating matrix R. For visual based multimedia recommendation, our goal is two-fold: 1) Multimedia Item Recommendation: Predict each user a's unknown preference ˆ rai to multimedia item i; 2) Key Frame Recommendation: For user a and the recommended multimedia item i, predict her unknown find-grained preference to multimedia content ˆlak , where k is a multimedia image content of i (ski = 1).

3 The ProposedModel

In this section, we introduce our proposed JIFR model for multimedia item and key frame recommendation. We start with the overall intuition, followed by the detailed model architecture and the model training process. At the end of this section, we analyze the proposed model.

With the absence of the fine-grained user behavior data, it is natural to leverage the user-item rating matrix R to learn each user's profile for key frame recommendation. Therefore, in each user's decisive process for multimedia items, we adopt a hybrid recommendation model that projects users and multimedia items into two spaces: a latent space to characterize the collaborative behavior, and a visual space that shows the visual dimensions that influences users' decisions. Let U ∈ Rd1×M and V ∈ Rd1×N denote the free user and item latent vectors in the collaborative space, and W ∈ Rd2×M and X ∈ Rd2×N are the visual representations of users and items in the visual dimensions. For the visual dimension construction, as each multimedia item i is composed of multiple images, its visual representation xi is summarized from the related visual embeddings of the corresponding frame set Si as:

(1)

where ck is the visual representation of image k. Due to the success of convolutional neural networks, similar as many visual modeling approaches, we use the last fully connected layer in VGG-19 to represent the visual content of each image k as ck ∈ R4096 [Simonyan and Zisserman, 2015; He and McAuley, 2016]. Pck transforms the original item visual content representation from 4096 dimensions into a low latent visual space, which is usually less than 100 dimensions.

Given the multimedia item representation, each user's predicted preference could be seen as a hybrid preference decision process that combines the collaborative filtering preference and the visual content preference as:

where wa is the visual embedding of user a from the user visual matrix W. In fact, by summarizing the item visual content from its related frames, the above equation is similar to the VBPR model that uncovers the visual and latent dimensions of users [He and McAuley, 2016].

With the implicit feedbacks of the rating matrix R, Bayesian Personalized Ranking (BPR) is widely used for modeling the pair-wise based optimization function [Rendle et al., 2009]:

where Θ1 = [U,V,W] is the parameter set, and σ(x) is a sigmoid function that transforms the output into range (0, 1). The training data for user a is DR a = {(i, j)|i ∈ Ra∧j ∈ V - Ra}, where Ra denotes the set of implicit positive feedbacks of a (i.e., rai=1), and j∈V -Ra is an unobserved feedback.

In key frame decision process, each key frame image presentation of the current item is the foremost visual impression to influence and persuade users. By borrowing the learned user visual representation matrix W from the user-item interaction behavior (Eq.(3)), each user's visual preference for image k is predicted as:

where wa is the visual latent embedding of user a learned from user-item interaction behavior. Please note that, the predicted ˆlak is only used in the test data for the key frame recommendation without any training process.

Under the above approach, for each user, by optimizing the user-item based loss function (Eq.(3)), we could align users and images in the visual space without any user-image interaction data for multimedia item and key frame recommendation. However, we argue that the above approach is not well designed for the proposed problem due to the overlook of the content indecisiveness and rating indecisiveness in the modeling process. The content indecisiveness is correlated to the visual representation of each item (Eq.(1)), which refers to which images are more important to reflect the visual content of the multimedia item are unknown. E.g., some frames in the movie convey more visual semantics than other frames that are not informative. Simply using an average operation that summarizes the item visual representation (Eq.(1)) would neglects the visual highlights of the item semantics. Besides, the rating indecisiveness appears in each user-item preference decision process as shown in Eq.(2), which refers to the implicitness of whether to concentrate more on the collaborative part or the visual item part for the preference decision process. For example, sometimes a user chooses a movie since this movie is visually stunning, even though this movie does not follow her historical watching histories. Therefore, the recommendation performance is limited by the assumption that the hybrid preference is equally contributed by the collaborative and visual content based models as shown in Eq.(2).

3.1 The Proposed JIFRModel

In this part, we illustrate our proposed Joint Item and key Frame Recommendation (JIFR) model, with the architecture is show in Fig. 2. The key idea of JIFR are two carefully designed attention networks for dealing with the content indecisiveness and rating indecisiveness. Specifically, to tackle the content indecisiveness, the visual content attention attentively learns the visual representation xi of each item. By taking the user-item predicted collaborative rating uTa vi, and the visual content based rating wT a xi, the rating attention module learns to attentively combine these two kinds of predictions for rating indecisiveness problem.

Visual Content Attention. For each multimedia item i, the goal of the visual attention is to select the frames from the frameset Si that are representative for item visual representation. Therefore, instead of simply averaging all the related images' visual embeddings as the item visual dimension (Eq.(1)), we model the item visual embedding as:

where sji = 1 if image j belongs to the image element of multimedia item i. αij denotes the attentive weight of image j for multimedia item i. The larger the value of αij , the more the current visual frame contributes to the item visual content representation.

Since αij is not explicitly given, we model the attentive weight as a three-layered attention neural network:

where Θc = [w1 ,W1 ,W0 ] is the parameter set in this three layered attention network, and f(x) is a non-linear activation function. Specifically, W0 is a dimension reduction matrix that transforms the original visual embeddings (i.e., cj ∈ R4096 ) in a low dimensional visual space.

Then, the final attentive upload history score αji is obtained by normalizing the above attention scores as:

Hybrid Rating Attention. The hybrid rating attention part models the importance of the collaborative preference and the content based preference for users' final decision making as:

where the first part models the collaborative predicted rating, and the second part denotes the user's visual preference for items. βai1 and βai2 are the weights that balance these two effects for the user's final preference.

As the underlying reasons for users to balance these two aspects are unobservable, we model the hybrid rating attention as:

Then, the final rating attention values βai1 and βai2 are also normalized as:

Model Learning and Prediction

With the two proposed attention networks, the optimization function is the same as Eq. (3). To optimize the objective function, we implement the model in TensorFlow [Abadi et al., 2016] to train model parameters with mini-batch Adam, which is a stochastic gradient descent based optimization model with adaptive learning rates. For the user-item interaction behavior, we could only observe the positive feedbacks with huge missing ratings. In practice, similar as many implicit feedback based optimization approaches, in each iteration, for each positive feedback, we randomly sample 10 missing feedbacks as pseudo negative feedbacks in the training process [Chen et al., 2017a; Wu et al., 2017]. As in each iteration, the pseudo negative feedbacks change with each missing record gives very weak negative signal.

After the model learning process, the multimedia recommendation could be directly computed as in Eq.(8). For each recommended item, as users and images are also learned in the visual dimensions, the key frame recommendation could be predicted as in Eq.(4).

3.2 Connections to RelatedModels

VBPR [He and McAuley, 2016] extends the classical collaborative filtering model with the additional visual dimensions of users and items (Eq.(4)).By assigning the same weight for all of an items' frames as shown in Eq (1), and without any hybrid rating attention, our proposed item recommendation task degenerates to VBPR.

ACF [Chen et al., 2017a] is proposed to combine each user's historical rated items and the item components with attentive modeling on top the CF model of SVD++ [Koren et al., 2009]. ACF did not explicitly model users in the visual space, and could not be transferred for visual key frame recommendation.

VPOI [Wang et al., 2017] is proposed for POI recommendation by leveraging the uploaded images of users at a particular POI. In VPOI, each item has a free embedding, and the relationship between images and POIs are used as side information in the regularization terms.

KFR [Chen et al., 2017b] is the one of the first attempts for personalized key frame recommendation. As KFR relied on users' fine-grained interaction data with frames, it fails when the user-frame interaction data is not available. Besides, KFR is not designed for item recommendation at the same time.

Attention Mechanism is also closely related to our modeling techniques. Attention has been widely used in recommendation, such as the importance of historical items in CF models [He et al., 2018; Chen et al., 2017b], the helpfulness of review for recommendation [Chen et al., 2018; Chen et al., 2017b; Cheng et al., 2018], the strength of social influence for social recommendation [Sun et al., 2018]. In particular, ACCM is an attention based hybrid item recommendation model. It also fuses users and items in the collaborative space and the content space for recommendation [Shi et al., 2018]. However, the content representation of users and items rely on the features of both users and items. As it is sometimes hard to collect user profiles, this model could not be applied when users do not have any features, including our proposed problem.

In summary, our proposed model differs greatly from these related models as we perform both joint personalized item and key frame recommendation at the same time. The application scenario has rarely been studied before. From the technical perspective, we carefully design two levels of attentions for dealing with content indecisiveness and rating indecisiveness, which is tailored to discern the visual profiles of users for joint item and key frame recommendation.

4 Experiments

We conduct experiments on a real-world dataset. To the best of our knowledge, there is no public available datasets with fine-grained user behavior data for evaluating the key frame recommendation performance. To this end, we crawl a large dataset from Douban 1 , which is one of the most popular movie sharing websites in China. We crawl this dataset as for each movie, this platform allows users to show their preference to each frame of this movie by clicking the "Like" button just below each frame.

After data crawling, we pre-process the data to ensure each user and each item have at least 5 rating records. In data splitting process, we randomly select 70% user-movie ratings for training, 10% for validation and 20% for test. The pruned dataset has 16,166 users, 12,811 movies, 379,767 training movie ratings, 98,425 test movie ratings, and 4,760 test frame ratings. For each user-item record in the test data, if the user has rated the images of this multimedia item, the correlated user-image records are used for evaluating the key frame recommendation performance in the test data. Please note that, to make the proposed model general to the multimedia recommendation scenarios, the fine-grained image ratings in the training data is not used for model learning.

4.1 Overall Performance

We adopt two widely used evaluation metrics to measure the top-K ranking performance: the Hit Ratio (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K) [He and McAuley, 2016; Sun et al., 2018; Chen et al., 2017a]. In our proposed JIFR model, we choose the collaborative latent dimension size d1 and the visual dimension size d2 in the set [16,32,64], and find when d1 = d2 = 32 reaches the best performance. The non-linear activation function f(x) in the attention networks is set as the ReLU function. Besides, the regularization parameter is set in range [0.0001, 0.001, 0.01, 01], and λ1 = 0.001 reaches the best performance.

1 www.douban.com

For better illustration, we summarize the details of the comparison models in Table 1, where the second and third column shows the input data of each model, and the last two columns show whether this model could be used for item recommendation and key frame recommendation. The last two rows are our proposed models, with JIFR NA is a simplified version of our proposed model without any attention modeling.

Item Recommendation Performance

In the item recommendation evaluation process, as the multimedia item size is large, for each user, we randomly select 1000 unrated items as negative samples, and then mix them with the positive feedbacks to select the top-K ranking items. To avoid bias at each time, the process is repeated for 10 times and we report the average results. Fig. 3 shows the results with different top-K values. In this figure, some top-K results for CDL are not shown as the performance is lower than the smallest range values of the y-axis. This is because CDL is a visual content based model without any collaborative filtering modeling, while the collaborative signals are very important for enhancing recommendation performance. Please note that, for item recommendation, our simplified model JIFR NA degenerates to VBPR without any attentive modeling. VPOI and VBPR improves over BPR by leveraging the visual data. ACF further improves the performance by learning the attentive weights in the user's history. Our proposed JIFR model achieves the best performance by explicitly modeling users and items in both the latent space and the visual space with two attention networks. For the two metrics, the improvement of NDCG is larger than HR as NDCG considers the ranking position of the hit items.

Key Frame Recommendation Performance

For key frame recommendation, as the detailed user-image information is not available, the models that relied on the collaborative information fails, including BPR, VBPR, KFR [Chen et al., 2017b]. We also show a simplified RND baseline that randomly selects a frame from the movie frames for evaluation. In the evaluation process, all the related frames of this movie is used as the candidate key frames. Please note that, the ranking list size K for key frame recommendation is very small as we could only recommend one key frame of each multimedia item, while the item recommendation list could be larger. As observed in Fig 4, all models improve over RND, showing the effectiveness of modeling user visual profile. Among all models, our proposed JIFR model shows the best performance, followed by the simplified JIFR NA model. This clearly shows the effectiveness of discerning the visual profiles and the collaborative interests of users with attentive modeling techniques.

4.2 DetailedModel Analysis

Attention Analysis

Table 2 shows the performance improvement of different attention networks compared to the average setting, i.e., αji = 1 ∑Ck=1 ski for content attention modeling, and βai1 = βai2 for rating fusion. The ranking list value K is set as K = 15 for item recommendation and K = 3 for key frame recommendation. As can be observed from this table, both attention techniques improve the performance of the two recommendation tasks. On average, the visual attention improvement is larger than the rating attention. By combining these two attention networks, the two recommendation tasks achieve the best performance.

Frame Recommendation Case Study

Fig 5 shows the a case study of the recommended frames for user a with the movie Interstellar. It is regarded as a sci-fi that describes a team explorers travel through a wormhole in space to ensure humanity's survival. In the meantime, the love between the leading actor, and his daughter also touches the audiences. For ease of understanding, we list the training data of this user in the last column, with each movie is represented with an official poster. In the training data, the liked movies contain both sci-fi and love categories. Our proposed JIFR model could correctly recommend the key frame, which is different from the official poster of this movie. However, the remaining models fail. We guess a possible reason is that, as shown in the four column, most frames of this movie are correlated to sci-fi. As the comparison models could not distinguish the important of these frames, the love related visual frame is overwhelmed by the sci-fi visual frames. Our model tackles this problem by learning the attentive frame weights for item visual representation, and user visual representation from her historical movies.

5 Conclusions

In this paper, we studied the general problem of personalized multimedia item and key frame recommendation in the absence of fine-grained user behavior. We proposed a JIFR model to project both users and items into a latent collaborative space and a visual space. Two attention networks are designed to tackle the content indecisiveness and rating indecisiveness for better discerning the visual profiles of users. Finally, extensive experimental results on a real-world dataset clearly showed the effectiveness of our proposed model for the two recommendation tasks.

Acknowledgments

This work was supported in part by grants from the National Natural Science Foundation of China( Grant No. 61725203, 61722204, 61602147, 61732008, 61632007), the Anhui Provincial Natural Science Foundation(Grant No. 1708085QF155), and the Fundamental Research Funds for the Central Universities(Grant No. JZ2018HGTB0230).