Collaborative and Attentive Learning for Personalized Image Aesthetic Assessment

1 Introduction

With the continuously generated and ever-expanding volume of visual images, automatic image aesthetics assessment is increasingly important in many applications e.g. image retrieval and editing, content management and photography [Datta et al., 2007; Marchesotti et al., 2015; Lu et al., 2015b]. Among various vision problems, there is a particular challenge to assess an image as 'good' or 'bad', due to the highly subjective and complex nature of human aesthetic preference.

Many research efforts have been made to address the computational image aesthetic quality assessment problem. A major line of research formulates the image aesthetic prediction as classification or regression problem that map the image to users' ratings [Datta et al., 2006; Nishiyama et al., 2011] etc. Among them, early attempts are based on the intuition of how an image is perceived, which design hand-crafted features by following the standard photographic rules of visual design such as the rule of thirds, the golden ratio, and color harmonies[Tong et al., 2004; Ke et al., 2006; Dhar et al., 2011; Bhattacharya et al., 2010; Zhang et al., 2013]. Despite the success of handcrafted features, recent work show that deep convolutional neural network (CNN) feature based methods [Kang et al., 2014; Lu et al., 2015b; Dong et al., 2015; Tian et al., 2015; Lu et al., 2015a] can yield leading performance on public benchmarks.

∗ indicates the corresponding author

One key complication of aesthetic analysis compared with other vision tasks is that different people may have diverse conclusions on the same image based on their subjective preferences. In [Segalin et al., 2016], pictures are mapped into attributed human traits defined as the Big-Five factor structure [Goldberg, 1990]. However, human traits refer to common group characteristics rather than individual ones. So far little personalization technique has been introduced to image aesthetic assessment problem, which we believe is the essence of this problem and plays as the key motivation of this paper.

Such limitation also exists at existing public image aesthetic assessment datasets, including the popular Aesthetic Visual Analysis (AVA) dataset [Murray et al., 2012] and the Aesthetics and Attributes Database (AADB) dataset [Kong et al., 2016]. In fact, these datasets only contain visual images without any textual reviews that can semantically reflect users' preferences. Moreover, the used ground truth for performance evaluation is often set as the average of individual users' ratings of an image, making the evaluation protocol inherently personalization-free.

We enrich the AVA benchmark by crawling its user-specific textual reviews and derive the user-specific ratings to each image. It is important to note that review comments can reflect users' personalized tastes. As such, we re-create a reviewaugmented version of AVA with multi-modal data (visual image and textual review), as well as derived personalized opinions. Note that [Zhang, 2016] developed a deep learning architecture for image aesthetic assessment, where image semantic tags were used in addition to image content. However, no (unstructured) text data was involved.

In this work, we propose a novel personalized multi-task image aesthetic assessment approach. User textual reviews, visual content of images and their semantic tags are simultaneously modeled based on the above motivation and intuition. We identify the contributions as follows:

i) we address the personalized user-specific image aesthetic assessment learning task. To our best knowledge, this is the first work for formally formulating this problem. We also create the review-augmented image aesthetic assessment benchmark based on the Aesthetic Visual Analysis (AVA) dataset [Murray et al., 2012].

ii) We present a novel network for integrating multi-modal information including image, review, and user-image relation for (personalized) image aesthetic assessment learning. In particular, attentive mechanism is devised for capturing users' personalized taste (reflected by their reviews) to an image w.r.t. the area of interest as well as the semantic tag of the images. The attention is fulfilled by the joint modeling of images and user-specific reviews.

iii) In addition with the new capability of predicting personalized user-image ratings, we also achieve the state-ofthe-art performance on traditional user-agnostic image aesthetic assessment benchmark.

2 Problem Settings

Review-augmented AVA As discussed above, we enrich the AVA dataset [Murray et al., 2012] by re-collecting the userspecific reviews for an image which is tagged with its semantic tags such as family, landscape etc. There are in total 66 semantic tags in AVA and in total over 250,000 images.

For each image, the data source of the AVA dataset, i.e. the website (http://www.dpchallenge.com/) in fact contains both user-image ratings and user-image text reviews. For each review, the user id is available. However, for the raw rating score (with 10), its user information is de-identified and anonymous thus it is only possible to obtain the rating distribution but unable to link each raw rating to a corresponding user id (and its reviews). As a tentative effort, for each user's review, we try to re-generate users' ratings to the image and assign a binary label ('good' or 'bad'), if the number of positive words is larger than the negative ones in the review to certain extent. Fig. 2 shows two image examples with their rating distributions: one is from de-identified raw score, the other is from the re-generated user-specific score from their review by our approach. In fact, we are able to reach 81% consistency between the raw ratings and the derived ratings from reviews for each image in average - see more details in the experiment section about the comparison protocol.

7.927

7.913

35.208

[1, 85]

[1, 87]

Table 1 discloses its preliminary statistics: for each image, at least there is one review text. The overview of the problem and working pipeline of our method is sketched in Fig. 1, which highlights the difference from existing problem setting: reviews are collected as extra inputs and user personalized ratings are additional outputs.

Problem & Notations Specifically, we integrate both visual image and associate user-specific reviews into a deep network for personalized image aesthetic assessment. Attention model is developed to attend to both particular image areas and visual attributes, to capture the users' personalized taste. First we introduce notations.

Denote users as u = {u1, u2, · · · , um}, and images as v = {v1, v2, · · · , vn}, the binary user-image matrix is defined where 1-entry means the image is reviewed by a user and 0 otherwise. User u's review on image v is defined as ruv with binary aesthetic rating suv ('good' or 'bad'). The word list in ruv is defined as wuv = {w0uv ,w1uv , · · · ,wLuv-1 uv }, where Luv is the length of the review text. The attention map which means the focus of user u on image v is denoted by auv. The original visual features of image v is defined as fv and the user specific one is fuv.

Given all user specific visual features F = {fuv|u ∈ u, v ∈ v}, reviews W = {wuv|u ∈ u, v ∈ v}, ratings S = {suv|u ∈ u, v ∈ v}, and image tag (e.g. portrait, landscape) T = {t1, t2, · · · , tl}, for a target user u and an unseen image vo ∈ v, the task is to find a function g(suvo |u,W, S,F, T ) to predict if user u would rate 'good'/'low' to vo where suvo is the predicted rating.

3 ProposedModels and Algorithms

In this section, we will introduce our network which involves standard building blocks like CNN (VGG-16) [Simonyan and Zisserman, 2014], multi-task loss [Caruana, 1997; Kao et al., 2017]. In particular, the personalization is fulfilled by a collaborative filtering network component that infuses the user and image relation information. Using this model as a starting point, we further add users' textual review data and correspondingly develop an attentive mechanism to bridge the visual images and textual reviews. Importantly, this mechanism can capture the particular region of interest in an image as well as the latent factors of semantic tags individual for rating an image.

3.1 Personalized Collaborative Filtering Attentive Aesthetic Assessment Network

As shown in Fig. 3, for image v reviewed by user u, we first embed its index in set v (by one-hot encoding vector) into an embedding layer with output qv. Then qv is combined with the image visual content feature fv (weighted byWi ) in to a merger operation by:

where Merge(·, ·) : Rd ×Rd → Rd is a function that merges two d dimension vectors into one. Element-wise multiplication is used in this paper while other mappings can be used depending on specific applications.

The latent variable qiv is combined with the user-wise index embedding γu to generate the binary prediction ˆ yuv :

where f(·, ·) is interaction function learned through hidden layers.

For the CNN structure used for image content feature fv extraction, VGG-16 [Simonyan and Zisserman, 2014] pretrained on the 1000-class ImageNet classification challenge 2012 dataset [Deng et al., 2009] is adopted (by removing its fully connected layers). Then we use bidirectional GRU [Bahdanau et al., 2014] model to combine the word embedding with the CNN visual features. ---→ GRU reads the review ruv fromw0uv to wluv-1 uv while ←--- GRU reads reversely.

As a result the annotation for word wiuv can be concatenated of forward hidden state -→h iuv and backward one ←- h iuv :

The annotation huv is projected to visual space from contextual information as the convolutional kernel K by K = σ(Wkhuv + bk), highlighting the visual features focused by users (reflected by their reviews). K has the same number of channels as the visual feature fv. The review-image attention map is calculated by:

where σ(x) = 1 1+e-x , and fuv = K∗fv is the convolution of the kernel K with fv. Note aij uv is the element of the attention map at position (i, j), and auv has the same size as fv, It characterizes the attention distribution across feature map.

We conjecture the attention framework can be useful in two aspects: 1) It can locate subjects in the photo and features in less relevant regions would be filtered out; 2) It can also focus on personalized semantic patterns (e.g. colorfulness, lighting) of a specific user. Fig. 4 shows such diversity for attention maps among different users. Note the attention map is user-specific.

Finally a multi-task learning part is added to the attentive network. The hidden layers and convolutional layers in this network can be regarded as shared layers, the added fullyconnected layers in multi-task learning part are task-specific layers. The final loss function is:

where λz and λβ are the weighting hyper-parameters, ˆ yuv is the prediction of user-specific rating task, ˆ zuv is the prediction of semantic tag classification task, and ˆ βuv is the prediction of user-agnostic rating task.

4 Experimental Results and Discussion

We evaluate the proposed method on one of the most largescale and challenging datasets, i.e. AVA dataset for visual aesthetic quality assessment (augmented by users' reviews). It contains more than 255,000 images gathered from www.dpchallenge.com, with each image tagged by 0 to 2 semantic tags. There are around 200 users giving a rating score ranging from 1 to 10 for each image.

We follow the protocol by the state-of-the-art work [Kao et al., 2017] to identify 29 major semantic tags (e.g. abstract, fashion, family, sky, sports) with 185,751 images used for our evaluation: i) the chosen tag has at least 3000 images; ii) the chosen image has at least one tag. If an image has two tags, the primary tag will be set as its tag in line with the protocol used by the state-of-the-art [Kao et al., 2017].

4.1 Protocol &Data Preparation

Review Augmentation

We try to recover the user-specific ratings which are missing in the AVA dataset. The main idea is to explore the text reviews to derive the underlying ratings. There are some challenges in extracting semantic ratings from textual reviews which consist of many short sentences. Moreover, the reviews may not well follow the grammar in written form. We design the following empirical protocol to recover user-image ratings, as illustrated in Fig. 5.

• Review classification To identify useful reviews, we first classify reviews into four types by the tool CoreNLP [Manning et al., 2014]: I) comments based on the photographic rules like "The color is wonderful and the composition is great!". II) users' subjective feelings like "I like this photo!". III) ambiguous not literally related to image aesthetic quality like "How handsome the boy is!" IV) only contain equivocal messages like "Do you live in the steeple?" The part-of-speech tagger, the parser and the open information extraction tools in CoreNLP are used to analyze the grammatical structures of sentences and extract relation tuples of words.
• Review cleaning The reviews in the type II are rewritten as "The photo is good (resp. bad)" and those in the type III and type IV are removed due to their inherent ambiguity.
• Review sentiment counting The high-frequency words are sorted and those with the highest relevance to photo aesthetic quality assessment (e.g. shot, composition, focus, light, angle, color, exposure) are selected. Then positive and negative words in reviews are counted by the sentiment analysis toolkit in CoreNLP, by which the reviews are divided into five levels of user's preference.
• Rating generation Since traditionally existing image assessment work almost use binary rating as ground truth [Murray et al., 2012], we derive the binary ratings suv ∈ {0, 1} for each review-to-image by setting the first two levels as 'bad', and the other three as 'good'.

Validity of Recovered User-specific Ratings

We perform a cross-check to verify the reliability of our recovered binary rating s by indirectly comparing them with the raw de-identified ratings φ from 1 to 10 as recorded in the AVA dataset. Note that there is no exact one-to-one matching between the review-recovered ratings and the raw ratings because sometimes a user may write a review with no rating and vice versa. Hence we try to compute the positive vs. negative ratio consistency between φ and s. For each image v, we first compute its overall binary rating, i.e. user-agnostic score bv - this is also the tradition widely adopted in the AVA dataset related studies [Murray et al., 2012]: let bv = 1 if the average of image v's K ratings {φv}Kk=1 is larger than 5 otherwise bv = 0. Then we compute the ratio of the 'good' samples against 'bad' ones for the recovered binary scores {sv}. More specifically the consistency ratio score is defined as follows for each image (#(·) indicates the number):

By dividing the above number to the total number of image, we reach a consistency of 81.06% which we believe shows our recovered ratings are reasonable. As such, the review dataset contains 1,163,258 positive review derived ratings (suv = 1) and 271,862 negative ones (suv = 0).

Experiment Settings

We use Fig. 6 to illustrate the detailed setting of our model. More detailed description can be found in the caption. The input raw images are resized to 320×320, and the feature map is reduced into 10×10×512 after five pooling layers in VGG16. The dimension of the attention-map is 10×10×512 which is the same as the feature map. The merging of attention-map and image feature is through transform matrix according to a normal distribution. The user encoding and image embedding have the same dimension and after element-wise product the result can be used to predict the aesthetic ratings for different users. In all of our networks, the VGG-16 part is initialized by pretraining on ImageNet, and the resting parameters are randomly initialized according to a standard normal distribution N(0, I), and then updated by executing stochastic gradient descent (SGD). The hyper-parameters in our models are tuned by conducting 10-fold cross validation on the training set. We set 90% of the data as training set, and the rest is testing set. Specifically, the weighting parameter λβ and λz in Eq. 6 is set to balance the contribution of semantic classification loss and aesthetic rating loss. We set the original learning rate as 0.005, the decay rate as 0.99, the decay step as 1000. k is set as 50, λβ is set as 1, and λz is set as 0.015.

4.2 Results &Discussion

User-agnostic Aesthetic Assessment

In the previous study, image aesthetic assessment is performed at the aggregated level, including the state-ofthe-art MTRLCNN (Multi-Task Relationship Learning Convolutional Neural Network) [Kao et al., 2017]. For a fair comparison, we use the same metric, i.e. binary classification accuracy on the rating to evaluate the performance of MTRLCNN and our proposed models.

The accuracy on the major 29 image semantic tags is illustrated in Fig. 7, which involves three compared methods: 1) CFNpure-2 task: pure CF based network without using any image or text review as input. Two tasks, i.e. image assessment prediction and image tag classification are jointly learned; 2) MTRLCNN: the state-of-the-art user-agnostic assessment approach which is a multi-task learning based network using only visual image as input; 3) our approach CFAN-3 task as depicted in Fig. 3 which uses CF and attention mechanism to fuse both text and image data. In general, our method outperforms in most tags, which verifies the efficacy of our model to traditional user-agnostic aesthetic assessment task. Note that our model meanwhile learns the user-specific assessment task.

Personalized Aesthetic Assessment

For comparison of our approach CFAN-3 task for personalized prediction, AUC score for ROC [Bradley, 1997] is compared. Since there are few tailored personalized models for image aesthetic assessment, we compare two recently proposed general personalized prediction.

BPR (Bayesian Personalized Ranking) [Rendle et al., 2012]: is a well-known personalized recommendation algorithm to model user implicit feedback (e.g. clicks, purchases) without visual features. In the experiment, for each positive feedback, a negative sample is randomly generated.

VBPR (Visual Bayesian Personalized Ranking) [He and Mcauley, 2016]: is an Bayesian Personalized Ranking method which models raw visual features for item recommendation. In the test, visual features are also extracted by the same deep features [Simonyan and Zisserman, 2014].

We also test degraded versions of our model as follows: 1) CFN-1 task: the input are user factors, image factors and image raw data. Only collaborative filter layers are included;

2) CFAN-1 task: the review is added as another modal of input and a RNN model is used to extract attention map; 3) CFAN-2 task: the network which jointly performs personalized assessment task and semantic tag classification task. For CFN-1 task and CFAN-1 task, no multi-task is involved and the loss focuses on the user-specific prediction.

Fig. 8 shows the averaged performance among various tags of images. Note our final model as depicted in Fig. 3 involves three loss, so it is termed by CFAN-3 task in the plot. We make the following observations and analyses:

• The basic method BPR only considers user and image factors. It performs worst because of the limited input information. VBPR and CFN-1 task have the same input but the latter performs better in binary classification problem. CFAN-1 task improves the performance by nearly 10% with the help of additional text/image information. Our main network CFAN-3 task is superior against other methods and the result shows that different tasks can synergetically contribute to each other.
• No surprisingly, the network with attentive mechanism, i.e. CFAN-1 task outperforms the CFN-1 task by a notable margin. It can be explained that attention map extracted from user review can reflect users' focuses. Features in less relevant regions would be filtered out. Fig. 4 shows the feature maps extracted by different users. In CFN-1 task network, the connection between user and image is based on the combination of the factors. In CFAN-1 task network, that is further based on the user focused image visual features.

As one of our main contributions is modeling the user reviews, we also test the performance of our methods changing reviews in each image. Fig. 9 shows the performance for CFN-1 task and CFAN-1 task learning user personalized assessment. The accuracy and AUC have minor fluctuations within 10%. Though reviews are not modeled in CFN-1 task, the result can confirm that the number of reviews makes no significant influence on the results of aesthetic rating. A special case is that when we only use reviews without image nor rating relation input, the personalized task accuracy is 0.18. In fact, the modeling of personalized information involves user ID, image ID, review, and image. The reviews contribute to but not dominate the results.

5 Conclusion

In this work, to the best of our knowledge, we are the first to predict personalized user specific rating of image aesthetics by training a neural network from images and texts. Our framework models visual features, semantic tags, and user reviews, leading to increasing accuracy of personalized image aesthetic rating. In the framework, the collaborative filtering and RNN network are combined to generate user specified visual features with attention maps. The aesthetic assessment tasks (i.e. user-specified and user agnostic) and semantic classification task are performed simultaneously. Empirical study shows state-of-the-art performance of our approach. In the future, we aim at using this research as a first step to perform explainable image aesthetic assessment.