FolkPopularityRank: Tag Recommendation for Enhancing Social Popularity using Text Tags in Content Sharing Services

1 Introduction

In image and video sharing services such as Flickr 1 , Instagram 2 , etc., the numbers of views, comments, and favorites (likes) are shown to represent its popularity. Hereafter, we refer to such numbers as "social popularity scores" or "social popularity," aligned with the definition in [Yamasaki et al.2014]. It is observed that users often care about by their social popularity scores in social networks. Users even sometimes do self-promotion to gain positive sentiments (e.g., asking their friends to show "favorite" or "like" for them), not to mention the companies which can leverage social media services for promotion. Therefore, in social media networks, it has been a critical issue for both individuals and enterprises to obtain high social-popularity scores since such scores directly reflect how much attention has been paid to certain contents.

1 https://www.flickr.com/

2 https://instagram.com/

Though one might think that quality of the image and video content is one of the most important factors in receiving higher social popularity scores in social networks, it has been shown that the image and video feature does not contribute much in predicting social popularity scores [Yamaguchi et al.2014, Gelli et al.2015, Hu et al.2016]. Text tags, on the other hand, play a very important role, because users typically locate content via text searches. Therefore, predicting social popularity scores using text tags along with other features has been studied in [Pedro and Siersdorfer2009, van Zwol et al.2010,Yamasaki et al.2014]. The review on such prediction of social popularity scores is summarized in Sec. 1.1

It is worthwhile to question whether users have sufficient knowledge to annotate their content with tags that are capable of boosting social popularity scores. In many cases, they lack this knowledge. Users often do not know which tags are more influential, and they cannot predict social popularity scores based only on tags. We conducted a preliminary experiment by uploading three identical photos to three different new accounts in Flickr, as shown in Fig. 1. The first uploaded photo included the original tags (lisbon, portugal, homeless, smoking, bw, thecontinuum, 1on1people, and myphotobook); the second included the top two tags in terms of tag frequency ranking (bw and portugal), and the third included the top two recommended tags from our system (bw and smoking). Surprisingly, four days after the upload, the numbers of views were 29, 8, and 30, respectively. The image with the recommended tags earned 3.8 times more views than the tag set #2. This experiment clearly shows that even though the images are identical, their social popularity scores are strongly affected by the tags.

In this paper, we propose a new algorithm called FolkPopularityRank (or FP-Rank). FP-Rank is inspired by PageRank [Brin and Page1998, Page et al.1999], the well-known web page ranking algorithm, and FolkRank [Hotho et al.2006b,Hotho et al.2006a], PageRank's folksonomy-based descendant that also considers users, resources, and tags. The contributions of this paper is that FP-Rank can recommend tags that enhance social popularity scores. The recommendation is done based on existing tags, and therefore the recommended tags are related to the content of the image/video. Although numerous tag recommendation systems have already been proposed (please refer to [Li et al.2015b] for reviews), they are focused only on tags that are semantically correct for

*Tag set #1:

lisbon, portugal, homeless, smoking, bw, thecontinuum, 1on1people, and myphotobook

*Tag set #2:

bw and portugal

*Tag set #3: smoking and portugal

# of views

of 4 days later

29

8

30

Figure 1: Preliminary results demonstrating the importance of proper tagging

describing the content. To the best of our knowledge, tag recommendations designed to increase social popularity scores is firstly presented.

1.1 RelatedWorks

PageRank and FolkRank

Our algorithm is inspired by the PageRank [Brin and Page1998,Page et al.1999] and FolkRank [Hotho et al.2006b, Hotho et al.2006a] algorithms. PageRank was developed to rank web pages. The basic concept of PageRank consists of three assumptions: (1) web pages linked to by many other pages are important, (2) web pages linked to by important pages are also important, and (3) if a page contains links to many other pages, the importance of being linked by the page is low. Thus, PageRank regards web pages as a directed graph, in which edge weights are the importance score of the source page divided by the number of outgoing links. For computational efficiency, the scores rP are calculated iteratively by the following equation:

where rP is the importance score vector and AP is the rowstochastic version of the adjacency matrix of the directed graph GP = (VP ,EP ). α is a dumping factor that can consider random jumps to other pages, and p is the random surfer component.

FolkRank is an extension of PageRank, and considers users, resources, and tags. FolkRank is constructed with an undirected tripartite graph GF = (VF ,EF ), where VF = U ˙∪R ˙∪T. U,R, and T represent users, resources, and tags, respectively. Edge weights represent all the co-occurrences of tags and users, users and resources, and resources and tags: EF = {{t, u}, {u, r}, {r, t}}. The scores rF can be calculated by the following equation.

where AF is the row-stochastic version of the adjacency matrix GF . It is assumed that resources tagged with important tags (and annotated by important users) are important; this assumption applies symmetrically to users, resources, and tags.

As already noted in [Brin and Page1998], the random surfer component p can be used as a personalized ranking. Further, tag recommendations based on already annotated tags are calculated by the FolkRank algorithm, using the following equation.

However, the recommended tags are only co-occurrencebased, which does not consider the effect to the social popularity at all.

Tag Ranking

One of the main purposes of tag ranking in social media is to extract influential tags for (1) classification of the content or (2) regression of the scores, in terms of attractiveness or popularity. In [Pedro and Siersdorfer2009], top terms for attractive and unattractive photos in Flickr were extracted according to mutual information values. It was shown that a regression model using the extracted tags can achieve a correlation value of 0.36 for the predicted and actual number of favorites in Flickr. In addition, visual features such as brightness and contrast were introduced. The combination of text and visual features achieved a correlation value of 0.48. In [Yamasaki et al.2014], a tf-idf-like tag ranking was proposed using the tag frequency and weights for the tags, which were obtained from a regression model.

Social Popularity Prediction

Predicting how much popularity the content would get is another important research topic [van Zwol et al.2010, Khosla et al.2014,McParlane et al.2014,Totti et al.2014,Cappallo et al.2015, Gelli et al.2015, Chen et al.2016b,Wu et al.2016]. In [van Zwol et al.2010], social connections, in addition to textual and visual information, were also utilized to predict which content would earn a higher number of favorites. [Khosla et al.2014] showed that social context is very important to predict the social popularity and they achieved the correlation value of 0.81. [Totti et al.2014] investigated the impact of the visual attributes and showed that visual properties have low predictive power compared that of social cues. In [McParlane et al.2014], an image's context, visual appearance, and user context were considered. [Gelli et al.2015] introduced visual sentiment features in addition to the above mentioned features. In most papers, context data such as tags were most dominant.

The disadvantage of using visual features is their computational complexity. Because billions of photos and videos exist on content sharing services, extracting visual features for all of them is not practical. It has also been demonstrated that visual features do not contribute much in some content sharing services [Yamaguchi et al.2014,Gelli et al.2015,Hu et al.2016]. The use of social connections suffers from the dynamic nature of the connections themselves. In addition, social connections cannot be used to analyze the influence of new users who do not yet have connections; this situation is known as the cold start problem. It is also difficult to control the social connections: even if the number of social connections with influential persons is important for higher social popularity, for instance, it is not easy to add such connections.

Tag Recommendation and Refinement

Tag recommendation and refinement in social network help users to annotate more tags with less effort, and to consolidate vocabulary across users.

One of the most prominent and frequently used recommendation techniques is collaborative filtering (CF). Because there are so many CF-based methods, please refer to the review papers [Su and Khoshgoftaar2009, Lops et al.2013] for details.

In Tagcoor [Sigurbj¨ornsson and van Zwol2008], candidate tags were extracted by evaluating their co-occurrences; they were sorted using a combination of three measures: stability, descriptiveness, and user-based tag frequency ranking.

FolkRank [Hotho et al.2006b, Hotho et al.2006a] can also be used for tag recommendations, as summarized in Sec. 1.1. Numerous techniques have been proposed to improve FolkRank. [Jschke et al.2007] compared several approaches, including adapted PageRank, FolkRank, CFs, and co-occurrence-based rankings; they demonstrated that FolkRank provides the best performance. In [Si et al.2009], the most popular tags and FolkRank were combined. [Gemmell et al.2009] and [Zhang et al.2009] separately proposed hybrid recommendation models in which tag rankings were calculated by the weighted sum of FolkRank scores and item-based CF scores. [Landia et al.2012] proposed ContentFolkRank by combining FolkRank with textual content to solve the cold start problem.

A comprehensive survey can be found in [Li et al.2015b]. About 60 approaches are summarized and 11 of them are reimplemented by the authors and evaluated using the same dataset. We would like to emphasize that the authors in [Li et al.2015b] compare the performance only in terms of average precision (AP). In other words, the tag recommendation and retrieval is regarded as an information retrieval problem.

Tag Generation and Scene Description

Object-recognition-based sentence generation systems [Kulkarni et al.2013,Han et al.2014,Chen et al.2016a] can be regarded as automatic tag generators. Although the descriptions are semantically correct, the sentences and descriptions are nothing more than objective facts. Therefore, these systems are useful for content retrieval by text search; however, they do not consider factors that would increase social popularity when generating descriptions.

Attractiveness/Emotion Prediction

Human emotion-based attractiveness of content has been investigated over the past decade. Sentiment classification and affective analysis have been conducted for texts [Pang et al.2002, Pang and Lee2008], images [Lu et al.2014, Li et al.2015a, Peng et al.2015, Zhao et al.2016], and videos [Irie et al.2010]. Aesthetic analysis of the content using image features [Bhattacharya et al.2010,Murray et al.2012,Zhang et al.2014,Lu et al.2014,Aydin et al.2015,Park and Zhang2015] has also been a frequently discussed topic in the community. However, we would like to point out that these papers did not discuss how such emotion-based attractiveness would affect the social popularity.

2 Folk Popularity Rank

2.1 Tag Ranking Algorithm

We would like to emphasize that conventional tag ranking and recommendation systems in social media are designed to add semantically proper tags. Namely, some of the tags in the original content were artificially deleted, and the recommender systems attempted to predict as many deleted tags as possible while suppressing the number of unrelated tags.

We propose an algorithm that can extract tags with a higher level of influence on social popularity scores. As a result, we can recommend tags that enhance social popularity. It is important to note that visual and social features can also be employed, as in [Pedro and Siersdorfer2009, van Zwol et al.2010,Yamaguchi et al.2014]. However, this paper focuses only on tags, to emphasize the importance of tags.

The assumption here is that (1) tags used for images and videos with high social popularity scores are important, (2) the tags co-occurring with such important tags are also important, and (3) the tags become less important when annotated with many other tags. The equation for our FP-Rank scores, s, is defined as follows.

where rtag F is the tag-only FolkRank score.

There are two fundamental differences from PageRank and FolkRank. The first involves the design of the matrix AFP . AFP is a square matrix whose size is T × T , where T is the number of unique tags in the service. The matrix AFP can be interpreted as a combination of two matrices by introducing the tag-content matrices:

Here, Bw and Bt are T × C row-stochastic and columnstochastic matrices, respectively, where C is the number of the images and videos in the service. The ith row vector in Bw is a set of social popularity scores for the content tagged with the tag i, normalized by the sum of the scores in the row. The jth column vector in Bt is a set of tag usage flags normalized by the total number of tags attached to the content c. As a result, the component of the matrix AFP , aij is calculated as follows.

where d is the index of the image or video content that contains both tags i and j, D is the training dataset, and u(d)i is the weight for the tag i annotated to the content d. In FolkRank, u(d)i is simply calculated by

In contrast, FP-Rank calculates u(d)i by considering the content's social popularity scores:

where k is a parameter to prevent u(d)i from being 0 and k = 1 in our study.

Using this procedure, tag scores are redistributed to other tags via the image/video, through weighting determined by social popularity. In other words, FP-Rank considers not only the number of links (co-occurrences) between tags, but also the popularity of the content to which the tag is annotated.

The concept of the procedures in eqs. (7) and (9) is visually explained by a toy example in Fig. 2. Let us assume that there are only three images and only three unique tags in the service. The initial importance scores for the tags are all set to 1. Image #1 is annotated with bird and owl, image #2 with bird, owl, and fly, and image #3 with bird and fly. The images' social popularity scores (which include the numbers of views, comments, or favorites) are 40, 7, and 3, respectively. The tag scores are distributed to the images by considering the social popularity scores. They are collected back to the tags by considering the number of co-occurring tags. Then, the tag scores are updated by summing the scores of the tags. In this case, Bw and Bt are defined as follows.

The second difference from PageRank and FolkRank is that the FP-Rank score is given by eq. (5), not by eq. (4). In rFP in eq. (4), both co-occurrence-based scores and popularity-based scores are inevitably mixed. Therefore, by subtracting by the tag-only FolkRank score rtag F , only the popularity-based scores are obtained.

The iteration in eq. (4) is repeated until it converges. The related literature [Page et al.1999] points out that eq. (1) converges after approximately 50 iterations, but 10 iterations are sufficient for practical ranking systems. We confirmed that this is also the case with our FP-Rank algorithm.

Different from the original FolkRank, resource factors need not be considered because the social popularity score is given instead. When users are included in the FP-rank model, it is possible to conduct user-aware recommendations.

2.2 Tag Recommendation

Similar to FolkRank, tags can be recommended by the following equation.

where wFP is a weight vector with one entry for each tag. The random surfer component for the already existing tags is set to 1, and the others are set to 0 - 1; the eq. (4) is iterated until convergence. Setting the random surfer component p in this manner causes tags co-occurring with the already existing tags to be extracted. If the random surfer components for the unused tags are 0, there is no chance of selecting random tags. In addition, as explained in Sec. 2.1, the tag scores are redistributed by considering the contents' social popularity. Thus, the final tag scores correspond to the influence of the tags' social popularity scores.

3 Experimental Results

The experiments were performed on the YFCC100M dataset [Thomee et al.2015]. We randomly selected 60,000 images that have more than 20 tags and over 5000 views of the whole YFCC100m dataset for training and randomly selected 947 images annotated with over 20 tags but different number of views for testing (originally it was 1,000, but users dynamically updated/deleted their content and we could download only 947 images). Table 1 gives an overview on the training and testing sets. We can observe that the training set contains 250K unique tags and the images in the training set are really popular in Flickr, which can be good resources for recommendation.

3.1 Tag Recommendation

We first describe the procedure of tag recommendation. We evaluated a tag recommender based on the proposed FPRank and compared with four other recommendation methods: (1) Doing nothing (using only the original tags), (2) Tagcoor [Sigurbj¨ornsson and van Zwol2008], (3) Collaborative Filtering (CF) [Su and Khoshgoftaar2009, Lops et al.2013], and (4) CF with DF-W [Yamasaki et al.2014].

We added 10 tags for each image using the five recommendation methods respectively. An example of the original and the recommended tags are shown in Table 2. Changing the number of recommended tags is our future plan.

We evaluated the five methods in two ways. First, we uploaded a set of images and tags to Flickr and tracked the number of views. Second, the quality of the tag sets were evaluated humans in a crowd working service.

3.2 Uploading to Flickr

Online experiments is one of the most convincing approaches to evaluate the efficiency in popularity boosting. We repeated the following procedure five times: creating a new Flickr account, uploading images with tags that were recommended by one of the five approaches along with the original tags, checking the number of views for 10 days, and deleting all the files and the account. To avoid the situation where the search engine of Flickr returns a series of identical images at the same time during the retrieval process, only one approach is evaluated at one time. A new account is generated each time in order to eliminate the other possible effects such as the account being "liked." Therefore, social connection-based approach is not valid in this case. In this way, the subsequent experiments will not be disturbed by previous results, thus it ensures that the experiments are done independently. We record the number of views of all the images every 12 hours (this did not affect the numbers of views) and the whole online experiment lasted fromNov. 2016 to Jan. 2017.

Table 2: Example of recommendation results.

The results are shown in Figure 3. FP-Rank performed the best not only by the number of views per image but also that per image and the number of tags. The average number of views is more than 2 times higher than using only the original tags. This proves that FP-Rank makes better recommendations with a higher level of influence on popularity boosting over the other three tag recommendation methods. The popularity of the original tag sets was the worst, which shows that the ordinary people are not aware of the importance of tag selection. And CF with DF-W also worked well, only a little worse than FP-Rank, and significantly better than CF. We conducted the paired Student's t-test between CF+DFW and our FP-Rank in terms of the number of views and p = 0.015 < 0.05. Although the results using FolkRank is not presented, its performance is the same as rtag F because tags are annotated only by the owner of the content in Flickr.

3.3 Tag Quality Assessment

One may doubt that it is possible to assign tags which have a high level of popularity influence but totally unrelated to the image, which will lead to tag spam. In order to eliminate this confusion and prove that our recommendation methods do recommend proper tags based on the content of web objects, we conduct a quality assessment of tags obtained by different recommendation algorithms.

We conducted our assessment on Amazon's Mechanical Turk [Buhrmester et al.2011]. We asked workers to rate a tag on a five-point scale, where 5 stars for good quality and 1 star for bad quality. The content of an image and necessary descriptions are provided for reference, such as title, owner, date taken and etc. Figure 4 shows the design of the assessment task. We assigned five different workers, who has a approval rate of 85 % or higher, to rate each image-tag pair. The whole assessment took about three weeks in Jan. 2017 and more than 1, 500 workers contributed. We carefully checked all the answers and rejected answers that were uncompleted or of poor quality.

The rating scores of tags recommended by different methods are summarized in Table 3. Original tags and those recommended by Tagcoor are rated the highest, and the FP-Rank is rated a little lower (by 0.1 point). The results show that FP-Rank recommend reasonably good tags that can properly describe the images. On the other hand, the average rating of CF-based methods are the worst among all the methods.

4 Conclusions

In this paper, a FolkPopularityRank algorithm was proposed to rank tags in content sharing services according to the order of their influence on social popularity scores (i.e., the numbers of views, comments, favorites, and other rankings). This concept is markedly different from conventional tag ranking and recommendation systems, which only recommend related tags without considering their impact on social popularity. We conducted a real-life experiment by uploading 947 image sets to Flickr, and showed a potential that the tags recommended by our system could efficiently increase social popularity while maintaining appropriateness to describe the content.

Acknowledgments

This work was partially supported by the Grants-in-Aid for Scientific Research (no. 26700008) from JSPS, JST-CREST (JPMJCR1686), and Microsoft IJARC core13. J. Hu is now with Microsoft Development, Co., Ltd. and S. Sano is now with Yahoo Japan Corporation.