{"type":"ScholarlyArticle","title":"Collaborative and Attentive Learning for Personalized Image Aesthetic Assessment","authors":[{"name":"Guolong Wang","slug":"guolong-wang","given":"Guolong","family":"Wang","affiliation":"BNRist , School of Software , Tsinghua University , China"},{"name":"Junchi Yan","slug":"junchi-yan","given":"Junchi","family":"Yan","affiliation":"Shanghai Jiao Tong University"},{"name":"Zheng Qin","slug":"zheng-qin","given":"Zheng","family":"Qin","affiliation":"BNRist , School of Software , Tsinghua University , China"}],"abstract":"The ever-increasing volume of visual images has stimulated the demand for organizing such data by aesthetic quality. Automatic and especially learning based aesthetic assessment methods have shown potential by recent works. Existing image aesthetic prediction is often user-agnostic which may ignore the fact that the rating to an image can be inherently individual. We fill this gap by formulating the personalized image aesthetic assessment problem with a novel learning method. Specifically, we collect user-image textual reviews in addition with visual images from the public dataset to organize a review-augmented benchmark. Using this enriched dataset, we devise a deep neural network with a user/image relation encoding input for collaborative filtering. Meanwhile an attentive mechanism is designed to capture the user-specific taste for image semantic tags and regions of interest by fusing the image and user's review. Extensive and promising experimental results on the review-augmented benchmark corroborate the efficacy of our approach.","doi":"10.24963/ijcai.2018/133","url":"https://ijcai.org/proceedings/2018/133","pdf":"https://ijcai.org/proceedings/2018/0133.pdf","markdown":"https://ijcai.org/proceedings/2018/133.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/133","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":133,"pages":{"first":957,"last":963},"keywords":["Computer Vision: Computer Vision","Computer Vision: Perception"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhattacharya et al., 2010] Subhabrata Bhattacharya, Rahul Sukthankar, and Mubarak Shah. A framework for photo- quality assessment and enhancement based on visual aes- thetics. In Multimedia, pages 271-280. ACM, 2010.","title":"A framework for photoquality assessment and enhancement based on visual aesthetics","authors":"Subhabrata Bhattacharya, Rahul Sukthankar, Mubarak Shah","year":"2010","venue":"Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bradley, 1997] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algo- rithms. Pattern recognition, 30(7):1145-1159, 1997.","title":"The use of the area under the roc curve in the evaluation of machine learning algorithms","authors":"P Andrew, Bradley","year":"1997","venue":"Pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Datta et al., 2006] Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang. Studying aesthetics in photographic im- ages using a computational approach. In ECCV, pages 288-301. Springer, 2006.","title":"Studying aesthetics in photographic images using a computational approach","authors":"Datta","year":"2006","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Datta et al., 2007] Ritendra Datta, Jia Li, and James Z Wang. Learning the consensus on visual quality for next- generation image management. In Multimedia, pages 533- 536. ACM, 2007.","title":"Learning the consensus on visual quality for nextgeneration image management","authors":"Datta","year":"2007","venue":"Multimedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, and Fei Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, Fei Fei, Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dhar et al., 2011] Sagnik Dhar, Vicente Ordonez, and Tamara L Berg. High level describable attributes for pre- dicting aesthetics and interestingness. In CVPR, pages 1657-1664. IEEE, 2011.","title":"High level describable attributes for predicting aesthetics and interestingness","authors":"Sagnik Dhar, Vicente Ordonez, Tamara L Berg","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2015] Zhe Dong, Xu Shen, Houqiang Li, and Xinmei Tian. Photo quality assessment with dcnn that understands image well. In International Conference on Multimedia Modeling, pages 524-535. Springer, 2015.","title":"Photo quality assessment with dcnn that understands image well","authors":"Zhe Dong, Xu Shen, Houqiang Li, Xinmei Tian","year":"2015","venue":"International Conference on Multimedia Modeling","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goldberg, 1990] Lewis R Goldberg. An alternative \"de- scription of personality\": The big-five factor struc- ture. Journal of Personality and Social Psychology, 59(6):1216-1229, 1990.","title":"An alternative \"description of personality\": The big-five factor structure","authors":"Lewis R Goldberg","year":"1990","venue":"Journal of Personality and Social Psychology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He and Mcauley, 2016] Ruining He and Julian Mcauley. Vbpr: visual bayesian personalized ranking from implicit feedback. In AAAI, pages 144-150, 2016.","title":"Ruining He and Julian Mcauley. Vbpr: visual bayesian personalized ranking from implicit feedback","authors":"Mcauley He","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kang et al., 2014] Le Kang, Peng Ye, Yi Li, and David Do- ermann. Convolutional neural networks for no-reference image quality assessment. In CVPR, pages 1733-1740, 2014.","title":"Convolutional neural networks for no-reference image quality assessment","authors":"Le Kang, Peng Ye, Yi Li, David Doermann","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kao et al., 2017] Yueying Kao, Ran He, and Kaiqi Huang. Deep aesthetic quality assessment with semantic in- formation. IEEE Transactions on Image Processing, 26(3):1482-1495, 2017.","title":"Deep aesthetic quality assessment with semantic information","authors":"Kao","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ke et al., 2006] Yan Ke, Xiaoou Tang, and Feng Jing. The design of high-level features for photo quality assessment. In CVPR, volume 1, pages 419-426. IEEE, 2006.","title":"The design of high-level features for photo quality assessment","authors":"Yan Ke, Xiaoou Tang, Feng Jing","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kong et al., 2016] Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In ECCV, 2016.","title":"Photo aesthetics ranking network with attributes and content adaptation","authors":"Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, Charless Fowlkes","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2015a] Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z Wang. Rating image aesthetics us- ing deep learning. IEEE Transactions on Multimedia, 17(11):2021-2034, 2015.","title":"Rating image aesthetics using deep learning","authors":"Lu","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2015b] Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z Wang. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In ICCV, pages 990-998, 2015.","title":"Deep multi-patch aggregation network for image style, aesthetics, and quality estimation","authors":"Lu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Manning et al., 2014] Christopher D. Manning, Mihai Sur- deanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In ACL, pages 55-60, 2014.","title":"The Stanford CoreNLP natural language processing toolkit","authors":"D Christopher, Mihai Manning, John Surdeanu, Jenny Bauer, Steven J Finkel, David Bethard, Mcclosky","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marchesotti et al., 2015] Luca Marchesotti, Naila Murray, and Florent Perronnin. Discovering beautiful attributes for aesthetic image analysis. IJCV, 113(3):246-266, 2015.","title":"Discovering beautiful attributes for aesthetic image analysis","authors":"Luca Marchesotti, Naila Murray, Florent Perronnin","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Murray et al., 2012] Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aes- thetic visual analysis. In CVPR, pages 2408-2415. IEEE, 2012.","title":"Ava: A large-scale database for aesthetic visual analysis","authors":"Naila Murray, Luca Marchesotti, Florent Perronnin","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nishiyama et al., 2011] Masashi Nishiyama, Takahiro Ok- abe, Imari Sato, and Yoichi Sato. Aesthetic quality classi- fication of photographs based on color harmony. In CVPR, pages 33-40. IEEE, 2011.","title":"Aesthetic quality classification of photographs based on color harmony","authors":"Masashi Nishiyama, Takahiro Okabe, Imari Sato, Yoichi Sato","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rendle et al., 2012] Steffen Rendle, Christoph Freuden- thaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. pages 452-461, 2012.","title":"Bpr: Bayesian personalized ranking from implicit feedback","authors":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, Lars Schmidt-Thieme","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Segalin et al., 2016] Cristina Segalin, Alessandro Perina, Marco Cristani, and Alessandro Vinciarelli. The pictures we like are our image: Continuous mapping of favorite pictures into self-assessed and attributed personality traits. IEEE Transactions on Affective Computing, pages 1-14, 2016.","title":"The pictures we like are our image: Continuous mapping of favorite pictures into self-assessed and attributed personality traits","authors":"Cristina Segalin, Alessandro Perina, Marco Cristani, Alessandro Vinciarelli","year":"2016","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. Computer Science, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian et al., 2015] Xinmei Tian, Zhe Dong, Kuiyuan Yang, and Tao Mei. Query-dependent aesthetic model with deep learning for photo quality assessment. IEEE Transactions on Multimedia, 17(11):2035-2048, 2015.","title":"Query-dependent aesthetic model with deep learning for photo quality assessment","authors":"Zhe Xinmei Tian, Kuiyuan Dong, Tao Yang, Mei","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tong et al., 2004] Hanghang Tong, Mingjing Li, Hong- Jiang Zhang, Jingrui He, and Changshui Zhang. Classi- fication of digital photos taken by photographers or home users. In Pacific-Rim Conference on Multimedia, pages 198-205. Springer, 2004.","title":"Classification of digital photos taken by photographers or home users","authors":"Tong","year":"2004","venue":"Pacific-Rim Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2013] Fang-Lue Zhang, Miao Wang, and Shi- Min Hu. Aesthetic image enhancement by dependence- aware object recomposition. IEEE Transactions on Multi- media, 15(7):1480-1490, 2013.","title":"Aesthetic image enhancement by dependenceaware object recomposition","authors":"Zhang","year":"2013","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang, 2016] Luming Zhang. Describing human aesthetic perception by deeply-learned attributes from flickr. arXiv preprint arXiv:1605.07699, 2016.","title":"Describing human aesthetic perception by deeply-learned attributes from flickr","authors":"Luming Zhang, Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}