{"type":"ScholarlyArticle","title":"Rethinking Image Aesthetics Assessment: Models, Datasets and Benchmarks","authors":[{"name":"Shuai He","slug":"shuai-he","given":"Shuai","family":"He","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Yongchang Zhang","slug":"yongchang-zhang","given":"Yongchang","family":"Zhang","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Rui Xie","slug":"rui-xie","given":"Rui","family":"Xie","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Dongxiang Jiang","slug":"dongxiang-jiang","given":"Dongxiang","family":"Jiang","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Anlong Ming","slug":"anlong-ming","given":"Anlong","family":"Ming","affiliation":"Beijing University of Posts and Telecommunications"}],"abstract":"Challenges in image aesthetics assessment (IAA) arise from that images of different themes correspond to different evaluation criteria, and learning aesthetics directly from images while ignoring the impact of theme variations on human visual perception inhibits the further development of IAA; however, existing IAA datasets and models overlook this problem. To address this issue, we show that a theme-oriented dataset and model design are effective for IAA. Specifically, 1) we elaborately build a novel dataset, called TAD66K, that contains 66K images covering 47 popular themes, and each image is densely annotated by more than 1200 people with dedicated theme evaluation criteria. 2) We develop a baseline model, TANet, which can effectively extract theme information and adaptively establish perception rules  to evaluate images with different themes. 3) We develop a large-scale benchmark (the most comprehensive thus far) by comparing 17 methods with TANet on three representative datasets: AVA, FLICKR-AES and the proposed TAD66K, TANet achieves state-of-the-art performance on all three datasets. Our work offers the community an opportunity to explore more challenging directions; the code, dataset and supplementary material are available at https://github.com/woshidandan/TANet.","doi":"10.24963/ijcai.2022/132","url":"https://ijcai.org/proceedings/2022/132","pdf":"https://ijcai.org/proceedings/2022/0132.pdf","markdown":"https://ijcai.org/proceedings/2022/132.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/132","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":132,"pages":{"first":942,"last":948},"keywords":["Computer Vision: Machine Learning for Vision","Machine Learning: Theory of Deep Learning","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986654","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984797","length_sec":257,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Barnbaum, 2017] Bruce Barnbaum. The Art of Photogra- phy: A Personal Approach to Artistic Expression. Rocky Nook, Inc., 2017.","title":"The Art of Photography: A Personal Approach to Artistic Expression","authors":"Bruce Barnbaum","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2017] Kuang-Yu Chang, Kung-Hung Lu, and Chu-Song Chen. Aesthetic critiques generation for photos. In ICCV, pages 3514-3523, 2017.","title":"Aesthetic critiques generation for photos","authors":"Kuang-Yu Chang, Kung-Hung Lu, Chu-Song Chen","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Qiuyu Chen, Wei Zhang, Ning Zhou, Peng Lei, Yi Xu, Yu Zheng, and Jianping Fan. Adaptive fractional dilated convolution network for image aesthetics assessment. In CVPR, pages 14114-14123, 2020.","title":"Adaptive fractional dilated convolution network for image aesthetics assessment","authors":"Qiuyu Chen, Wei Zhang, Ning Zhou, Peng Lei, Yi Xu, Yu Zheng, Jianping Fan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Datta et al., 2008] Ritendra Datta, Jia Li, and James Z Wang. Algorithmic inferencing of aesthetics and emotion in natural images: An exposition. In ICIP. IEEE, 2008.","title":"Algorithmic inferencing of aesthetics and emotion in natural images: An exposition","authors":"Datta","year":"2008","venue":"ICIP. IEEE","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2020] Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In CVPR, pages 3677-3686, 2020.","title":"Perceptual quality assessment of smartphone photography","authors":"Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, Zhou Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hosu et al., 2019] Vlad Hosu, Bastian Goldlucke, and Diet- mar Saupe. Effective aesthetics prediction with multi-level spatially pooled features. In CVPR, 2019.","title":"Effective aesthetics prediction with multi-level spatially pooled features","authors":"Vlad Hosu, Bastian Goldlucke, Dietmar Saupe","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin et al., 2019] Xin Jin, Le Wu, Geng Zhao, Xiaodong Li, Xiaokun Zhang, Shiming Ge, Dongqing Zou, Bin Zhou, and Xinghui Zhou. Aesthetic attributes assessment of im- ages. In ACMMM, pages 311-319, 2019.","title":"Aesthetic attributes assessment of images","authors":"Xin Jin, Le Wu, Geng Zhao, Xiaodong Li, Xiaokun Zhang, Shiming Ge, Dongqing Zou, Bin Zhou, Xinghui Zhou","year":"2019","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Joshi et al., 2011] Dhiraj Joshi, Ritendra Datta, Elena Fe- dorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo. Aesthetics and emotions in images. IEEE Signal Process. Mag., 28(5):94-115, 2011.","title":"Aesthetics and emotions in images","authors":"Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, Jiebo Luo","year":"2011","venue":"IEEE Signal Process. Mag","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ke et al., 2021] Junjie Ke, Qifei Wang, Yilin Wang, Pey- man Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In ICCV, pages 5148-5157, 2021.","title":"Musiq: Multi-scale image quality transformer","authors":"Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, Feng Yang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kong et al., 2016] Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In ECCV, pages 662-679. Springer, 2016.","title":"Photo aesthetics ranking network with attributes and content adaptation","authors":"Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, Charless Fowlkes","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, and Weisi Lin. Personality-assisted multi- task learning for generic and personalized image aesthetics assessment. IEEE TIP, 29:3898-3910, 2020.","title":"Personality-assisted multitask learning for generic and personalized image aesthetics assessment","authors":"Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, Weisi Lin","year":"2020","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2014] Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. RAPID: Rating pictorial aesthetics using deep learning. In ACMMM, pages 457-466, 2014.","title":"RAPID: Rating pictorial aesthetics using deep learning","authors":"Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, James Z Wang","year":"2014","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2015] Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z. Wang. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In ICCV, pages 990-998, 2015.","title":"Deep multi-patch aggregation network for image style, aesthetics, and quality estimation","authors":"Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, James Z Wang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2011] Wei Luo, Xiaogang Wang, and Xiaoou Tang. Content-based photo quality assessment. In ICCV, pages 2206-2213. IEEE, 2011.","title":"Content-based photo quality assessment","authors":"Wei Luo, Xiaogang Wang, Xiaoou Tang","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lv et al., 2021] Pei Lv, Jianqi Fan, Xixi Nie, Weiming Dong, Xiaoheng Jiang, Bing Zhou, Mingliang Xu, and Changsheng Xu. User-guided personalized image aes- thetic assessment based on deep reinforcement learning. arXiv:2106.07488, 2021.","title":"User-guided personalized image aesthetic assessment based on deep reinforcement learning","authors":"Lv","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2017] Shuang Ma, Jing Liu, and Chang Wen Chen. A-lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assess- ment. In CVPR, pages 722-731, 2017.","title":"A-lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment","authors":"Shuang Ma, Jing Liu, Chang Wen, Chen","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mai et al., 2016] Long Mai, Hailin Jin, and Feng Liu. Composition-preserving deep photo aesthetics assess- ment. In CVPR, pages 497-506, 2016.","title":"Composition-preserving deep photo aesthetics assessment","authors":"Mai","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Murray et al., 2012] Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aes- thetic visual analysis. In CVPR. IEEE, 2012.","title":"Ava: A large-scale database for aesthetic visual analysis","authors":"Naila Murray, Luca Marchesotti, Florent Perronnin","year":"2012","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[O'Donovan et al., 2011] Peter O'Donovan, Aseem Agar- wala, and Aaron Hertzmann. Color compatibility from large datasets. In SIGGRAPH, pages 1-12. 2011.","title":"Color compatibility from large datasets","authors":"O'donovan","year":"2011","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ren et al., 2017] Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David J Foran. Personalized image aesthetics. In ICCV, pages 638-647, 2017.","title":"Personalized image aesthetics","authors":"Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, David J Foran","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[She et al., 2021] Dongyu She, Yu-Kun Lai, Gaoxiong Yi, and Kun Xu. Hierarchical layout-aware graph convolu- tional network for unified aesthetics assessment. In CVPR, pages 8475-8484, 2021.","title":"Hierarchical layout-aware graph convolutional network for unified aesthetics assessment","authors":"Dongyu She, Yu-Kun Lai, Gaoxiong Yi, Kun Xu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sheng et al., 2018] Kekai Sheng, Weiming Dong, Chongyang Ma, Xing Mei, Feiyue Huang, and Bao- Gang Hu. Attention-based multi-patch aggregation for image aesthetic assessment. In ACMMM, 2018.","title":"Attention-based multi-patch aggregation for image aesthetic assessment","authors":"Kekai Sheng, Weiming Dong, Chongyang Ma, Xing Mei, Feiyue Huang, Bao-Gang Hu","year":"2018","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Talebi and Milanfar, 2018] Hossein Talebi and Peyman Mi- lanfar. NIMA: Neural image assessment. TIP, 27(8):3998-4011, 2018.","title":"NIMA: Neural image assessment","authors":"Hossein Talebi, Peyman Milanfar","year":"2018","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018] Guolong Wang, Junchi Yan, and Zheng Qin. Collaborative and attentive learning for personalized image aesthetic assessment. In IJCAI, 2018.","title":"Collaborative and attentive learning for personalized image aesthetic assessment","authors":"Wang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/133"},{"position":28,"raw":"[Zeng et al., 2019] Hui Zeng, Zisheng Cao, Lei Zhang, and Alan C Bovik. A unified probabilistic formulation of im- age aesthetic assessment. TIP, 29:1548-1561, 2019.","title":"A unified probabilistic formulation of image aesthetic assessment","authors":"Hui Zeng, Zisheng Cao, Lei Zhang, Alan C Bovik","year":"2019","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452-1464, 2017.","title":"Places: A 10 million image database for scene recognition","authors":"Zhou","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2020] Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. Per- sonalized image aesthetics assessment via meta-learning with bilevel gradient optimization. TCYB, 2020.","title":"Personalized image aesthetics assessment via meta-learning with bilevel gradient optimization","authors":"Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, Guangming Shi","year":"2020","venue":"TCYB","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}