{"type":"ScholarlyArticle","title":"Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network","authors":[{"name":"Jufeng Yang","slug":"jufeng-yang","given":"Jufeng","family":"Yang","affiliation":"College of Computer and Control Engineering , Nankai University , Tianjin , China"},{"name":"Dongyu She","slug":"dongyu-she","given":"Dongyu","family":"She","affiliation":"College of Computer and Control Engineering , Nankai University , Tianjin , China"},{"name":"Ming Sun","slug":"ming-sun","given":"Ming","family":"Sun","affiliation":"College of Computer and Control Engineering , Nankai University , Tianjin , China"}],"abstract":"Visual sentiment analysis is attracting more and more attention with the increasing tendency to express emotions through visual contents. Recent algorithms in convolutional neural networks (CNNs) considerably advance the emotion classification, which aims to distinguish differences among emotional categories and assigns a single dominant label to each image. However, the task is inherently ambiguous since an image usually evokes multiple emotions and its annotation varies from person to person. In this work, we address the problem via label distribution learning (LDL) and develop a multi-task deep framework by jointly optimizing both classification and distribution prediction. While the proposed method prefers to the distribution dataset with annotations of different voters, the majority voting scheme is widely adopted as the ground truth in this area, and few dataset has provided multiple affective labels. Hence, we further exploit two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category. The experiments conducted on both distribution datasets, i.e., Emotion6, Flickr_LDL, Twitter_LDL, and the largest single emotion dataset, i.e., Flickr and Instagram, demonstrate the proposed method outperforms the state-of-the-art approaches.","doi":"10.24963/ijcai.2017/456","url":"https://ijcai.org/proceedings/2017/456","pdf":"https://ijcai.org/proceedings/2017/0456.pdf","markdown":"https://ijcai.org/proceedings/2017/456.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/456","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":456,"pages":{"first":3266,"last":3272},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Multidisciplinary Topics and Applications: AI and Ubiquitous Computing Systems","Machine Learning: Multi-instance/Multi-label/Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Borth et al., 2013] Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In MM, 2013.","title":"Large-scale visual sentiment ontology and detectors using adjective noun pairs","authors":"Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, Shih-Fu Chang","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Campos et al., 2015] Víctor Campos, Amaia Salvador, Xavier Giro-i Nieto, and Brendan Jou. Diving deep into sentiment: Understanding fine-tuned cnns for visual sentiment prediction. In ACM ASM, 2015.","title":"Diving deep into sentiment: Understanding fine-tuned cnns for visual sentiment prediction","authors":"Víctor Campos, Amaia Salvador, Xavier Giro-I Nieto, Brendan Jou","year":"2015","venue":"ACM ASM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. TIST, 2(3):27, 2011.","title":"Libsvm: a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"TIST","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2014] Tao Chen, Damian Borth, Trevor Dar- rell, and Shih-Fu Chang. Deepsentibank: Visual sentiment concept classification with deep convolutional neural net- works. arXiv preprint arXiv:1410.8586, 2014.","title":"Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks","authors":"Tao Chen, Damian Borth, Trevor Darrell, Shih-Fu Chang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2017] Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. Deep label distribution learn- ing with label ambiguity. TIP, 2017.","title":"Deep label distribution learning with label ambiguity","authors":"Gao","year":"2017","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geng et al., 2013] Xin Geng, Chao Yin, and Zhi-Hua Zhou. Facial age estimation by learning from label distributions. TPAMI, 35(10):2401-2412, 2013.","title":"Facial age estimation by learning from label distributions","authors":"Xin Geng, Zhi-Hua Chao Yin, Zhou","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geng, 2016] Xin Geng. Label distribution learning. TKDE, 28(7):1734-1748, 2016.","title":"Label distribution learning","authors":"Xin Geng","year":"2016","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia et al., 2012] Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, and Jie Tang. Can we understand van gogh's mood? learning to infer affects from images in so- cial networks. In ACM MM, 2012.","title":"Can we understand van gogh's mood? learning to infer affects from images in social networks","authors":"Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, Jie Tang","year":"2012","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2012] Xin Lu, Poonam Suryanarayan, Reginald B Adams Jr, Jia Li, Michelle G Newman, and James Z Wang. On shape and the computability of emotions. In ACM MM, 2012.","title":"On shape and the computability of emotions","authors":"Xin Lu, Poonam Suryanarayan, Reginald B Adams, Jia Li, Michelle G Newman, James Z Wang","year":"2012","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Machajdik and Hanbury, 2010] Jana Machajdik and Allan Hanbury. Affective image classification using features in- spired by psychology and art theory. In ACM MM, 2010.","title":"Affective image classification using features inspired by psychology and art theory","authors":"Hanbury Machajdik, Jana Machajdik, Allan Hanbury","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikels et al., 2005] Joseph A Mikels, Barbara L Fredrick- son, Gregory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz. Emotional category data on images from the international affective picture system. Behavior research methods, 37(4):626-630, 2005.","title":"Emotional category data on images from the international affective picture system","authors":"Barbara L Joseph A Mikels, Fredrickson, Casey M Gregory R Larkin, Sam J Lindberg, Patricia A Maglio, Reuter-Lorenz","year":"2005","venue":"Behavior research methods","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mirzazadeh et al., 2015] Farzaneh Mirzazadeh, Siamak Ra- vanbakhsh, Nan Ding, and Dale Schuurmans. Embedding inference for structured multilabel prediction. In NIPS, 2015.","title":"Embedding inference for structured multilabel prediction","authors":"Farzaneh Mirzazadeh, Siamak Ravanbakhsh, Nan Ding, Dale Schuurmans","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nicolaou et al., 2011] Mihalis A Nicolaou, Hatice Gunes, and Maja Pantic. A multi-layer hybrid framework for di- mensional emotion classification. In ACM MM, 2011.","title":"A multi-layer hybrid framework for dimensional emotion classification","authors":"A Mihalis, Hatice Nicolaou, Maja Gunes, Pantic","year":"2011","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pang and Ngo, 2015] Lei Pang and Chong-Wah Ngo. Mut- limodal learning with deep boltzmann machine for emo- tion prediction in user generated videos. In ICMR, 2015.","title":"Mutlimodal learning with deep boltzmann machine for emotion prediction in user generated videos","authors":"Ngo Pang, Lei Pang, Chong-Wah Ngo","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peng et al., 2015] Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C Gallagher. A mixed bag of emo- tions: Model, predict, and transfer emotion distributions. In CVPR, 2015.","title":"A mixed bag of emotions: Model, predict, and transfer emotion distributions","authors":"Kuan-Chuan, Tsuhan Peng, Amir Chen, Andrew C Sadovnik, Gallagher","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Plutchik, 1980] Robert Plutchik. A general psychoevolu- tionary theory of emotion. Theories of emotion, 1:3-31, 1980.","title":"A general psychoevolutionary theory of emotion","authors":"Robert Plutchik","year":"1980","venue":"Theories of emotion","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Plutchik, 2001] Robert Plutchik. The nature of emotions. American Scientist, 89(4):344-350, 2001.","title":"The nature of emotions","authors":"Robert Plutchik","year":"2001","venue":"American Scientist","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rao et al., 2016] Tianrong Rao, Min Xu, and Dong Xu. Learning multi-level deep representations for image emo- tion classification. arXiv preprint arXiv:1611.07145, 2016.","title":"Learning multi-level deep representations for image emotion classification","authors":"Rao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2016] Ming Sun, Jufeng Yang, Kai Wang, and Hui Shen. Discovering affective regions in deep convo- lutional neural networks for visual sentiment prediction. In ICME, 2016.","title":"Discovering affective regions in deep convolutional neural networks for visual sentiment prediction","authors":"Ming Sun, Jufeng Yang, Kai Wang, Hui Shen","year":"2016","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Jufeng Yang, Ming Sun, and Xiaoxiao Sun. Learning visual sentiment distribution via augmented conditional probability neural network. In AAAI, 2017.","title":"Learning visual sentiment distribution via augmented conditional probability neural network","authors":"Jufeng Yang, Ming Sun, Xiaoxiao Sun","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[You et al., 2015] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Robust image sentiment analysis using progressively trained and domain transferred deep net- works. In AAAI, 2015.","title":"Robust image sentiment analysis using progressively trained and domain transferred deep networks","authors":"You","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[You et al., 2016] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In AAAI, 2016.","title":"Building a large scale dataset for image emotion recognition: The fine print and the benchmark","authors":"You","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2014] Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. Ex- ploring principles-of-art features for image emotion recog- nition. In ACM MM, 2014.","title":"Exploring principles-of-art features for image emotion recognition","authors":"Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, Xiaoshuai Sun","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2015] Sicheng Zhao, Hongxun Yao, Xiaolei Jiang, and Xiaoshuai Sun. Predicting discrete probability distribution of image emotions. In ICIP, 2015.","title":"Predicting discrete probability distribution of image emotions","authors":"Sicheng Zhao, Hongxun Yao, Xiaolei Jiang, Xiaoshuai Sun","year":"2015","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2016] Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, and Tat-Seng Chua. Predicting personalized emotion perceptions of so- cial images. In MM, 2016.","title":"Predicting personalized emotion perceptions of social images","authors":"Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, Tat-Seng Chua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}