Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network

Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network

Jufeng Yang, Dongyu She, Ming Sun

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 3266-3272. https://doi.org/10.24963/ijcai.2017/456

Visual sentiment analysis is attracting more and more attention with the increasing tendency to express emotions through visual contents. Recent algorithms in convolutional neural networks (CNNs) considerably advance the emotion classification, which aims to distinguish differences among emotional categories and assigns a single dominant label to each image. However, the task is inherently ambiguous since an image usually evokes multiple emotions and its annotation varies from person to person. In this work, we address the problem via label distribution learning (LDL) and develop a multi-task deep framework by jointly optimizing both classification and distribution prediction. While the proposed method prefers to the distribution dataset with annotations of different voters, the majority voting scheme is widely adopted as the ground truth in this area, and few dataset has provided multiple affective labels. Hence, we further exploit two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category. The experiments conducted on both distribution datasets, i.e., Emotion6, Flickr_LDL, Twitter_LDL, and the largest single emotion dataset, i.e., Flickr and Instagram, demonstrate the proposed method outperforms the state-of-the-art approaches.
Keywords:
Machine Learning: Transfer, Adaptation, Multi-task Learning
Machine Learning: Multi-instance/Multi-label/Multi-view learning
Multidisciplinary Topics and Applications: AI and Ubiquitous Computing Systems