Affective Image Content Analysis: A Comprehensive Survey

1 Introduction

With the rapid development of photography technology and the wide popularity of social networks, people have become used to sharing their experiences and expressing their opinions online using images and videos together with text. This gives rise to a huge volume of multimedia data, which results in urgent demand of processing and understanding the visual content. Compared to low-level visual appearances, humans could better perceive and understand the high cognitive level and affective level of semantics [Zhao et al., 2014a]. Existing works on image content analysis from the computer vision community mainly focus on understanding the cognitive aspects of images, such as object detection. Recently, a lot of research attention has been paid from the psychology, affective computing and multimedia communities to affective level analysis of image content. As what people feel may directly determine their decision making, affective image content analysis (AICA) is of great importance, which can enable wide applications [Chen et al., 2014], ranging from humancomputer interaction to image retrieval.

∗Corresponding author: Sicheng Zhao (schzhao@gmail.com)

Specifically, the task of AICA is often composed of three steps: human annotation, visual feature extraction and learning of mapping between visual features and perceived emotions [Zhao et al., 2017a]. One main challenge for AICA is the affective gap, which can be defined as "the lack of coincidence between the features and the expected affective state in which the user is brought by perceiving the signal" [Hanjalic, 2006]. Recently, various hand-crafted or learning-based features have been designed to bridge this gap. Current AICA methods mainly assign an image with the dominant (average) emotion category (DEC) with the assumption that different viewers react similarly to the same image. This task can be performed as a traditional single-label learning problem.

However, labeling the emotions in ground-truth generation is in fact highly inconsistent. Different viewers may have totally different emotional reactions to the same image, which is caused by many personal and situational factors, such as the cultural background, personality and social context [Peng et al., 2015; Zhao et al., 2016; Yang et al., 2017b]. This phenomenon causes the so-called subjective perception problem, as shown in Figure 1. In such cases, just predicting the DEC is insufficient for this highly subjective variable.

To tackle the subjectivity issue, we can conduct two kinds of AICA tasks [Zhao et al., 2016]: for each viewer, we can predict personalized emotion perceptions; for each image, we can assign multiple emotion labels. For the latter one, we can employ multi-label learning methods, which associates one instance with multiple emotion labels. However, the importance or extent of different labels is in fact unequal. In such cases, emotion distribution learning would make more sense, which aims to learn the degree to which each emotion describes the instance [Yang et al., 2017b].

In this paper, we concentrate on reviewing the state-of-theart methods on AICA and outlining research trends. First, we introduce the widely-used emotion representation models and the available datasets for performing AICA evaluation. Second, we summarize and compare the representative approaches on emotion feature extraction, personalized emotion prediction, and emotion distribution learning, corresponding to the affective gap and perception subjectivity challenges. Finally, we discuss potential research directions to pursue.

2 EmotionModels fromCognitive Science Community

Psychologists mainly employ two typical models to represent emotions: categorical emotion states (CES), and dimensional emotion space (DES). CES models consider emotions to be one of a few basic categories, such as fear and joy, etc. Some typical models include Ekman's six basic emotions [Ekman, 1992] and Mikels's eight emotions [Mikels et al., 2005]. Specifically, emotions can be classified into positive and negative [Zhao et al., 2018b], sometimes including neutral. In such case, "emotion" is usually called "sentiment". DES models usually employ a 3D or 2D Cartesian space to represent emotions, such as valence-arousal-dominance (VAD) [Schlosberg, 1954] and activity-temperature-weight [Lee and Park, 2011]. VAD is the most widely used DES, where valence represents the pleasantness ranging from happy to unhappy, arousal represents the intensity of emotion ranging from excited to calm, while dominance represents the degree of control ranging from controlled to in control. Dominance is difficult to measure and is often omitted, leading to the commonly used two dimensional VA space [Hanjalic, 2006], where V is easier to recognize in AICA than A.

The relationship between CES and DES and the transformation from one to the other are studied in [Sun et al., 2009]. For example, positive valence relates to a happy state, while negative valence relates to a sad or angry state. CES is easier for users to understand and label, but may not reflect the subtlety and complexity of emotions. DES is more flexible and richer in descriptive power, but absolute continuous values are difficult for users to distinguish and may be unmeaningful because of the lack of agreed-upon standards for subjective emotion rating. CES is mainly used in classification task, while DES is often employed for regression. If discretized into several constant scores, DES can also be used for classification [Lee and Park, 2011]. Ranking based labeling can be applied to ease DEC comprehension difficulties in raters.

Expected emotion is the emotion that the image creator intends to make people feel, while perceived emotion is the actual emotion that is perceived by the image viewers. Discussing the difference or correlation of various emotion models is out of the scope of this paper. We just list some typical emotion models that have been widely used in AICA, as shown in Table 1.

3 Datasets

The early datasets for AICA mainly come from the psychology community with small-scale images. Recent large-scale datasets are constructed using images from social networks. The released datasets are summarized in Table 2.

The International Affective Picture System (IAPS) is an emotion evoking image set in psychology [Lang et al., 1997]. It consists of 1,182 documentary-style natural color images depicting complex scenes, such as portraits, babies, animals, landscapes, etc. Each image is associated with an empirically derived mean and standard deviation (STD) of VAD ratings in a 9-point rating scale by about 100 college students.

The (IAPSa) dataset includes 246 images [Mikels et al., 2005] selected from IAPS, which are labeled by 20 undergraduate participants with a mean age 19.55 years.

The (Abstract) includes 279 peer rated abstract paintings without contextual content, while the Artistic database (ArtPhoto) consists of 806 artistic photographs from a photo sharing site [Machajdik and Hanbury, 2010], the emotions of which are determined by the artist who uploaded the photo.

The IAPSa, ArtPhoto, and Abstract datasets are categorized into eight discrete categories [Mikels et al., 2005]. Each image is labeled with only one emotion in ArtPhoto, the images in IAPSa may be labeled with more than one emotion, while the detailed votes of all emotions are provided for the images in Abstract, from which one can obtain the DEC and emotion distributions. Totally, 395 emotions are labeled on the 246 images in IAPSa. 228 Abstract images are usually used for emotion classification after DEC selection.

The Geneva affective picture database (GAPED) is composed of 520 negative images, 121 positive images and 89 neutral images [Dan-Glauser and Scherer, 2011], labeled by 60 participants with a mean age of 24 years (ranging from 19

to 43, STD = 5.9). Besides, these images are also rated with valence and arousal values, ranging from 0 to 100 points.

The MART dataset is a collection of 500 abstract paintings from the Museum of Modern and Contemporary Art of Trento and Rovereto [Alameda-Pineda et al., 2016], which were realized by professional artists. The devArt dataset is a collection of 500 amateur abstract paintings obtained from the "DeviantArt" online social network [Alameda-Pineda et al., 2016], one of the largest online art communities. Using the relative score method in [Sartori et al., 2015], these abstract paintings are labeled as positive or negative sentiment.

Tweet dataset (Tweet) includes 470 positive tweets and 133 negative tweets [Borth et al., 2013].

The FlickrCC dataset is constructed by retrieving the Flickr creative common (CC) images for the 3,000 adjective noun pairs (ANPs) [Borth et al., 2013]. By excluding the images that do not contain the ANP string in the title, tag or description, about 500k Flickr CC images in total are generated for 1,553 ANPs. These images are then mapped to the Plutchnik's Wheel of Emotions with 8 basic emotions, each with 3 scales, such as ecstasy→joy→serenity.

The Flickr dataset consists of 301,903 images based on the Ekman emotion model [Yang et al., 2014]. A word list for each of the six emotion categories is manually defined based on WordNet and HowNet. The emotion category whose word list has the most same words as the adjective words of an image's tags and comments is assigned to the image.

The original FI dataset consists of 90,000 noisy images collected from Flicker and Instagram by searching the emotion keywords [You et al., 2016b]. The weakly labeled images are further labeled by 225 Amazon Mechanical Turk (AMT) workers, which are selected through a qualification test. The 23,308 images that receive at least three votes from their assigned 5 AMT workers are kept. The number of images in each Mikels emotion category is larger than 1,000.

The Emotion6 dataset [Peng et al., 2015] consists of 1,980 images collected from Flickr by using the emotion keywords and synonyms as search terms. There are 330 images for each emotion category. AMT workers were invited to label the images into the Ekman's 6 emotions and neutral to obtain the emotional responses. Each image was scored by 15 subjects. The discrete emotion distribution information is released.

The IESN dataset is constructed for personalized emotion prediction [Zhao et al., 2016], with 1,012,901 images from Flickr. Lexicon-based methods are used to segment the text of metadata from uploaders for expected emotions and comments from viewers for actual emotions. Synonym based searching is employed to obtain the Mikels' emotion category by selecting the most frequent synonyms. The average VAD values of the segmentation results are computed as DES ground truth based on the VAD norms of 13,915 English lemmas [Warriner et al., 2013]. 7,723 active users with more than 50 involved images are selected. The DEC and emotion distributions can also be easily obtained.

Two image datasets for discrete emotion distribution learning are released in [Yang et al., 2017b]. One is FlickrLDL dataset, a subset of FlickrCC. FlickrLDL contains 10,700 images, which are labeled by 11 viewers using Mikels' emotion model. 30,000 images are collected by searching various sentiment key words from Twitter. After duplication removal, the images are labeled by 8 viewers. In this way, the TwitterLDL dataset is generated with 10,045 images. In both datasets, the ground truth emotion distribution for each image is obtained by integrating the votes from the workers.

4 Emotion Feature Extraction

As shown in [Zhao et al., 2014c], there are various types of features that may contribute to the perception of image emotions. In this section, we introduce the hand-crafted features of different levels (Table 3) and the learning based features (Table 5) that have been widely extracted for AICA.

4.1 Low-level Features

Low-level features suffer from being difficult to be interpreted by humans. The widely used features include GIST, HOG2x2, self-similarity and geometric context color histogram features as in [Patterson and Hays, 2012], because they are each individually powerful and can describe distinct visual phenomena in a scene perspective.

Features derived from elements of art, including color and texture are extracted [Machajdik and Hanbury, 2010]. Lee and Park [2011] used the MPEG-7 visual descriptors, including four color-related ideas and two texture-related ideas. Lu et al. [2012] investigated how shape features in natural images influence emotions by modeling the concepts of roundness-angularity and simplicity-complexity. Based on Itten's color wheel, Sartori et al. [2015] designed two types of features to represent the color combinations.

4.2 Mid-level Features

Mid-level features are more interpretable, more semantic and more relevant to emotions than low-level features. 102 attributes are detected [Patterson and Hays, 2012], including 5 different types: materials, surface properties, functions or affordances, spatial envelop attributes and object presence. Besides the 102 attributes, Yuan et al. [2013] also incorporated eigenfaces corresponding to different emotions, which contribute a lot to the images containing faces. Rao et al. [2016a] extracted SIFT features as basic feature and adopted bagof-visual-words (BoVW) to represent the multi-scale blocks. The latent topic distribution estimated by probabilistic latent semantic analysis is used as another mid-level representation.

Harmonious composition is essential in an artwork and Machajdik and Hanbury [2010] extracted several features to analyze an image's compositional character. Interpretable aesthetic features [Wang et al., 2013] are designed based on the fact that artists often jointly use figure-ground relationships, color patterns, shapes and their diverse combinations to express emotions in their art creations. Features inspired from principles of art are designed in [Zhao et al., 2014a].

4.3 High-level Features

High-level features are the semantic contents contained in images. People can easily understand the emotions conveyed in images by recognizing the semantics. In the early years, Machajdik and Hanbury [2010] extracted simple semantic content by detecting faces and skins contained in an image. Facial expressions may determine the emotion of the images containing faces. 8 kinds of facial expressions are extracted as high-level features [Yang et al., 2010]. The expressions of images detected without faces are set as neutral. An 8 dimensional vector, each element of which indicates the number of related facial expressions in the image, is generated.

Borth et al. [2013] proposed to describe the semantic concepts by 1,200 adjective noun pairs (ANPs), which are detected by SentiBank. The advantages of ANP are that it turns a neutral noun into an ANP with strong emotions and makes the concepts more detectable, as compared to nouns and adjectives, respectively. A 1,200 dimensional double vector representing the probability of the ANPs is obtained.

Some representative works based on hand-crafted features of the above 3 levels are summarized in Table 4. Generally, high-level features (such as ANP) perform better for images with rich semantics, mid-level features (such as Principles) perform better for artistic photos, while low-level features (such as Elements) are effective for abstract paintings.

4.4 Learning-based Features

With the advent of deep learning, emphasis has been shifted from designing hand-crafted features to learning features in an end-to-end fashion. To tackle the weakly labeled images, You et al. [2015] proposed to progressively select a potentially cleaner subset of the training instances. An initial convolutional neural network (CNN) model is trained on the training data. According to the prediction score of the trained model on the training data itself, the training samples with distinct sentiment scores between the two classes with a high probability are selected. The trained model is fine-tuned using the newly selected instances. Later, they fine-tuned the pre-trained AlexNet on ImageNet to classify emotions into 8 categories by changing the last layer of the neural network from 1000 to 8 [You et al., 2016b]. An SVM classifier is also trained using features extracted from the second to the last layer of the pre-trained AlexNet model.

Rao et al. [2016b] proposed to learn multi-level deep representations for image emotion classification (MldrNet). The input image is segmented into 3 levels of patches, which are input to 3 different CNN models, including Alexnet, aesthetics CNN (ACNN) and texture CNN (TCNN). Multiple instance learning (MIL) is employed to generate the emotion label of an input image. Based on MldrNet, Zhu et al. [2017] integrated the different levels of features by a Bidirectional GRU model (BiGRU) to exploit their dependencies. Two features generated from our Bi-GRU model are concatenated as the final features to predict the emotions. Apart from the traditional cross-entropy loss, an additional contrastive loss is jointly optimized to enforce the feature vectors extracted from each pair of images from the same category to be close enough, and those from different categories to be far away.

To explore the correlation of emotional labels with the same polarity, Yang et al. [2018] employed deep metric learning and proposed a multi-task deep framework to optimize both retrieval and classification tasks. Besides the crossentropy loss, a novel sentiment constraint is jointly optimized by considering the relations among emotional categories in the Mikels' wheel, which extends triplet constraints to a hierarchical structure. A sentiment vector based on the texture information from the convolutional layer is proposed to measure the difference between affective images.

The deep representations generally outperform the handcrafted features, which are designed based on several smallscale datasets for specific domains. However, how the deep features correlate to specific emotions is unclear.

5 Personalized Emotion Prediction

Zhao et al. [2016; 2018d] proposed to predict the personalized emotions (see Figure 1 (d)) of a specified user after viewing an image, associated with online social networks. Different types of factors that may influence the emotion perception are considered: the images' visual content, the social context related to the corresponding users, the emotions' temporal evolution, and the images' location information. Rolling multi-task hypergraph learning is presented to jointly combine these factors. Each hypergraph vertex is a compound triple (u, x, S), where u represents user, x and S are the current image and the recent past images, termed as 'target image' and 'history image set', respectively. Based on the 3 vertex components, different types of hyperedges are constructed, including target image centric, history image set centric, and user centric hyperedges. Visual features (512dimensional GIST, Elements, Attributes, Principles, ANP, Expressions) in both target image and history image set are extracted to represent visual content. User relationship is exploited from the user component to take social context into account. Past emotion is inferred from history image set to reveal temporal evolution. Location is embedded in both target image and history image set. Semi-supervised learning is conducted on the multi-task hypergraphs to classify personalized emotions for multiple users simultaneously. The average F1 of emotion classification on the IESN dataset is 0.582.

6 Emotion Distribution Learning

According to probability theory, there are typically two types of probability distributions: discrete and continuous. Generally, distribution learning can be formalized as a regression problem. For CES, the task aims to predict the discrete probability of different emotion categories, the sum of which is equal to 1 (see Figure 2). For DES, the task usually transfers to predict the parameters of specified continuous probability distributions, the form of which should be firstly determined, such as Gaussian distribution (see Figure 1 (e)) and exponential distribution.

6.1 Discrete Probability Distribution Learning

Zhao et al. [2015] proposed shared sparse learning (SSL) to represent the probability distribution of one test image as the linear combination of the training images' distributions, the coefficients of which are learned from the feature space. Only uni-modal features are considered in SSL, which can simply adopt early or late fusion to handle multi-modal features (indicate multiple features from images unless otherwise specified) without considering the latent correlations. Multi-modal features are fused by weighted multi-modal SSL (WMMSSL) [Zhao et al., 2017a; 2018a] to explore useful information by the constraint of joint sparsity across different features. The representation abilities of different features are jointly explored with the optimal weight automatically learned.

Both SSL and WMMSSL can only model one test image each time. For different test images, the shared coefficients have to be learned repeatedly. Geng et al. [2013] proposed a conditional probability neural network (CPNN) for distribution learning by modeling the conditional probability of labels given features as a three layer neural network. Yang et al. [2017b] replaced the signless integers in CPNN with a binary representation, since adding two emotion labels or subtracting one label from another is meaningless. By adding noises to the ground truth emotion labels, more roughly labeled distributions and samples are generated. Zhao et al. [2017b] extended CPNN into multi-modal settings and proposed weighted multi-model CPNN (WMMCPNN) to jointly explore the representation abilities of different features. The linear combination of different CPNN loss functions are minimized with a sparse constraint on the combination coefficients. Once the parameters of the augmented CPNN and WMMCPNN are learned, the emotion distributions of a test image would be easily obtained.

The CPNN series have only 3 layers and the input is extracted visual features instead of original images. A deeper CNN regressor (CNNR) for each emotion category is trained in Emotion6 [Peng et al., 2015] with the exact AlexNet architecture. The number of output nodes is changed to 1 to predict a real value and the Softmax loss layer is replaced with the Euclidean loss layer. The predicted probabilities of all emotion categories are normalized to sum to 1. However, the CNNR cannot guarantee that the predicted probability is non-negative. Further, the probability correlations among different emotions are actually ignored, since the regressor for each emotion category is trained independently. Yang et al. [2017a] designed a multi-task deep framework based on VGG16 by jointly optimizing the cross-entropy loss for emotion classification and Kullback-Leibler (KL) divergence loss for emotion distribution learning, which achieves the stateof-the-art performances. For the single emotion dataset, the single label is transformed to emotion distribution with emotion distances computed on Mikels' wheel. The representative methods are summarize in Table 6.

6.2 Continuous Probability Distribution Learning

Zhao et al. [2017c] proposed to learn continuous emotion distributions in VA space. Based on the assumption that the VA emotion labels can be well modeled by a mixture of 2 bidimensional Gaussian mixture models (GMMs, see Figure 1 (e)), the task turns to predict the parameters of GMMs, including the mean vector and covariance matrix of the 2 Gaussian components as well as the mixing coefficients.

Shared sparse regression (SSR) is proposed as the learning model by assuming that the test feature and test parameters can be linearly represented by the training features and training parameters but with shared coefficients. SSR can only model one test image each time. To explore the task relatedness, multi-task SSR is further presented to simultaneously predict the parameters of different test images by utilizing appropriate shared information across tasks. GIST, Elements, Attributes, Principles, ANP, and 4,096-dimensional deep features from AlexNet are extracted as visual features. Experiments are conducted on a subset of IESN, which consists of 18,700 images each with more than 20 VA labels. The average KL divergence of multi-task SSR using ANP is 0.436.

7 Conclusion and Future Directions

This paper attempted to provide an overview of recent developments on affective image content analysis (AICA). Obviously, it cannot cover all the literature on AICA, and we focused on a representative subset of the latest methods. We summarized the widely employed emotion representation models, released datasets, and compared the existing methods on emotion feature extraction, personalized emotion prediction and emotion distribution learning. We believe that AICA will continue to be an active and promising research area with broad potential applications, such as an emotion-aware personalized music slide show [Chen et al., 2014], emotion based image musicalization [Zhao et al., 2014b], and image captioning with sentiment [Mathews et al., 2016]. Many issues in AICA, however, are still open.

Understanding image content and context. Accurately analyzing what is contained in an image can significantly improve the performance of AICA. Sometimes we even need subtle analysis of visual contents. For example, if an image is about the laugh of a lovely child, it is more likely that we feel "amused"; but if it is about the laugh of a known evil ruler or criminal, we may feel "angry". The context of image content is also important in AICA. Similar visual content under different contexts may evoke different emotions. For example, we may feel "happy" about beautiful flowers. But if the flowers are placed in a funeral, we possibly feel "sad".

Recognizing group emotions. Recognizing the dominant emotion is too generic, while predicting personalized emotion is too specific. Modeling emotions for groups or cliques of users, who share similar tastes or interests, may be a good choice - e. g., by cultural or societal background. Analyzing the user profiles provided by each individual to classify users into different types of backgrounds, tastes and interests may help to tackle this problem.

Understanding emotions of 3D data and videos. Compared with traditional intensity images, 3D data contain more spatial information, being useful in low light levels and being color and texture invariant, while videos (such as animated GIF) contain rich temporal correlation information. Combining the spatial and temporal correlation together with the visual content would make more sense.

Further, jointly exploring the complementarity of multimodal data (such as image and text) [You et al., 2016a; Zhao et al., 2018c; Chen et al., 2018] would improve the emotion recognition performance. How to adapt the emotions from one labeled domain to another unlabeled domain [Patel et al., 2015] is another interesting research topic.

Acknowledgments

This work is supported by the Berkeley Deep Drive, the National Natural Science Foundation of China (Nos. 61701273, 61571269, 61332016, 61620106009, U1636214), the China Postdoctoral Science Foundation Project (No. 2017M610897), and the European Union's Horizon 2020 Programme through the Innovative Action No. 645094 (SEWA). This research is part of NExT++ project, supported by the National Research Foundation, Prime Ministers Office, Singapore under its IRC@Singapore Funding Initiative.