Looking at Mondrian's Victory Boogie-Woogie: What Do I Feel?

1 Introduction

Throughout the centuries, Western Art was dominated by representational artworks, which were powerful tools to express

∗The title aims to provide an idea about the feeling a title of a painting may induce. The Boogie-Woogie was a cultural movement of music and dance in the late 1920s, and it is characterized by its vivacity, syncopated beat and irreverent approach to melody. This movement fascinated Mondrian as he considered it similar to his own work: "destruction of natural appearance; and construction through continuous opposition of pure means dynamic rhythm." [The Museum of Modern Art, 2015]

G¨ozde ¨Ozbal

Fondazione Bruno Kessler Trento, Italy gozbalde@gmail.com

Nicu Sebe

DISI, University of Trento Trento, Italy sebe@disi.unitn.it

feelings and ideas of artists. With the introduction of modern art movements like abstract art, this emphasis shifted and artists explored different means to evoke emotions. For example, abstract artists used the relationship between colour, shapes and textures to convey emotions in a "non-figurative" way: "artists sought to express in their work only internal truths, renouncing in consequence all consideration of external form" [Kandinsky, 1914]. It is therefore intriguing to see if we have the same feelings when we look at these paintings, and if we can train a computer algorithm to report the same.

The relationship between paintings, emotions and art appreciation has been extensively studied in many fields. Psychological studies on aesthetics and emotional responses to art have shown that titles and descriptions influence the perceptual experience of paintings. These works demonstrated that people describe paintings differently after they read the title [Franklin et al., 1993]. They also postulate that title and description of paintings can improve understanding the meaning of the artwork [Leder et al., 2006] and the aesthetic evaluation [Millis, 2001; Hristova et al., 2011]. The influence of title and description is even more crucial for abstract artworks where the visual clues are open to interpretation.

In this paper, we analyze the influence of the metadata (i.e., titles, description and/or artist's statement) associated with the abstract painting and investigate how it can be a significant feature to the automatic emotion recognition of abstract paintings. Specifically, we extract the corresponding metadata for the paintings and apply sentiment analysis to detect the emotional meaning of these texts. We use state-of-theart computer vision techniques employing colour, shapes and textures to augment computation of the emotional information of the artwork. Finally, we propose two approaches to combine textual features with visual ones: the first approach is based on weighted linear combination, and the second, we propose a novel joint flexible Schatten p-norm model. We apply our multimodal approach on two datasets of abstract paintings: (1) a collection of professional paintings from the MART Museum and (2) a collection of amateur paintings from deviantArt (dA), an online social network site designated to user-generated art.

To summarize, our main contributions are: (1) we study the contribution of metadata on positive and negative feelings induced by abstract paintings; (2) we propose a novel joint flexible Schatten p-norm model which can exploit the sharing patterns between visual and textual information for emotion analysis of paintings; (3) we apply our approach on two different datasets of abstract artworks and make a qualitative analysis. This study will contribute to various disciplines and research areas ranging from computer vision (image retrieval) to psychology and aesthetics.

2 RelatedWork

Several psychological studies provide empirical analysis of how the titles and description of artworks can affect the perception and understanding of paintings. Franklin et al. [1993] showed to the subjects the same painting with two different titles (i.e., the original title and a fabricated title), and asked them to describe the painting. They observed that the change of titles affected the interpretation of artworks. Millis [2001] compared the effects of metaphorical (or elaborative) versus descriptive titles on the understanding and the enjoyment of artworks. He found out that metaphorical titles increase the aesthetic experience more than descriptive titles, or no-titles. In a similar study, Leder et al. [2006] showed that elaborative titles increase the understanding of abstract paintings, but do not affect their appreciation. Hristova et al. [2011] analysed how the style of the paintings and the titles can influence the fixation duration and saccade amplitude of the viewers. When the viewers are presented with the title of a painting while looking at the artwork, the average number of saccades of each viewer to observe the work tend to decline. This finding draws the conclusion that titles lead to a more focal processing of the paintings. The above mentioned research demonstrate that the metadata influence the perception of artworks.

Recently there is an increase on the research focusing on emotions in artworks from the computational perspective. Yanulevskaya et al. [2008] proposed an emotion categorization system for masterpieces based on the assessment of local image statistics, applying supervised learning of emotion categories using Support Vector Machines. Their system was trained on the International Affective Picture System (IAPS), which is a standard emotion evoking image set [Lang et al., 1999], and was applied to a collection of masterpieces. Machajdik & Hanbury [2010] employed low-level features and combined with concepts from psychology and art theory to categorize images and artworks emotionally. They obtained better accuracy in affective categorization of semantically rich images than abstract paintings. Yanulevskaya et al. [2012] trained a Bag-of-Visual-Words model to classify abstract paintings in positive or negative emotions. With the backprojection technique the authors determined which parts of the paintings evoke positive or negative emotions. Recently, Zhao et al. [2014] extracted emotion features of images from IAPS dataset of [Lang et al., 1999] based on the principles of art, including balance, emphasis, harmony, variety, gradation, and movement. They concluded that these features can improve the performance of emotion recognition in images. In this work, we use metadata as an additional feature to improve the affective classification of abstract paintings.

Various computational approaches have shown the importance of detecting emotion in textual information [Strapparava and Mihalcea, 2008; Balahur et al., 2011]. Sentiment analysis techniques using computational linguistics to extract the human sentiment in text, is increasingly used in many areas, such as marketing and social networks to improve their products and/or services. Recent works, including [Hwang and Grauman, 2012; Gong et al., 2014], empathize the importance of using textual information associated with images for creating stronger models for image classification. Liu et al. [2011] collected textual information from Internet in the form of tags describing images. They applied two different methods to extract emotional meaning from these tags and combined them with visual features. This study demonstrated how textual information associated with images can improve affective image classification. In our work, we use a multimodal approach to investigate how the title, descriptions and/or artist statements associated with paintings can lead to a stronger model to perform affective analysis of abstract paintings.

3 Datasets and Ground Truth Collection

We conduct our analysis on two sets of abstract paintings, a professional and an amateur one. 1 Both are composed of 500 abstract paintings. In the following subsections we detail the selection process as well as the nature of each dataset.

3.1 MART Dataset: ADataset of Professional Abstract Paintings

The MART dataset is a set collected in our previous work [Yanulevskaya et al., 2012; Sartori et al., 2015] from the electronic archive of the Museum of Modern and Contemporary Art of Trento and Rovereto (MART). The selected paintings are masterpieces of abstract art, which were created by a total of 78 artists between 1913 and 2008. Most of these artists are Italians, however there are also European and American artists. Some of the artists, such as Wassily Kandinsky, Josef Albers, Paul Klee and Luigi Veronesi, are known for their studies on abstract art in terms of colour, shapes and texture. As most of the artists are from Italy, the titles are in Italian. The available descriptions or the statements of the artists about these paintings are also in Italian.

3.2 deviantArt Dataset: ADataset of Amateur Abstract Paintings

The amateur collection of abstract paintings has been collected from the deviantArt (dA) website 2 , an online social network dedicated to user-generated art. This set was collected in our previous work [Sartori et al., 2015]. dA is one of the largest online art communities with more than 280 million artworks and 30 million registered users. We selected the artworks that were under the category Traditional Art/Paintings/Abstract, and downloaded 8,000 artworks. We used the information reflecting how many times an artwork was favorited as a parameter to downsize the collection from 8000 to 500 paintings 3 . We selected the paintings with the highest and least number of favourites, as well as some paintings randomly in the middle. The collection thus has 500 paintings with 406 different authors. The titles and the artist descriptions in this collection are in English.

1 The datasets with their respective ground truths are publicly available at: http://disi.unitn.it/∼sartori/datasets/

2 http://www.deviantart.com/

3.3 User Study to Analyse Emotions Evoked by Artworks

To collect our ground truth we use the relative score method from our previous work [Sartori et al., 2015] in which we asked people to choose the more positive painting in a pair. The annotation was done online and we provided the following instructions to each annotator: "Which painting in the pair looks more positive to you? Let your instinct guide you and follow your first impression of the paintings."

To annotate and calculate the emotional scores from the paintings we follow the method of TrueSkill ranking system [Herbrich and Graepel, 2006; Moser, 2010]. Developed by Microsoft Research for Xbox Live, the TrueSkill ranking system recognizes and ranks the skills of the players in a game and matches players with similar skills for a new game. With this method the annotation task is more manageable, as it yields a representative annotation with only 3,750 pairs of paintings, instead of 124,750 comparisons (500 ∗ (500 - 1) ∗ 0.5), if each painting is compared with all the remaining paintings in the dataset.

During the annotation process, all paintings are initially considered with the same 'skill' and the painting which is chosen as more positive in a single trial wins a 'game'. Then, the rankings of the compared paintings are updated. Afterwards, the paintings with similar rankings are compared, until each painting is compared with at least 15 other paintings. We consider the results as emotional scores of the paintings, in which lower values correspond to negative feelings and the higher values to positive feelings.

25 subjects (11 females, 14 males) participated in the annotation of the MART dataset. Each person annotated from 19 to 356 pairs of paintings, with a mean of 145. For the annotation of the deviantArt dataset 60 people participated (27 females and 33 males). Each participant annotated from 2 to 436 pairs of paintings, with a mean of 63. The participants were anonymous and they participated in this annotation voluntarily, without getting any reward. There was no time-limit to the annotation procedure: each participant was free to annotate whenever he/she had time to do it.

To compose the ground truth of both datasets, a threshold was defined to separate the paintings in positive and negative emotions. As each painting was compared 15 times, we assume that the threshold should be related to one of the paintings chosen 8 times as more positive. Therefore, we consider the TrueSkill ranking values of these paintings, which are the ratings resulted from the comparisons of paintings by using the TrueSkill algorithm. The paintings with TrueSkill

3 We selected 500 paintings to construct the deviantArt dataset in order to make an impartial comparison with the MART dataset.

ranking equal to or lower than the threshold are defined as negative and paintings with ranking higher than the threshold are defined as positive. For MART dataset, the painting with TrueSkill ranking value equal to 25.1 is used as threshold. In total, we obtain 131 paintings in the negative class and 369 in the positive class. To compose the ground truth for deviantArt dataset, we considered as threshold the painting with TrueSkill ranking equal to 21.0. As a result, 140 paintings were assigned to the negative class and 360 to the positive class.

4 Textual Features for Assessing Emotions in Abstract Paintings

In this section we describe how the sentiment features were extracted from the metadata (i.e., titles, descriptions, artist statement) associated with a painting.

4.1 Text Data Selection

To obtain the descriptions and/or artist statements for MART dataset, we searched through museums and art related websites, as well as the official pages of the artists. In total, we found descriptions and/or artist statements for 158 paintings. Most of these descriptions were in Italian, as the artists were Italian. The titles of the paintings were already provided by the dataset. In deviantArt dataset, the artists themselves provided an opinion and/or a description on/about their paintings. From those, we selected 158 descriptions which represent the artists' intention in detail. The titles and descriptions of this dataset are in English. Figure 1 provides two examples of titles and descriptions for paintings in the MART (top) and deviantArt (bottom) datasets.

4.2 Applying Sentiment Analysis

To determine the sentiment conveyed by the title and the description of a painting in the deviantArt dataset, we tokenize and part-of-speech (POS) tag the related text with the Stanford Parser [Klein and Manning, 2003], and we use WordNet [Miller, 1995] for lemmatization. For each title or description we simply average the scores coming from SentiWords [Guerini et al., 2013] based on the lemma and POS information. SentiWords is a high coverage resource associating ∼155,000 English lemma-POS pairs to sentiment scores between -1 (very negative) and 1 (very positive).

As (to the best of our knowledge) there is no resource for sentiment scores in Italian, we utilize a multilingual lexical database called MultiWordNet 4 [Artale et al., 1997] for the painting descriptions in the MART dataset. More specifically, we first lemmatize and POS tag the Italian titles and descriptions with TextPro [Pianta et al., 2008]. Then, by using MultiWordNet we find all the English synsets associated with the Italian lemma and POS pairs. We calculate the sentiment score of a lemma-POS pair as the average of the scores provided in SentiWordNet [Baccianella and Sebastiani, 2010] for each synset that it is connected to. SentiWordNet is a lexical resource built for supporting sentiment classification and opinion mining applications and it provides three sentiment scores (i.e., positivity, negativity, objectivity) for each WordNet synset. We consider the difference between the positivity and negativity score as the final sentiment score of a synset. As we did for deviantArt dataset, we average the scores of each lemma-POS pair to determine the overall emotion conveyed by a painting title or description.

5 Fusion of Visual and Textual Features

In this section we describe the classification approach to automatically estimate the emotional valence of a given painting, and provide details on how we used both late fusion and joint flexible Schatten p-norm learning to combine visual and textual features.

5.1 Visual Features

To extract the visual features, we follow our previous work [Yanulevskaya et al., 2012; Sartori et al., 2015]. We use a standard bag-of-words paradigm to extract Colour based LAB Visual Words and Texture Based SIFT Visual Words. We train a Support Vector Machine with a histogram intersection kernel for supervised learning. To test the approach, a 5-fold cross-validation setup is used, where the images are assigned to folds randomly. We run two separate frameworks (Lab descriptors and SIFT descriptors) and then average the scores to combine them.

5.2 Late Fusion withWeighted Linear Combination

To combine visual and textual features we use late fusion with Weighted Linear Combination. To calculate the Weighted Linear Combination we use the following equation:

4 http://multiwordnet.fbk.eu/

where T is the sentiment score from text and V is the visual score (the fusion of LAB and SIFT Visual Words) and a ∈ [0, 1] is a parameter. To choose the parameter a we analyze the performance of the decision fusion (Visual + Text) and set a to the value resulting in the highest mean performance.

5.3 Estimating Paintings' Emotion through Joint Flexible Schatten p-norm Learning

The drawback of late fusion with weighted linear combination approach is that it considers visual and texture information separately before classification. This may not capture well the emotion of the viewers about a painting because the viewers usually judge a painting by looking at it and considering its title and available description at the same time. To better exploit the visual and textual information for the emotion analysis of a painting, we propose a novel joint flexible Schatten p-norm model.

Assuming that xi is a d-dimensional feature vector, Xa = {x1, x2, ..., xna } ∈ IRd×na is the feature matrix from visual information and Xb = {x1, x2, ..., xnb } ∈ IRd×nb is the corresponding feature matrix from textual information. Ya = {y1, y2, ..., yna } ∈ IRm×na and Yb = {y1, y2, ..., ynb } ∈ IRm×nb are label information matrices with m classes. yi = [0, ..., 0, 1, 0, ..., 0] (the position of non-zero element indicates the emotion class). To extract the painting's emotion both from visual and texture information, we propose the following optimization problem:

where U ∈ IRm×d are the common patterns shared across visual and textual information. Va ∈ IRm×d and Vb ∈ IRm×d are the individual patterns for visual and textual information respectively. In Eqn.(2), the first two terms are loss functions and the regularization term ‖U‖2F is used for preventing overfitting. One way to capture the relationship is to constrain the models from both visual and textual information to share a low-dimensional subspace resulting in the rank minimization problem. The last term of rank regularization captures the correlation of the visual and textual information. However, the rank minimization problem is in general NP-hard. One popular approach is to replace the rank function by the trace norm (or nuclear norm).

Inspired by [Nie et al., 2012; Yan et al., 2013; 2015; 2014], we adopt the Schatten p-norm instead of the traditional trace norm to discover the low rank matrix. The Schatten p-norm (0 < p < inf) of a matrix M is defined as ‖M‖Sp = (Trace((M′M) p2 )) 1p . If p = 1, it reduces to the trace norm. Eqn.(2) can be reformulated as:

Based on [Nie et al., 2012], ‖V‖pSp = trace(V′VD), where D is the diagonal matrix D = 2p (V′V) p-2 2 . Then if we define V = [Va; Vb] and Y = [Ya; Yb], Eqn.(2) becomes:

Optimization: Since Eqn.(4) involves the Schatten p-norm which is non-smooth and cannot be solved in a closed form, we adopt the alternating minimization algorithm to optimize the objective function with respect to U and V respectively in the following steps:

(I). Fix V, optimize U. By setting the derivative of Eqn.(4) w.r.t. U to zero, we have:

(II). Fix U, optimize V. By setting the derivative of Eqn.(4) w.r.t. V to zero, we have:

According to the optimization, we propose Algorithm 1 to solve the objective function of Eqn.(4).

Algorithm 1: An efficient iterative algorithm to solve the optimization problem in Eq.(4).

Input:

The training data Xa ∈ IRd×na , Xb ∈ IRd×nb ; The training data labels Ya ∈ IRm×na , Yb ∈ IRm×nb ; Parameters λ1, λ2, 0 ≤ p ≤ 2. Output: Optimized U, V ∈ IRm×d . 1: Set t = 0, randomly initialize Dt ∈ IRd×d ; 2: repeat Fix V, Optimize U: Ut = (YXT - VtXXT )(XXT + λ1I)-1 ; Fix U, Optimize V: Vt = (YXT -UtXXT )(XXT + λ2Dt)-1 ; Update Dt+1 = p2 (VT t+1Vt+1) p-2 2 ; t = t + 1. until Convergence;

6 Experiments

In this section we present the experimental results on the classification of the artworks with their corresponding textual information into positive and negative emotions.

6.1 Setup

For the late fusion with the weighted linear combination strategy, the weight a is tuned within the range {0.1, 0.2, 0.3, ..., 1}. For the joint flexible Schatten p-norm model, the parameters λ1 and λ2 are tuned within the range {10-3 , 10-2 , ..., 103} and p is tuned within the range {0.01, 0.05, 0.1, 0.2, 0.5, 0.8, 1}. 5-fold cross-validations are used in all experiments and the best performance is reported.

6.2 Results

Since the metadata associated with a painting contains title and description, we perform three types of experiments: (i) visual + title; (ii) visual + descriptions; (iii) visual + title + descriptions. Table 1 shows the classification accuracy results using only title information. Since there are descriptions available only for 158 paintings, we consider this set to perform the experiments (ii) and (iii). Table 2 shows the classification accuracy results based only on descriptions information and Table 3 shows the classification accuracy results based on both paintings' title and description information. From the results, it can be observed that the combination of the textual and visual features improves the performance of the classification. Moreover, we observe that more than 3% improvement in accuracy is achieved by sharing patterns between visual and textual information through the joint flexible Schatten p-norm model compared with the late fusion strategy, which shows the effectiveness of our proposed method.

With these results, we investigated in which cases the title improves the emotional classification of abstract paintings. The textual features increase or even correct the classification results mostly in paintings displaying mixed visual cues. For example, light colors and smooth shapes generate a positive emotion, whereas dark colors and sharp edges generate a negative emotion [Sartori et al., 2015]. The paintings with both light and dark colors, and/or smooth and sharp shapes are harder to classify. In such cases using titles and descriptions is expected to increase the classification accuracy.

Figure 2 provides example of paintings together with their respective titles from each dataset. These paintings (MART dataset on the left, dA dataset on the right) are classified as positive when we use only the visual features. However, they are classified as negative when we use only the title information. When we apply the late fusion of visual and textual features, the results are consistent with the ground truth, classifying these paintings as negative. We also observe that in deviantArt 0.717 ±0.024

0.658 ±0.031

0.725 ±0.033 0.751 ±0.021

Figure 2: Two examples where title helped the classification algorithm. On the left: MART dataset, Spazio inquieto (Restless Space), 1953 - Emilio Vedova Courtesy of MART photographic archive, Rovereto.) On the right: deviantArt dataset, Conflict, 2013 - Georgiana Beligan, phylactos.deviantart.com (Courtesy of deviantArt.)

both datasets, the paintings for which the descriptions helped the classification are mostly considered positive by people. Examples of such paintings from each dataset are provided in Figure 3. Both paintings, as they are composed of dark colours (dark green and dark blue) are considered as negative when using only visual features. However, when the descriptions are combined with the visual features the outcome is positive, which is consistent with human annotation.

By generally comparing both datasets, we observe that adding text as a feature is more effective for deviantArt than for MART. One reason may be that the descriptions in deviantArt are made by the artists themselves. Besides, only a few paintings are untitled in deviantArt (only 1%). On the contrary, in MART 10.4% of the paintings have no titles and the descriptions are usually made by others (e.g., art historians, art critics).

Finally, we also study the parameter sensitivity of the proposed method as demonstrated in Fig.4. Here, we fix λ1 = 1 and analyze the regularization parameters λ2 and p. As shown in Fig.4, the proposed method is more sensitive to p compared with λ2, which confirms the importance of flexibility of our proposed Schatten p-norm model.

7 Conclusions

In this paper we showed that the metadata associated with abstract paintings is a useful addition to automatically identify the positive and negative emotions evoked by these paintings. We proposed a multimodal approach, in which we employed computer vision techniques and sentiment analysis to learn statistical patterns correlated to the valence emotions conveyed by abstract paintings. We proposed two approaches to combine visual and textual features. The first is late fusion based on weighted linear combination and the second is a novel joint flexible Schatten p-norm model which can exploit the sharing patterns between visual and textual information. We evaluated our method in two datasets of abstract artworks consisting of professional and amateur paintings respectively. We observed that using textual features (i.e., titles and artist descriptions) improves the accuracy in both approaches of combination.

A qualitative analysis showed that the combination of metadata associated with the paintings can help computers to efficiently recognize positive and negative emotions in the cases when the painting composition is mixed, for instance, when the painting is composed by dark colours (which conveys negative emotions) and light colours (which conveys positive emotions) at the same time.

Acknowledgments:

This work was partially supported by the MIUR Cluster project Active Ageing at Home, the EC project xLiMe and A*STAR Singapore under the Human-Centered Cyberphysical Systems (HCCS) grant.