Towards Context Aware Emotional Intelligence in Machines: Computing Contextual Appropriateness of Affective States

1 Introduction

In recent years there has been a rising tendency in Artificial Intelligence research to enhance Human-Computer Interaction (HCI) by humanizing machines making them more userfriendly [Treur et al., 2007]. However, to create a robot capable of acting and talking with a user on the human level requires procedures allowing the robot to understand human cognitive behaviors. Such robots could be very valuable as intelligent companions for humans, e.g. helping children with their development [Rzepka et al., 2008] or helping drivers not to fall asleep during long rides home [Takahashi et al., 2003]. Often described as a vital part of human intelligence, one of the most important cognitive behaviors in humans is expressing and understanding emotions [Salovey and Mayer, 1990].

Therefore, one of the current issues in Artificial Intelligence is to produce methods for efficient processing of emotional states in humans. The field embracing this subject, Affective Computing, is a rather young discipline of study which has been gathering popularity among researchers since its introduction around ten years ago [Picard, 1997]. This research is usually focused on recognizing the emotions of users in HCI. In the most popular methods, emotions are recognized using facial expressions [Hager et al., 2002], voice [Kang et al., 2000], or biometric data [Teixeira et al., 2008]. However, these methods, usually based on a behavioral approach, ignore the semantic and pragmatic context of emotions. Therefore, although they achieve good results in laboratory settings, such methods lack usability in real life. A system for recognition of emotions from facial expressions, assigning "sadness" when a user is crying would be critically mistaken, if the user was, e.g., cutting an onion in the kitchen.

This leads to the need to apply contextual analysis to emotion processing. Furthermore, although recent discoveries prove that affective states should be analyzed as emotionspecific [Lerner and Kelter, 2000], most of the behavioral approach methods simply classify them to opposing pairs such as joy-anger, or happiness-sadness [Teixeira et al., 2008]. A positive change in this tendency can be seen in text mining and information extraction approaches to emotion estimation [Tokuhisa et al., 2008]. However, the lack of standardization often causes inconsistencies in emotion classification.

In this paper we present a method capable of specifying users' emotional states in a more sophisticated way than simple valence classification. This method contributes to standardization of the classification of emotions expressed in the Japanese language since it does not propose a new classification of emotions, but uses the most reliable one available today. Finally, our method does not only specify what type of emotion was expressed, but also determines whether the expressed emotion is appropriate for the context it appears in.

2 Emotional Intelligence

The idea of Emotional Intelligence (EI) was first officially proposed by Salovey and Mayer [Salovey and Mayer, 1990], who defined it as a part of human intelligence responsible for the ability to: I) perceive emotions; II) integrate emotions to facilitate thoughts; III) understand emotions; IV) regulate emotions. In the EI Framework [Mayer and Salovey, 1997] the first step is divided into a) the ability to identify emotions and b) discriminate between accurate (appropriate) and inaccurate (inappropriate) expressions of emotion. This is the key ability for the third step in EI Framework: understanding and analysis of emotional states.

Salovey and Mayer argue that recognizing emotions is only the first basic step to acquire full scope of Emotional Intelligence and does not tell us anything about whether it is appropriate for a given situation, and what actions should be undertaken as a reaction [Salovey and Mayer, 1990]. Solomon So93 argues that the valence of emotions is determined by the context they are expressed in. For example, anger can be warranted (a reaction to a direct offence) or unwarranted (scolding one's own children unjustly) and the reactions should be different for the two different contexts of anger. Not taking this fact into consideration in the grand majority of emotion processing research can drive us to the conclusion that such research assumes that positive emotions are always appropriate and negative ones inappropriate.

The attempts to implement the EI Framework usually do not go beyond theory [Andre et al., 2004], and the few practical attempts eventually still do not go beyond the first basic recognition step [Picard et al., 2001]. This paper presents an attempt to go beyond this simple approach. Following affective state recognition, the appropriateness of those states is checked against their contexts. This represents a step forward in the implementation of EI framework.

3 Linguistic Approach to Emotions

The semantic and pragmatic diversity of emotions is best conveyed in language [Solomon, 1993]. Therefore we designed our method to be language-based. There are different linguistic means used to inform other interlocutors of emotional states. The elements of speech used to convey emotive meaning in sentence make up the feature of language called emotiveness [Stevenson, 1963], which is described by the emotive function of language [Jakobson, 1960].

The emotive function of language is realized lexically in Japanese through such parts of speech as exclamations [Beijer, 2002], hypocoristics (endearments), vulgar language [Crystal, 1989; Potts and Kawahara, 2004] and mimetic expressions (gitaigo 1 ) [Baba, 2003]. A key role in expressing emotions is also played by the lexicon of words describing emotional states [Nakamura, 1993]. The analysis of elements of language such as intonation or tone of voice, in the communication channel limited to transmission of lexical symbols, must focus on its textual manifestations, like exclamation marks or ellipsis.

4 Definition and Classification of Emotions

Our working definition of emotions is based on the work of Nakamura [Nakamura, 1993], who defines them as every temporary state of mind, feeling, or affective state evoked by experiencing different sensations. This definition is complemented by Solomon's, who argues that people are not passive participants in their emotions, but rather the emotions are strategies by which people engage with the world. Since we operate on language, the above is further complemented by Beijer's definition of emotive utterances, which he describes as every utterance in which the speaker is emotionally involved, and this involvement, expressed linguistically, is informative for the listener [Beijer, 2002].

1 Italics are used to indicate Japanese expressions

Table 1: An example of an analysis performed by ML-Ask. Top line: example in Japanese; middle: emotive information annotation; lower: English translation. Emotive elements (ML) - underlined, emotive expressions (MX) - bold type font. Other information omitted (ellipsis).

Kyo wa nante kimochi ii hi nanda ! . . . [ML:nante] [MX:joy] . . . [ML:nanda] [ML:!] (Translation: Today is such a nice day!)

Nakamura [Nakamura, 1993], after a thorough study on emotions in the Japanese language, proposed a classification of emotions into 10 types - most appropriate for Japanese. That is: ki, yorokobi (joy, delight), do, ikari (anger), ai, aware (sorrow, sadness), fu, kowagari (fear), chi, haji (shame, shyness, bashfulness), kou, suki (liking, fondness), en, iya (dislike, detestation), kou, takaburi (excitement), an, yasuragi (relief) and kyou, odoroki (surprise, amazement).

5 ML-Ask

Based on the linguistic assumptions described above, Ptaszynski et al. constructed ML-Ask, a system for analyzing the emotive content of utterances [Ptaszynski et al., 2008]. This system uses a two-step procedure: 1) Analyzing the general emotiveness of utterances by calculating the emotive value; 2) Recognizing the particular emotion types in utterances described as emotive. ML-Ask is based on Ptaszynski's idea of two-part classification of realizations of emotions in language into: 1) Emotive elements (ML); indicate that emotions have been conveyed, but do not detail what specific emotions there are. This group is realized by such subgroups as interjections, mimetics, and vulgar language. Examples are: sugee (great!), wakuwaku (heart pounding), - yagaru (a kind of verb vulgarization); 2) Emotive expressions (MX); parts of speech that describe emotional states in emotive sentences. This group is realized by such parts of speech as nouns, verbs or adjectives. Examples are: aijou (love), kanashimu (sadness), and ureshii (happiness).

The emotive element database was built using data from different research [Baba, 2003; Potts and Kawahara, 2004; Oshima-Takane et al., 1995-1998; Sj ¨obergh and Araki, 2008] and divided into interjections, mimetics, endearments, vulgarities, and representations of non-verbal emotive elements, such as exclamation mark or ellipsis. The emotive expression database contains Nakamura's collection [Nakamura, 1993].

5.1 Affect Analysis Procedure

For a textual input provided by the user, two features are calculated in order: emotiveness of an utterance and the specific type of emotion. Firstly, the system searches for emotive elements in the utterance to determine the emotiveness (emotive / non-emotive). Secondly, in utterances described as emotive, the system searches for emotive expressions to determine the specific type of the conveyed emotions. An example of analysis performed by ML-Ask is shown in Table 1.

5.2 Evaluation of ML-Ask

An evaluation experiment was performed to verify the system's performance. The evaluation was based on a corpus of natural utterances gathered through an anonymous survey. 30 people from different age and social groups participated in the survey. Each of them was asked to imagine or remember conversations with persons they know and write three sentences from those conversations: one free, one emotive, and one non-emotive. After that, the each participant tagged their utterances in the same way using the system's method: they first determined whether free utterance was emotive and specified the emotion type of all emotive utterances. The corpus was then tagged in the same way by third party human evaluators (average of 10 people per sentence) to determine then base human level for recognizing emotions from the text. ML-Ask then analyzed this corpus of 90 sentences, and the results of the system were compared to the participants' tags.

Emotive / Non-emotive

The system's accuracy in determining the emotiveness was calculated as the approximated balanced F-score for the accuracies on emotive and non-emotive utterances separately. We also calculated the system's agreement with the human standard. The overall accuracy of ML-Ask for this task evaluated on 90 items was approximated F=.84, with F for emotive annotation =.85 (Precision=.95, Recall=.76) and F for nonemotive =.84 (P=.75, R=.95). The same value calculated for the third party human evaluators gave a wide range of results from .4 to .86. ML-Ask was placed close to the top of this ranking and therefore we can say that the system recognizes emotiveness on a very high human level. This was confirmed by the agreement calculation, in which the system achieved a good score of kappa=.7.

Specified Emotion Types

The system can potentially extract up to ten emotion types for one utterance. However, some of them can be extracted wrongly, and there is a possibility that there would still be some emotion types left unextracted. Therefore we calculated the system's results as balanced F-score with emotive tags added by authors of the utterances as gold standard. The system's accuracy in estimating the specific types of emotions including "non-emotive" reached F=.45 (P=.62, R=.35) of balanced F-score. Since for human evaluators the average accuracy was .72 (P=.84, R=.64), the system's accuracy was 62.5% (.45/.72) of the human level.

6 WebMining Technique

To verify the appropriateness of the speaker's affective states we applied Shi's Web mining technique for extracting emotive associations from the Web [Shi et al., 2008]. This web mining technique performs common-sense reasoning about what emotions are the most natural to appear in a context of the utterance, and which emotions should be associated with it. Emotions expressed, which are unnatural for the context are perceived as inappropriate. The technique is composed of three steps: a) phrase extraction from an utterance; b) morpheme modification; c) extraction of emotion associations.

6.1 Phrase Extraction Procedure

An utterance is first processed by MeCab, a tool for part-ofspeech analysis of Japanese [Kudo, 2001]. Every element separated by MeCab is treated as a unigram. All unigrams are grouped into larger n-gram groups preserving their word order in the utterance. The groups are arranged from the longest n-gram (the whole sentence) down to all groups of trigrams.

6.2 Morpheme Modification Procedure

In the list of n-gram phrases the ones ending with a verb or an adjective are then modified grammatically modified in line with Yamashita's research [Yamahsita, 1999]. He argues that Japanese people tend to convey emotive meaning after causality morphemes. This research was independently confirmed experimentally by Shi et al. [Shi et al., 2008]. They distinguished eleven emotively stigmatized morphemes for the Japanese language using statistical analysis of Web contents. They performed a cross reference of appearance of the eleven morphemes with the emotive expression database (see section 5) using the Google search engine. This provided the results (hit-rate) showing which of the eleven causality morphemes were the most frequently used to express emotions. They came to the conclusion that for the five most frequent morphemes, the coverage of Web mining procedure still exceeds 90% of all content. Therefore for Web mining they decided to use only those five most emotively stigmatized causality morphemes, namely: -te, -to, -node, -kara and -tara (see Table 2).

6.3 Emotion Types Extraction Procedure

In this step the modified phrases are used as a query in Google search engine with 100 snippets for one morpheme modification per query phrase. This way a maximum of 500 snippets for each queried phrase is extracted from the Web and crossreferenced with the database of emotional expressions (see Figure 2). The emotive expressions extracted from the snippets are collected, and the results for every emotion type are listed in a descending order. In this way a list of emotions associated in a common-sense manner with the queried sentence is obtained (an example is shown in Table 3).

6.4 Evaluation of WebMining Technique

To evaluate the Web mining technique we used the same collection of utterances as in evaluation of ML-Ask. However, as mentioned above, the Web mining technique is meant not to recognize the emotions of a particular user, but rather to find a general common sense about what emotion should be expressed in a particular utterance. Therefore, here, we use the emotions tagged by the third party evaluators as the gold standard.

Table 3: Example of emotion association extraction from the Web.

For a result to be correct, at least one of the two conditions was necessary: 1) one or more of the extracted emotive associations belonged to the group of emotion types tagged by the third party evaluators; 2) the extracted emotive associations agreed with the majority of the human-generated tags. Under these conditions, Shi's Web mining technique attained an accuracy rate of 72% when extracting emotive associations from the Web.

7 Two-dimensional Model of Affect

According to Solomon [Solomon, 1993], people sometimes misattribute the specific emotion types, but they rarely misattribute their valence. One would, for example, confuse such emotions as anger and irritation, but it is unlikely they would confuse admiration with detestation. Therefore, we checked if at least general features matched even when specific emotion types did not match perfectly with the emotive associations. By "general features" we refer to those proposed by Russell [Russell, 1980] in the theory of the two-dimensional model of affect. He argues that all emotions can be described in a space of two dimensions: valence and activation. An example of positive-activated emotion would be excitement, while a positive-deactivated emotion would be relief. Examples of negative-activated and negative-deactivated emotions would be anger and gloom, respectively (see Fig. 3).

Nakamura's emotion types were mapped onto Russell's model and their affiliation to one of the spaces was determined. The emotion types for which the affiliation is not obvious (e.g. surprise can be both positive as well as negative) were placed in all groups they could belong to. However, no emotion type was mapped onto more than two adjacent fields. These groups are then used for estimating whether the emotion types tagged by ML-Ask belong to the same space, even if they do not perfectly match the emotive associations gathered from the Web.

8 Appropriateness Verification Procedure

The two above-described systems are combined as follows: when ML-Ask discovers an emotive utterance and successfully specifies an emotion type, the Web mining technique begins the process of verification as to whether the expressed emotion type is appropriate to be used in such context. The emotions commonly associated with the context of the utterance are extracted from the Web and the emotions reported by ML-Ask from the user utterances are compared to the Web mining results. If the two match, the emotions expressed by the user are determined to be appropriate for the context. In such situations, a conversational agent equipped with this system could choose a dialogue strategy that sympathizes with the user (two examples are presented below).

Positive-appropriate emotion:

User: I'm so happy I passed the exam!

ML-Ask: joy; Web mining: joy, excitement Agent: Yeah! That's great!

Negative-appropriate emotion:

User: I hate him for making a fool of me in front of everyone.

ML-Ask: dislike; Web mining: anger, dislike

Agent: You have a reason to be angry.

However if the verification procedure indicates that the expressed emotion is inappropriate for the context, the agent could undertake different measures, e.g. helping the user to manage their emotions. Two examples are shown below.

Positive-inappropriate emotion:

User: I'd be happy if he died for cancer!

ML-Ask: joy; Web mining: fear, sadness

Agent: Are you sure that is what you are really feeling?

Negative-inappropriate emotion:

User: I won the prize, but I feel so bored.

ML-Ask: dislike, depression; Web mining: excitement, joy Agent: You should be happy!

9 Evaluation

To evaluate our method we performed an experiment. In the experiment we used two different conversational agents as interlocutors for human users. First one, Modalin, is a non-task oriented keyword-based conversational agent, which uses modality to enhance Web-based propositions for dialogue.

This agent was developed by Higuchi et al. [Higuchi et al., 2008]. The second agent, Pundalin, is a non-task oriented conversational agent created by combining Modalin with Dybala's Pun generating system PUNDA [Dybala et al., 2008]. Pundalin therefore is a humor-equipped conversational agent, which uses puns to enhance communication with the user.

13 users (undergraduate students) performed a ten-turn conversation with both agents. All conversations were analyzed by ML-Ask. 6 of the 26 conversations contained no specific emotional states and were excluded from the further evaluation. For conversations containing sentences described by ML-Ask as emotive with a specified emotion type, the Web mining procedure was carried out to determine whether the emotions expressed by the user were contextually appropriate. All the results were stored and a questionnaire was designed to evaluate how close they were to human thinking. One questionnaire set consisted of one conversation record and three questions inquiring 1) what was the valence of emotions conveyed in emotive utterances; 2) what were the specific emotion types; and 3) whether these were contextually appropriate. Every questionnaire set was answered by 10 different people (undergraduate students unrelated to the previous group). For every conversation set we calculated how many of them confirmed the system's results. The evaluated items were: A) general valence, B) specific emotion types detection accuracy in ML-Ask, C) the accuracy of the whole system at detecting contextual appropriateness of specific emotion types and D) for valence-activation dimensions.

9.1 Results and Discussion

The Achilles' Heel in the majority of research on emotion processing is the process of evaluation. It is usually performed by a small number of evaluators, sometimes only one [Tokuhisa et al., 2008]. To overcome this in our research, we used 10 independent evaluators for each agent conversation. Additionally, we provided two interpretations of the results. The primary, rigorous interpretation, assumes that the results are correct when at least 4 of 10 participants confirm the systems' results. This is a fair ratio when we consider that it means that at least four people of ten had to provide exactly the same results as the system - a difficult thing to achieve in the emotion processing research, because of many ambiguities that go along with expressing emotions. Therefore, we also allowed for the traditional approach, which assumes that if at least one human provides the same results as a machine, the machine performed the task on a human level. Regardless of the approach to evaluation, the survey provided many positive results. Firstly, in most cases the results of ML-Ask's affect analysis were confirmed. The primary evaluation of this system by Ptaszynski et al. [Ptaszynski et al., 2008] was performed on a small collection of 90 utterances (see section 5.2). The evaluation "in the field" defined ML-Ask as fully operational.

Since one of the agents was using humorous responses we also checked whether the jokes influenced the humancomputer interaction. Most of the emotions expressed in the conversations with Pundalin were positive whereas for Modalin most of the emotions were negative (see Table 5), which confirms that users tend to be positively influenced by the use of jokes in conversational agents.

There were as many as 7 cases in which there was a perfect agreement with all 10 human evaluators. In conversations with Modalin ML-Ask two times reached perfect agreement in both valence and emotion-specific determination. As for Pundalin, perfect agreement cases appeared in valence determination and, which is more important, in both aspects of determining about appropriateness of emotions. These results are shown in Table 4.

10 Conclusions and Future Work

In this paper we introduced a method for verifying contextual appropriateness of emotions conveyed in conversations. Most of the popular methods focus only on a simple emotion recognition ignoring the complexity and the context dependence of emotions. However, to create a machine capable to communicate with a user on a human level, there is a need to equip it with Emotional Intelligence Framework [Mayer and Salovey, 1997]. The method described in this paper makes a step towards practical implementation of this framework, by providing machine computable means for verifying whether an emotion conveyed in a conversation is contextually appropriate. Our method uses affect analysis system to recognize user's emotions and a Web mining technique to verify their contextual appropriateness. In a rigorous evaluation, the affect analysis method was evaluated at 75% of accuracy in determining both valence and the specific emotion types. The accuracy of determining contextual appropriateness of emotions was 45% for specific emotion types and 50% for valence. The result, although not perfect, is very encouraging, since the same evaluation performed with lenient conditions used popularly in the field gave the results of 80-85%. An agent equipped with our system can be provided with hints about what communication strategy would be the most desirable at any point. For example, a conversational agent can choose to either sympathize with the user or to take precautions and help themmanage their emotions.

We were able to prove that computing emotions in a more sophisticated manner than simple division of positive and negative is a feasible task. Although the system as a whole is still not perfect and its components (ML-Ask and the Web mining technique) need improvement, it defines a new set of goals for Affective Computing. The computation of contextual appropriateness of emotional states is a key task on the way to full implementation of Emotional Intelligence in machines and as such is valuable to the research of Artificial Intelligence in general.

Acknowledgments

This research is partially supported by a Research Grant from the Nissan Science Foundation and The GCOE Program founded by Japan's Ministry of Education, Culture, Sports, Science and Technology.