Affect Sensing in Metaphorical Phenomena and Dramatic Interaction Context

1 Introduction

It is inspiring and challenging to produce an intelligent agent who is capable of conducting drama performance, interpreting social relationships, context, general mood and emotion, sensing or reasonably predicating others' interconversion, identifying its role and participating intelligently in open-ended improvisational interaction. Online interaction with such an agent may also enable (especially disadvantaged) young people to engage in effective personalized learning. Thus our research has focused on the production of intelligent agents with emotion and social intelligence. Since affect interpretation plays important roles in how effectively an agent is able to help users in the provision of personalized training, we previously developed an affect detection component, EMMA (emotion, metaphor and affect) on detecting 25 affective states including basic and complex emotions, meta-emotions, etc [Zhang et al., 2009].

Previously the affect detection component was embedded in an online multi-user role-play plateform that could be used for education or entertainment. In this plateform young people could interact online in a 3D virtual drama stage with others under the guidance of a human director. In one session, up to five virtual characters are controlled on a virtual stage by human users ("actors"), with characters' (textual) "speeches" typed by the actors operating the characters. The actors are given a loose scenario around which to improvise, but are at liberty to be creative. An intelligent agent controlled by EMMA is also involved in improvisation. It makes attempts to produce appropriate responses to help stimulate the improvisation based on the detected affect from user input. The detected affect is also used to drive the animations of the avatars so that they react bodily in ways that is consistent with the affect that they are expressing.

However, the affect detection processing we conducted previously only identifies emotions from the analysis of individual turn-taking literal input. Sperber and Wilson [1995] stated that the intention of communication is to achieve the greatest possible cognitive outcome with the smallest possible processing effort, i.e. "to communicate only what is relevant". From the above perspectives, emotion and interaction context in our application has great potential to create such a relevant cognitive environment to facilitate effective communication. Thus in this paper, we discuss the contextual affect sensing based on the emotion modelling using personal and social improvisational interaction to discover affect embedded in emotionally ambiguous input. We have employed 'neutral' and 12 most commonly used emotions (caring, arguing, disapproving, approving, grateful, happy, sad, threatening, embarrassed, angry/rude, scared and sympathetic) out of the 25 affective states in our present work on contextual emotion analysis and prediction.

Also, in the collected transcripts, metaphorical expressions are used extensively to convey emotions and feelings (such as animal metaphor ("X is a rat") and affects as external entities metaphor ("joy ran through me") [Kšvecses, 1998; Zhang et al., 2009; Zhang, 2010]). As Fainsilber and Ortony [1987] commented, "an important function of metaphorical language is to permit the expression of that which is difficult to express using literal language alone". Thus in this paper we also equip EMMA with the capabilities of interpreting a few typical metaphorical phenomena as a testbed to inspire theoretical figurative language study.

We used two scenarios previously, the school bullying 1 and Crohn's disease 2 . In both scenarios, the AI agent plays a minor role in drama improvisation (a close friend to the bullied victim in school bullying scenario; the sick character's best friend in the Crohn's disease scenario). Transcripts were collected from our previous testing created by young people age 14 - 16. We use such example transcripts from our previous study to demonstrate our approach of contextual affect analysis and metaphor interpretation. Our paper is organized as follows. We report relevant work in section 2 and the new developments on contextual affect detection and metaphor interpretation respectively in section 3 & 4. The evaluation of the updated affect detection component and conclusion are provided in section 5.

1 The bully, Mayid, is picking on a new schoolmate, Lisa. Elise and Dave (Lisa's friends), and Mrs Parton (the school teacher) are trying to stop the bullying.

2 Peter has Crohn's disease and has the option to undergo a lifechanging but dangerous surgery. He needs to discuss the pros

2 Related Work

Much research has been done on creating affective virtual characters in interactive systems. Aylett et al. [2006] focused on the development of affective behaviour planning for their synthetic characters. Endrass, Rehm and AndrŽ [2011] carried out study on the culture-related differences in the domain of small talk behaviour. Their agents were equipped with the capabilities of generating culture specific dialogs. There is much other work in a similar vein.

Textual affect sensing has also become a rising research branch recently and drawn researchers' attention. ConceptNet [Liu & Singh, 2004] is a toolkit to provide practical textual reasoning for affect sensing for six basic emotions, text summarization and topic extraction. Shaikh et al. [2007] provided sentence-level affect sensing to recognize evaluations (positive and negative). They adopted a domainindependent approach, but have not made attempts to recognize different affective states from open-ended input. Zhe and Boucouvalas [2002] demonstrated an emotion extraction module embedded in an Internet chatting environment. It used a part-of-speech tagger and a syntactic chunker to detect the emotional words and to analyze emotion intensity. The detection focused only on emotional adjectives and first-person emotions, and did not address deep issues such as figurative expression of emotion. Ptaszynski et al. [2009] employed a context-sensitive affect detection approach with the integration of a web-mining technique to detect affect from users' input and verify the contextual appropriateness of the detected emotions. The detected results made an AI agent either sympathize with the player or disapprove the user's expression by the provision of persuasion. However, their system targeted conversations only between an AI agent and one human user in non-role-playing situations, which greatly reduced the complexity of the modelling of the interaction context.

Moreover metaphorical language can be used to convey emotions implicitly and explicitly, which also inspires cognitive semanticists [Kšvecses, 1998]. Indeed, the metaphorical description of emotional states is common and has been extensively studied [Fussell and Moss, 1998], for example,

and cons with friends and family. Janet (Mum) wants Peter to have the operation. Matthew (younger brother) is against it. Arnold (Dad) is not able to face the situation. Dave (the best friend) mediates the discussion.

"he nearly exploded" and "joy ran through me," where anger and joy are being viewed in vivid physical terms. Such examples describe emotional states in a relatively explicit if metaphorical way. But affect is also often conveyed more implicitly via metaphor, as in "his room is a cess-pit"; affect (such as 'disgust') associated with a source item (cess-pit) gets carried over to the corresponding target item (the room). There is also other work conducting theoretical research on metaphor in general (see, e.g., Barnden [2007]), which could be beneficial to our application as a useful source of theoretical inspiration.

Our work distinguishes on the following aspects: (1) affect detection from metaphorical expressions; (2) real-time affect sensing for basic and complex emotions in improvisational role-play situations; (3) affect detection for second and third person cases (e.g. 'you', 'she'); and (4) affect interpretation based on contextual profiles.

3 Affect Detection based on Context Profiles

Since our previous affect detection has been performed solely based on the analysis of individual turn-taking user input, the context information is ignored. As discussed earlier, the contextual and character profiles may influence the affect conveyed in the current input. Moreover, Relevance theory [Sperber & Wilson, 1995] also indicates that "communication aims at maximizing relevance and speakers presume that their communicative acts are indeed relevant". I.e. people contribute to effective communication by mentioning the most relevant discussion topics in the direct context. Therefore contextual affect detection has drawn our research attention. Also Lopez et al. [2008] suggested that context profiles for affect detection included social, environmental and personal contexts. In our study, personal context may be regarded as one's own emotion inclination or improvisational mood in communication context. We believe that one's own emotional states have a chain effect, i.e. the previous emotional status may influence later emotional experience. We make attempts to include such effects into emotion modelling. Bayesian networks are used to simulate such personal causal emotion context. In the Bayesian network example shown in Fig. 1, we regard the first, second and third emotions experienced by a particular user respectively as A, B and C. We assume that the affect B is dependent on the first emotion A. Further, we assume that the third emotion C, is dependent on both the first and second emotions, A and B. In our application, given two or more most recent emotional states a user experiences, we may predict the most probable emotion this user implies in the current input using a Bayesian network.

Briefly, a Bayesian network employs a probabilistic graphical model to represent causality relationship and conditional (in)dependencies between domain variables. It allows combining prior knowledge about (in)dependencies among variables with observed training data via a directed acyclic graph. It has a set of directed arcs linking pairs of nodes: an arc from a node X to a node Y means that X (parent emotion) has a direct influence on Y (successive child emotion). It uses the conditional probabilities (e.g. P[B|A], P[C|A,B]) to reflect such causal influence between prior emotional experiences to successive emotional expression. The following network topology is used to model personal emotion context in our application.

As mentioned earlier, we mainly consider 'neutral' and 12 most frequently used emotions for contextual affect analysis. Any combination of these emotional states could be used as prior emotional experience of the user. Also each conditional probability for each potential emotional state given two or more prior emotional experiences (such as P[approval|A,B] etc) will be calculated. The emotional state with the highest conditional probability is selected as the most probable emotion the user conveys in the current turntaking. We construct a Bayesian network for each character to sense his/her improvisational mood. At the training stage, two human judges (not involved in development) marked up 3 example transcripts of the school bullying scenario, which consisted of 460 turn-taking inputs. For each character, we extract 3 sequences of emotions (50 emotions per sequence on average) from the improvisation of the 3 example transcripts to produce prior conditional probabilities.

The following example is taken from the school bullying scenario. E.g., firstly based on the affect detection purely from the analysis of each individual input, we attach the following emotional label for each input.

1. Mrs Parton: y r u upset [caring]
2. Mayid: ugh!! u r such a wimp lisa. [insulting/angry]
3. Lisa: leave me alone [angry]
4. Elise: cuz mayid bullys her [angry]
5. Mayid: u stink lisa [insulting/angry]
6. Lisa: i dont have anyone to talk to and feel very lonely and frightened [scared]
7. Mrs Parton: detention mayid [threatening]
8. Mayid: shut it ugly girl elise [angry/insulting]
9. Elise: you big hairy fatty! [angry]
10. Mayid: shes upset coz shes weak! [neutral] -> [insulting/angry]

We derive 'neutral' for the 10 th input without any contextual inference. Since the input has a conjunction (because), it is probably caused by communication context. Thus the context-based affect analysis is activated to adjust the affect conveyed in this input. The emotional profile of Mayid ('angry (2 nd input)', 'angry (5 th input)', and 'angry (8 th input)') is used to construct the Bayesian probability matrix. The conditional probability of P[C|angry, angry, angry] is calculated for each potential emotion C. Finally 'angry' is regarded as the most probable emotion implied in the input "shes upset coz shes weak!".

However, social emotional context also has great potential to affect the emotional experience of the current speaking character. E.g., a recent negative context contributed by Mayid may cause Lisa and her friends to be 'angry'. A

neural network algorithm, Adaptive Resonance Theory (ART-1), is used to sense the positive/negative implication of the direct related context. ART is a collection of models for unsupervised learning and mainly used to deal with object identification and recognition. ART-1 in particular has the ability to maintain previously learned knowledge while still capable of learning new information. In our application, we use affect implication attached with four most recent interactions as input to ART-1 to sense positive/negative/neutral implication in the direct context.

Therefore, in this example, first we convert affect attached with the four recent emotional interactions (6 th - 9 th ) into binary values (positive/negative/neutral), i.e. 'negative' is attached respectively for the 6 th - 9 th input. The sequence of four binary affect values is used as the input to predict its influence to Mayid. The ART-1 inference indicates the direct related interaction as a 'negative' context, which justifies our previous prediction using personal emotion Bayesian modeling (angry). Thus Mayid is most likely to imply 'angry' in the 10 th input embedded in a 'negative' context.

4 Affect Sensing inMetaphorical Phenomena

Since various metaphorical expressions were used to convey emotions in our collected transcripts, several types of metaphorical expressions draw our research attention. E.g. in cooking metaphor, very often, the agent himself/herself becomes the victim of slow or intensive cooking (e.g. grilled); or one agent can perform cooking like actions towards another to realize punishment or torture. Examples include "he basted her with flattery to get the job", "she knew she was fried when the teacher handed back her paper" etc.

In these examples, the suffering agents have been figuratively conceptualized as food. They bear the results of intensive or slow cooking. Thus, these agents who suffer from such cooking actions carried out by other agents tend to feel pain and sadness, while the 'cooking performing' agents may take advantage of such actions to achieve their intentions, such as persuasion, punishment or even enjoyment. The syntactic structures of some of the above examples also indicate the submissive stance of the suffering agents. E.g. passive sentences ("he knew he was cooked when he saw his boss standing at the door") are used to imply unwillingness and victimization of the subject agents who are in fact the objects of the cooking actions described by the verb phrases ("X + copular form + passive cooking action"). In other examples, the cooking actions are explicitly performed by the subject agents towards the object agents to imply the former's potential willingness and enjoyment and the latter's suffering and pain ("A + [cooking action] + B").

Thus in our previous work, we focused on the interpretation of such cooking metaphor using off-the-shelf language processing tools including Rasp [Briscoe & Carroll, 2002], WordNet [Fellbaum, 1998] and semantic profiles [Esuli & Sebastiani, 2006]. Since WordNet has provided hypernyms (Y is a hypernym of X if every X is a (kind of) Y) for the general noun and verb lexicon, it is used to recover hypernyms of verbs describing cooking actions. Thus, for example, user input could be interpreted as 'subject human agents suffer from cooking actions' which may lead to negative emotional implication. An example processing for the input "I was fried by the head teacher" is presented as follows:

1. Rasp identifies the input has the following structure: 'PPIS1 (I) + copular form (was) + VVN (fried)';

2. 'Fry' (base form of the main verb) is sent to WordNet to obtain its hypernyms: 'COOK', 'HEAT' and 'KILL';

3. The input has the following syntactic semantic structure: 'PPIS1 (I) + copular form (was) + VVN (Hypernym: COOK)', thus it is recognized as a cooking metaphor;

4. The three hypernyms are sent to the evaluation resource -> 'KILL' is labeled as negative while others cannot obtain any evaluation values from the profile;

5. The input is transformed into: ''PPIS1 (I) + copular form (was) + VVN (KILL: negative)'

6. The subject is a first person case, then the input indicates the user who is speaking suffered from a negative action and may have a 'negative' emotional state.

In the Crohn's disease scenario, metaphorical expressions are also used to indicate both battles between family members and Peter's stress towards his life changing operation. An example interaction is as follows:

Arnold: waiter can you give us some privacy.

Matthew: yeah

Dave: im so much more than a waiter. ill have you know

Arnold: I hardly appreciate your attitude

Matthew: u r an old waiter with a smelly attitude.

In Crohn's disease, Dave played as a close friend to Peter and also a waiter in the restaurant who seemed not welcomed to be involved in the family discussion. The last input from Matthew expressed his unsatisfaction to the waiter's service and unfriendly attitude. It includes a sensory metaphor ('a smelly attitude') to imply 'insulting'. 'Smelly' is in fact normally used to describe physical entities (e.g. feet) but instead in this example it is used to describe an abstract cognitive concept. Also as indicated in the character profiles, Dave is Peter's schoolmate and they are both teenagers. Thus in this example, 'old' is used metaphorically to further strengthen the 'insulting' implication.

Moreover, with the inspiration of this sensory metaphor, we notice that emotions could also be implied explicitly by such sensory and another type of cooking metaphors. The sensory metaphor we are interested in includes temperature, smell, taste, and light metaphors. We gathered the following examples for our study of the semantic and syntactical structures of such metaphorical expressions, including cooking metaphor: "the news inflamed her temper", "he dishes out more criticism than one can take", "she was burned by a shady deal", light metaphor: "you lighted up my life", temperature metaphor: "they are kindling a new romance", smell metaphor: "love stinks", "the stench of failure", and taste metaphor: "bittersweet memories", "a spicy new outfit" etc.

In the new cooking metaphor examples, the cooking actions have been performed on cognitive abstract entities ('temper', 'criticism') or human agents ('she') [physical cooking actions + abstract entities/human agents]. Sometimes, human agents are also the objects of cooking actions performed by abstract subject entities ("she was burned by a shady deal"), which also leads to human agents' negative implication. Moreover, in the sensory metaphorical examples, the light and temperature metaphors show similar syntactical structures with actions conducting respectively on existence ('my life') or relationship abstract entities ('romance') [physical actions + abstract entities]. Emotion abstract entities are also used as subjects that are capable of performing actions such as love in smell metaphors [abstract subject entities + physical actions]. Overall, the above cooking and sensory metaphors contain some general principles, i.e. abstract entities are able to perform physical actions while they can also be the objects of physical actions. Also examples show cognitive abstract entities may also have characteristics of cooking, temperature, taste or smell. In another word, some cognitive abstract entities could be uncooked ('a raw talent'), tasty ('bittersweet memories') or have temperature ('heated debate', 'burning love') or smell ('the stench of failure'). We use such principles to recognize these metaphor phenomena and derive affect from them.

First, we use Rasp to indentify each subject, verb phrase and object in each sentence. Then we particularly send the main terms in these three components to Wordnet to recover their hypernyms. We also focus on the analysis of phrases with a structure of 'adjective + noun' by deriving the synonyms or related nouns for the adjective and hypernyms for the noun term using Wordnet. If the inputs indicate structures of 'abstract subject entities + actions', 'physical actions + abstract object entities' or 'temperature/smell/taste/cooking adjectives + abstract entities', then the inputs are recognized as metaphors. E.g., EMMA carries out the following processing to sense the metaphorical expression "u r an old waiter with a smelly attitude":

1. Rasp: 'PPY (you) + VBR (are) + AT1 (an) + JJ (old) + NN1 (waiter) + IW (with) + AT1 (a) + JJ (smelly) + NN1 (attitude)'

2. WordNet: 'waiter' -> hypernym: person -> physical entity; 'attitude' -> hypernym: psychological feature -> abstract entity.

3. WordNet: 'old' -> synonyms: 'elderly' & 'retired' which conflict with the character profile (Dave is a young person). 'Smelly' -> synonyms & related nouns: ill-smell, foul, and malodorous.

1. Semantic processing: foul, malodorous -> 'negative'.

5. Part of the input is interpreted as: 'a cognitive abstract entity has negative smell (i.e. a smell adj with negative indication + an abstract cognitive entity) -> identified as a smell metaphor with negative implication.

6. The input becomes: 'PPY (you) + VBR (are) + elderly (conflicting with character profile description: 'young') + person + IW (with) + a smell metaphor with negative indication' -> implies 'insulting/angry'.

EMMA is also capable of using the above identified principles to recognize other metaphors, e.g. 'stir up emotions, 'food for thought' etc.

As shown earlier, contextual information plays important roles in discovering affect conveyed in the emotionally ambiguous literal input, while there are also cases that it may help to determine affect and emotions implied in metaphorical input. We extract another example as follows to demonstrate our processing of contextual metaphor interpretation.

1. Peter: im going to have an ileostomy [sad]
2. Peter: im scared [scared]
3. Dave: i'm ur friend peter and i'll stand by you [caring]
4. Peter: yeah i know, but the disease stuff sucks [sad]
5. Dave: if it's what u want, you should go for it though [neutral]
6. Janet: peter you must go throu with this operation you understand its for the best [neutral]
7. Peter: but no 1 else can do nethin [neutral] -> [sad]
8. Arnold: consider all your options peter [neutral]
9. Matthew: u have had operations b4 I'm sure u'll b ok [caring]
10. Dave: what are your other options peter [neutral]
11. Peter: im trying very hard but there is too much stuff blocking my head up [neutral] -> [sad] (Detection error: it implies 'stressed')
12. Peter: my plate is already too full.... there aint otha options dave [disapproval]

For example, in the last two input from Peter, 'thoughts' have been regarded as physical solid objects that can occupy physical space such as a plate or head. With the contextual inference, plate has also been metaphorically used to refer to one's head. Moreover, we can hardly consider the last input as a metaphorical expression if without any contextual inference. It is theoretically and practically challenging.

Moreover for the partial input "there is too much stuff blocking my head up" in the above example, we have the following processing to recognize the metaphor input:

1. Rasp: 'EX (there) + VBZ (is) + RG (too) + DA1 (much) + NN1 (stuff) + VVG (blocking) + APP$ (my) + NN1 (head) + RP(up)'
2. WordNet: 'stuff' -> hypernym: information abstract entity, since 'stuff' has been described by a singular afterdeterminer ('much'). 'Head' -> hypernym: a body part physical entity. 'Block' -> hypernyms: PREVENT, KEEP.
3. The evaluation profile cannot indicate any emotional implication for PREVENT and KEEP.
4. The input implies -> 'an abstract subject entity (stuff) + an action (block) + a physical object entity' (head) -> recognised as a metaphorical input.

However, we cannot recover any affect implied in this metaphorical input purely based on the analysis of the input itself. Context-based emotion detection is resorted to further justify the affect implied in it. Also since Peter is the leading character who suffers from the disease and leads a dramatic improvisation, we focus on his contribution in the above example to demonstrate our processing. We start the contextual affect analysis in the 7 th input from Peter: "but no 1 else can do nethin", since it is also detected as non-emotional.

The personal emotion context of Peter since the beginning until the 7 th input: 'sad (1 st input)', 'scared (2 nd input)', and 'sad (4 th input)', is used as input to the Bayesian networks. It deduces that Peter is most likely to imply 'sadness'. The most relevant social context from the 3 rd to the 6 th input is used as input to the ART-1 reasoning. It predicts the recent interaction implies 'neutral' and Peter is not affected by other participants' recent neutral contribution. Thus overall the 7 th input indicates 'sad'. Similarly, the affect detection processing of the 11 th input is as follows:

1. The personal improvisational mood prediction using Bayesian networks with Peter's updated emotion context: 'sad', 'scared', 'sad' and 'sad (7 th input)' -> 'sad' again as the predicted affect.
2. The social emotional context from Peter (7 th : negative), Arnold (8 th : neutral), Matthew (9 th : positive) and Dave (10 th : neutral) is used as input to the ART-1 neural network -> 'neutral' context and no emotional influence to Peter.
3. Thus Peter implies 'sadness' in the 11 th metaphorical input.

However, human judges believe that the 11 th input conveys Peter's 'stress' towards the decision-making of the lifechanging operation, which leads to detection errors. However the system is capable of sensing the negative implication from the metaphorical input with contextual affect inference.

5 Evaluations & Conclusions

We carried out user testing with 220 secondary school students from Birmingham and Darlington schools for the improvisation of school bullying and Crohn's disease scenarios. Generally, our previous statistical results based on the collected questionnaires indicate that the involvement of the AI character has not made any statistically significant difference to users' engagement and enjoyment with the emphasis of users' notice of the AI character's contribution throughout. Briefly, the methodology of the testing is that we had each testing subject have an experience of both scenarios, one including the AI minor character only and the other including the human-controlled minor character only. After the testing sessions, we obtained users' feedback via questionnaires and debriefings. Improvisational transcripts were automatically recorded so that it allows further evaluation of the performance of the affect detection component.

Therefore, we produce a new set of results for the evaluation of the updated affect detection component with metaphorical and context-based affect detection based on the analysis of some recorded transcripts of school bullying scenario. Generally two human judges (not engaged in development) marked up the affect of 250 turn-taking user input from the recorded 4 transcripts. In order to verify the efficiency of the new developments, we provide Cohen's Kappa inter-agreements for EMMA's performance with and without the new developments for the detection of the most commonly used 12 affective states. In the school bullying scenario, EMMA played a minor character, Dave. The agreement for human judge A/B is 0.41. The interagreements between human judge A/B and EMMA with and without the new developments are presented in Table 1.

Although further work is needed, the new developments on metaphorical and contextual affect sensing have improved EMMA's performance of affect detection in the test transcripts comparing with the previous version.

But there are still some cases: when the human judges both believed that user inputs carried negative affective states (such as threatening), EMMA regarded them as neutral. One most obvious reason is that some of the previous pipeline processing (such as dealing with mis-spelling, acronyms etc, and syntactic processing from Rasp etc) failed to recover the standard user input or recognize the complex structure of the input which led to less interesting and less emotional context and may affect the performance of contextual affect sensing. Currently we achieved 88% average accuracy rate for the contextual affect sensing for the emotion interpretation of all the human controlled characters in school bullying scenario. Using a metaphorical resource (http://knowgramming.com), our approach for cooking and sensory metaphor recognition obtains 50% average accuracy rate. We also aim to extend the evaluation of the contextbased affect detection using transcripts from other scenarios and further evaluate the new development on metaphorical sensing using other resources (e.g. The Wall Street Journal).

Overall, we make initial developments of an AI agent with emotion and social intelligence, which employs context profiles for affect interpretation using Bayesian networks and unsupervised neural network algorithms and performs metaphor recognition and learning. Although EMMA could be challenged by the rich diverse variations of the language phenomena, we believe these areas are very crucial for development of effective intelligent user interfaces and our processing has made promising initial steps towards these aspects. We also intend to compare our system's performance with that of other well-known similar tools.