{"type":"ScholarlyArticle","title":"Inside Out: Emotional Multiagent Multimodal Dialogue Systems","authors":[{"name":"Andrey V. Savchenko","slug":"andrey-v-savchenko","given":"Andrey V.","family":"Savchenko","affiliation":"Sber AI Lab , Moscow , Russia; ISP RAS Research Center for Trusted Artificial Intelligence , Moscow , Russia; HSE University , Laboratory of Algorithms and Technologies for Network Analysis , Nizhny Novgorod , Russia"},{"name":"Lyudmila V. Savchenko","slug":"lyudmila-v-savchenko","given":"Lyudmila V.","family":"Savchenko","affiliation":"ISP RAS Research Center for Trusted Artificial Intelligence , Moscow , Russia; HSE University , Laboratory of Algorithms and Technologies for Network Analysis , Nizhny Novgorod , Russia"}],"abstract":"In this paper, we introduce the novel technological framework for the development of emotional dialogue systems. Inspired by the \"Inside Out\" film, we propose to use multiple emotional agents based on Large Language Models (LLMs) to prepare answers to a user query. Their answers are aggregated into a single response, taking into account the current emotional state of a user. The latter is estimated by video-based facial expression recognition (FER). We introduce several publicly available lightweight neural networks that show near state-of-the-art results on the AffectNet dataset. Qualitative examples using either GPT-3.5 or LLama2 and Mistral demonstrate that the proposed approach leads to more emotional responses in LLMs.","doi":"10.24963/ijcai.2024/1032","url":"https://ijcai.org/proceedings/2024/1032","pdf":"https://ijcai.org/proceedings/2024/1032.pdf","markdown":"https://ijcai.org/proceedings/2024/1032.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1032","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1032,"pages":{"first":8784,"last":8788},"keywords":["Computer Vision: CV: Biometrics, face, gesture and pose recognition","Humans and AI: HAI: Human-computer interaction","Natural Language Processing: NLP: Language generation","Agent-based and Multi-agent Systems: MAS: Applications"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VggFace2: A dataset for recognising faces across pose and age. In Proceedings of the 13th International Conference on Automatic Face & Gesture Recognition (FG), pages 67-74. IEEE, 2018.","title":"VggFace2: A dataset for recognising faces across pose and age","authors":"Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, Andrew Zisserman","year":"2018","venue":"Proceedings of the 13th International Conference on Automatic Face & Gesture Recognition (FG)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. MobileFaceNets: Efficient CNNs for accurate real-time face verification on mobile devices. In Proceed- ings of the 13th Chinese Conference on Biometric Recog- nition (CCBR), pages 428-438. Springer, 2018.","title":"MobileFaceNets: Efficient CNNs for accurate real-time face verification on mobile devices","authors":"Sheng Chen, Yang Liu, Xiang Gao, Zhen Han","year":"2018","venue":"Proceedings of the 13th Chinese Conference on Biometric Recognition (CCBR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2023] Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. From static to dynamic: Adapt- ing landmark-aware image models for facial expression recognition in videos. arXiv preprint arXiv:2312.05447, 2023.","title":"From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos","authors":"Yin Chen, Jia Li, Shiguang Shan, Meng Wang, Richang Hong","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Damasio, 2001] Antonio R Damasio. Descartes error re- visited. Journal of the History of the Neurosciences, 10(2):192-194, 2001.","title":"Descartes error revisited","authors":"Antonio R Damasio","year":"2001","venue":"Journal of the History of the Neurosciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dukes et al., 2021] Daniel Dukes, Kathryn Abrams, Ralph Adolphs, Mohammed E Ahmed, Andrew Beatty, Kent C Berridge, Susan Broomhall, Tobias Brosch, Joseph J Cam- pos, Zanna Clay, et al. The rise of affectivism. Nature human behaviour, 5(7):816-820, 2021.","title":"The rise of affectivism","authors":"Daniel Dukes, Kathryn Abrams, Ralph Adolphs, Mohammed E Ahmed, Andrew Beatty, Kent C Berridge, Susan Broomhall, Tobias Brosch, Joseph J Campos, Zanna Clay","year":"2021","venue":"Nature human behaviour","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2022] Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, and Sergey Tulyakov. Show me what and tell me how: Video synthesis via multimodal condi- tioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3615- 3625, 2022.","title":"Show me what and tell me how: Video synthesis via multimodal conditioning","authors":"Han","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.","title":"Mistral 7b","authors":"Alexandre Albert Q Jiang, Arthur Sablayrolles, Chris Mensch, Devendra Bamford, Diego Singh Chaplot, Florian De Las Casas, Gianna Bressand, Guillaume Lengyel, Lucile Lample, Saulnier","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kalia et al., 2019] Anup K. Kalia, Nirav Ajmeri, Kevin S. Chan, Jin-Hee Cho, Sibel Adalı, and Munindar P. Singh. The interplay of emotions and norms in multiagent sys- tems. In Proceedings of the Twenty-Eighth Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 19, pages 371-377. International Joint Conferences on Ar- tificial Intelligence Organization, 7 2019.","title":"The interplay of emotions and norms in multiagent systems","authors":"K Anup, Nirav Kalia, Kevin S Ajmeri, Jin-Hee Chan, Sibel Cho, Munindar P Adalı, Singh","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/53"},{"position":9,"raw":"[Kharchevnikova and Savchenko, 2018] Angelina S. Kharchevnikova and Andrey V Savchenko. Neural networks in video-based age and gender recognition on mobile platforms. Optical Memory and Neural Networks, 27:246-259, 2018.","title":"Neural networks in video-based age and gender recognition on mobile platforms","authors":"Angelina S Kharchevnikova, Andrey V Savchenko","year":"2018","venue":"Optical Memory and Neural Networks","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lei et al., 2022] Yi Lei, Shan Yang, Xinsheng Wang, and Lei Xie. MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:853-864, 2022.","title":"MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis","authors":"Yi Lei, Shan Yang, Xinsheng Wang, Lei Xie","year":"2022","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lei et al., 2023] Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. InstructERC: Re- forming emotion recognition in conversation with a re- trieval multi-task LLMs framework, 2023.","title":"InstructERC: Reforming emotion recognition in conversation with a retrieval multi-task LLMs framework","authors":"Lei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023a] Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Large language models understand and can be enhanced by emotional stimuli, 2023.","title":"Large language models understand and can be enhanced by emotional stimuli","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023b] Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Xinyi Wang, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. The good, the bad, and why: Unveiling emotions in generative ai, 2023.","title":"The good, the bad, and why: Unveiling emotions in generative ai","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023c] Jia Li, Jiantao Nie, Dan Guo, Richang Hong, and Meng Wang. Emotion separation and recognition from a facial expression by generating the poker face with vision transformers. arXiv preprint arXiv:2207.11081, 2023.","title":"Emotion separation and recognition from a facial expression by generating the poker face with vision transformers","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023d] Ming Li, Yusheng Su, Hsiu-Yuan Huang, Jiali Cheng, Xin Hu, Xinmiao Zhang, Huadong Wang, Yujia Qin, Xiaozhi Wang, Zhiyuan Liu, and Dan Zhang. Language-specific representation of emotion- concept knowledge causally supports emotion inference, 2023.","title":"Language-specific representation of emotionconcept knowledge causally supports emotion inference","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luginov and Makarov, 2023] Albert Luginov and Ilya Makarov. Swiftdepth: An efficient hybrid CNN- transformer model for self-supervised monocular depth estimation on mobile devices. In Proceedings of the In- ternational Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 642-647. IEEE, 2023.","title":"Albert Luginov and Ilya Makarov. Swiftdepth: An efficient hybrid CNNtransformer model for self-supervised monocular depth estimation on mobile devices","authors":"Makarov Luginov","year":"2023","venue":"Proceedings of the International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Makarov et al., 2016] Ilya Makarov, Mikhail Tokmakov, Pavel Polyakov, Peter Zyuzin, Maxim Martynov, Oleg Konoplya, George Kuznetsov, Ivan Guschenko-Cheverda, Maxim Uriev, Ivan Mokeev, et al. First-person shooter game for virtual reality headset with advanced multi-agent intelligent system. In Proceedings of the 24th ACM In- ternational Conference on Multimedia (ACMMM), pages 735-736, 2016.","title":"First-person shooter game for virtual reality headset with advanced multi-agent intelligent system","authors":"Makarov","year":"2016","venue":"Proceedings of the 24th ACM International Conference on Multimedia (ACMMM)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mehta and Rastegari, 2021] Sachin Mehta and Mohammad Rastegari. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.","title":"MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer","authors":"Sachin Mehta, Mohammad Rastegari","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mollahosseini et al., 2017] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-31, 2017.","title":"AffectNet: A database for facial expression, valence, and arousal computing in the wild","authors":"Ali Mollahosseini, Behzad Hasani, Mohammad H Mahoor","year":"2017","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nandwani and Verma, 2021] Pansy Nandwani and Rupali Verma. A review on sentiment analysis and emotion de- tection from text. Social Network Analysis and Mining, 11(1):81, 2021.","title":"Pansy Nandwani and Rupali Verma. A review on sentiment analysis and emotion detection from text","authors":"Verma Nandwani","year":"2021","venue":"Social Network Analysis and Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paech, 2024] Samuel J. Paech. EQ-Bench: An emotional intelligence benchmark for large language models, 2024.","title":"EQ-Bench: An emotional intelligence benchmark for large language models","authors":"J Samuel, Paech","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Picard, 2015] Rosalind W Picard. The promise of affective computing. The Oxford handbook of affective computing, pages 11-20, 2015.","title":"The promise of affective computing","authors":"Rosalind W Picard","year":"2015","venue":"The Oxford handbook of affective computing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pourmirzaei et al., 2021] Mahdi Pourmirzaei, Gholam Ali Montazer, and Farzaneh Esmaili. Using self-supervised auxiliary tasks to improve fine-grained facial representa- tion. arXiv preprint arXiv:2105.06421, 2021.","title":"Using self-supervised auxiliary tasks to improve fine-grained facial representation","authors":"Pourmirzaei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Savchenko and Savchenko, 2024] Andrey V Savchenko and Lyudmila V Savchenko. Video demo for Inside Out: Emotional multiagent multimodal dialogue sys- tems. https://youtu.be/xxvkbyFsIs4, 2024. Accessed: 2024-05-03.","title":"Video demo for Inside Out: Emotional multiagent multimodal dialogue systems","authors":"V Andrey, Lyudmila V Savchenko, Savchenko","year":"2024","venue":null,"doi":null,"url":"https://youtu.be/xxvkbyFsIs4,2024","internal_url":null},{"position":25,"raw":"[Savchenko, 2022] Andrey V Savchenko. MT-EmotiEffNet for multi-task human affective behavior analysis and learn- ing from synthetic data. In Proceedings of European Con- ference on Computer Vision (ECCV) Workshops, pages 45-59. Springer, 2022.","title":"MT-EmotiEffNet for multi-task human affective behavior analysis and learning from synthetic data","authors":"Andrey V Savchenko","year":"2022","venue":"Proceedings of European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Savchenko, 2023a] Andrey Savchenko. Facial expression recognition with adaptive frame rate based on multiple testing correction. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th Interna- tional Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 30119-30129. PMLR, 23-29 Jul 2023.","title":"Facial expression recognition with adaptive frame rate based on multiple testing correction","authors":"Andrey Savchenko, Savchenko","year":"2023","venue":"Proceedings of the 40th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Savchenko, 2023b] Andrey V Savchenko. EmotiEffNets for facial processing in video-based valence-arousal predic- tion, expression classification and action unit detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5715-5723, 2023.","title":"EmotiEffNets for facial processing in video-based valence-arousal prediction, expression classification and action unit detection","authors":"Andrey V Savchenko, Savchenko","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Savchenko, 2024a] Andrey V Savchenko. HSE- motion library. https://github.com/av-savchenko/ face-emotion-recognition/tree/main/src/emotions llm, 2024. Accessed: 2024-05-03.","title":"HSEmotion library","authors":"Andrey V Savchenko, Savchenko","year":"2024","venue":null,"doi":null,"url":"https://github.com/av-savchenko/face-emotion-recognition/tree/main/src/emotionsllm,2024","internal_url":null},{"position":29,"raw":"[Savchenko, 2024b] Andrey V Savchenko. Leveraging pre- trained multi-task deep models for trustworthy facial anal- ysis in affective behaviour analysis in-the-wild. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024.","title":"Leveraging pretrained multi-task deep models for trustworthy facial analysis in affective behaviour analysis in-the-wild","authors":"Andrey V Savchenko, Savchenko","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. Efficient- Net: Rethinking model scaling for convolutional neural networks. In Proceedings of International Conference on Machine Learning (ICML), pages 6105-6114. PMLR, 2019.","title":"Efficient-Net: Rethinking model scaling for convolutional neural networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"Proceedings of International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[tse Huang et al., 2024] Jen tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R. Lyu. Emotionally numb or empathetic? evaluating how LLMs feel using Emotion- Bench, 2024.","title":"Emotionally numb or empathetic? evaluating how LLMs feel using Emotion-Bench","authors":"Jen Tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, Michael R Lyu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wagner et al., 2023] Johannes Wagner, Andreas Triantafyl- lopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W Schuller. Dawn of the transformer era in speech emotion recognition: closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.","title":"Dawn of the transformer era in speech emotion recognition: closing the valence gap","authors":"Wagner","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wen et al., 2023] Zhengyao Wen, Wenzhong Lin, Tao Wang, and Ge Xu. Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics, 8(2):199, 2023.","title":"Distract your attention: Multi-head cross attention network for facial expression recognition","authors":"Zhengyao Wen, Wenzhong Lin, Tao Wang, Ge Xu","year":"2023","venue":"Biomimetics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu et al., 2023] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: En- abling next-gen LLM applications via multi-agent con- versation framework. arXiv preprint arXiv:2308.08155, 2023.","title":"Autogen: Enabling next-gen LLM applications via multi-agent conversation framework","authors":"Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, Chi Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}