{"type":"ScholarlyArticle","title":"Automatic Multimodal Emotion Recognition Using Facial Expression, Voice, and Text","authors":[{"name":"Hélène Tran","slug":"helene-tran","given":"Hélène","family":"Tran","affiliation":"Université Clermont-Auvergne , CNRS , Mines de Saint-Étienne , Clermont-Auvergne-INP , LIMOS , 63000 Clermont-Ferrand , France; Jeolis Solutions , 63000 Clermont-Ferrand , France"}],"abstract":"It has been a long-time dream for humans to interact with a machine as we would with a person, in a way that it understands us, advises us, and looks after us with no human supervision. Despite being efficient on logical reasoning, current advanced systems lack empathy and user understanding. Estimating the user's emotion could greatly help the machine to identify the user's needs and adapt its behaviour accordingly. This research project aims to develop an automatic emotion recognition system based on facial expression, voice, and words. We expect to address the challenges related to multimodality, data complexity, and emotion representation.","doi":"10.24963/ijcai.2022/843","url":"https://ijcai.org/proceedings/2022/843","pdf":"https://ijcai.org/proceedings/2022/0843.pdf","markdown":"https://ijcai.org/proceedings/2022/843.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/843","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":843,"pages":{"first":5881,"last":5882},"keywords":["Machine Learning (ML): General"],"videos":[],"references":[{"position":1,"raw":"[Dai et al., 2021] Wenliang Dai, Samuel Cahyawijaya, Zi- han Liu, and Pascale Fung. Multimodal End-to-End Sparse Model for Emotion Recognition. In Proceedings of the 2021 Conference of the North American Chapter 1 http://multicomp.cs.cmu.edu/acl2018multimodalworkshop-2/ 2 http://multicomp.cs.cmu.edu/acl2020multimodalworkshop/ of the Association for Computational Linguistics: Human Language Technologies, pages 5305-5316, 2021.","title":"Multimodal End-to-End Sparse Model for Emotion Recognition","authors":"Wenliang Dai, Samuel Cahyawijaya, Zihan Liu, Pascale Fung","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter 1","doi":null,"url":"http://multicomp.cs.cmu.edu/acl2018multimodalworkshop-2/2http://multicomp.cs.cmu.edu/acl2020multimodalworkshop/oftheAssociationforComputationalLinguistics","internal_url":null},{"position":2,"raw":"[Delbrouck et al., 2020] Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. A Transformer-based joint-encoding for Emotion Recogni- tion and Sentiment Analysis. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge- HML), pages 1-7, 2020.","title":"A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis","authors":"Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, Stéphane Dupont","year":"2020","venue":"Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[D'mello and Kory, 2015] Sidney K D'mello and Jacqueline Kory. A review and meta-analysis of multimodal af- fect detection systems. ACM computing surveys (CSUR), 47(3):1-36, 2015.","title":"A review and meta-analysis of multimodal affect detection systems","authors":"Kory ; Sidney, K, Jacqueline Kory","year":"2015","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goleman, 1995] Daniel Goleman. Emotional Intelligence. Bantam Books, 1995.","title":"Emotional Intelligence","authors":"Daniel Goleman","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Mittal et al., 2020] Trisha Mittal, Uttaran Bhattacharya, Ro- han Chandra, Aniket Bera, and Dinesh Manocha. M3ER: Multiplicative Multimodal Emotion Recognition using Fa- cial, Textual, and Speech Cues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1359-1367, 2020.","title":"M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues","authors":"Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, Dinesh Manocha","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Tran et al., 2022] Hélène Tran, Lisa Brelet, Issam Falih, Xavier Goblet, and Engelbert Mephu Nguifo. L'ambiguïté dans la représentation des émotions : état de l'art des bases de données multimodales. Revue des Nouvelles Technolo- gies de l'Information, Extraction et Gestion des Connais- sances, RNTI-E-38:87-98, 2022.","title":"L'ambiguïté dans la représentation des émotions : état de l'art des bases de données multimodales","authors":"Hélène Tran, Lisa Brelet, Issam Falih, Xavier Goblet, Engelbert Mephu Nguifo","year":"2022","venue":"Revue des Nouvelles Technologies de l'Information, Extraction et Gestion des Connaissances","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal Transformer for Un- aligned Multimodal Language Sequences. In Proceedings of the conference. Association for Computational Linguis- tics. Meeting, volume 2019, page 6558. NIH Public Ac- cess, 2019.","title":"Multimodal Transformer for Unaligned Multimodal Language Sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Williams et al., 2018] Jennifer Williams, Steven Klei- negesse, Ramona Comanescu, and Oana Radu. Recogniz- ing Emotions in Video Using Multimodal DNN Feature Fusion. In Proceedings of Grand Challenge and Work- shop on Human Multimodal Language (Challenge-HML), pages 11-19, 2018.","title":"Recognizing Emotions in Video Using Multimodal DNN Feature Fusion","authors":"Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, Oana Radu","year":"2018","venue":"Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Zadeh et al., 2018] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Mul- timodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018.","title":"Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph","authors":"Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Zadeh et al., 2020] Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, and Louis- Philippe Morency. CMU-MOSEAS: A multimodal lan- guage dataset for Spanish, Portuguese, German and French. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, vol- ume 2020, page 1801. NIH Public Access, 2020.","title":"CMU-MOSEAS: A multimodal language dataset for Spanish, Portuguese, German and French","authors":"Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, Louis-Philippe Morency","year":"2020","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}