{"type":"ScholarlyArticle","title":"Towards Discriminative Representation Learning for Speech Emotion Recognition","authors":[{"name":"Runnan Li","slug":"runnan-li","given":"Runnan","family":"Li","affiliation":"Graduate School at Shenzhen , Tsinghua University; Dept. of Computer Science and Technology , Tsinghua University"},{"name":"Zhiyong Wu","slug":"zhiyong-wu","given":"Zhiyong","family":"Wu","affiliation":"Graduate School at Shenzhen , Tsinghua University; Dept. of Computer Science and Technology , Tsinghua University"},{"name":"Jia Jia","slug":"jia-jia","given":"Jia","family":"Jia","affiliation":"Graduate School at Shenzhen , Tsinghua University; Dept. of Computer Science and Technology , Tsinghua University"},{"name":"Yaohua Bu","slug":"yaohua-bu","given":"Yaohua","family":"Bu","affiliation":"Dept. of Computer Science and Technology , Tsinghua University"},{"name":"Sheng Zhao","slug":"sheng-zhao","given":"Sheng","family":"Zhao","affiliation":"Search Technology Center Asia (STCA) , Microsoft"},{"name":"Helen Meng","slug":"helen-meng","given":"Helen","family":"Meng","affiliation":"Dept. of Systems Engineering and Engineering Management , The Chinese University of Hong Kong"}],"abstract":"In intelligent speech interaction, automatic speech emotion recognition (SER) plays an important role in understanding user intention.  While sentimental speech has different speaker characteristics but similar acoustic attributes, one vital challenge in SER is how to learn robust and discriminative representations for emotion inferring.  In this paper, inspired by human emotion perception, we propose a novel representation learning component (RLC) for SER system, which is constructed with Multi-head Self-attention and Global Context-aware Attention Long Short-Term Memory Recurrent Neutral Network (GCA-LSTM). With the ability of Multi-head Self-attention mechanism in modeling the element-wise correlative dependencies, RLC can exploit the common patterns of sentimental speech features to enhance emotion-salient information importing in representation learning. By employing GCA-LSTM, RLC can selectively focus on emotion-salient factors with the consideration of entire utterance context, and gradually produce discriminative representation for emotion inferring.  Experiments on public emotional benchmark database IEMOCAP and a tremendous realistic interaction database demonstrate the outperformance of the proposed SER framework, with 6.6% to 26.7% relative improvement on unweighted accuracy compared to state-of-the-art techniques.","doi":"10.24963/ijcai.2019/703","url":"https://ijcai.org/proceedings/2019/703","pdf":"https://ijcai.org/proceedings/2019/0703.pdf","markdown":"https://ijcai.org/proceedings/2019/703.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/703","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":703,"pages":{"first":5060,"last":5066},"keywords":["Natural Language Processing: Sentiment Analysis and Text Mining","Natural Language Processing: Speech"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015] Martín Abadi, Ashish Agarwal, and et al. TensorFlow: Large-scale machine learning on het- erogeneous systems, 2015.","title":"TensorFlow: Large-scale machine learning on heterogeneous systems","authors":"Martín Abadi, Ashish Agarwal","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Busso et al., 2008] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, et al. Iemocap: In- teractive emotional dyadic motion capture database. Lan- guage resources and evaluation, 42(4):335, 2008.","title":"Iemocap: Interactive emotional dyadic motion capture database","authors":"Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower","year":"2008","venue":"Language resources and evaluation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chorowski et al., 2015] Jan K Chorowski, Dzmitry Bah- danau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Advances in neural information processing systems, pages 577-585, 2015.","title":"Attention-based models for speech recognition","authors":"Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee and Tashev, 2015] Jinkyu Lee and Ivan Tashev. High- level feature representation using recurrent neural network for speech emotion recognition. 2015.","title":"Highlevel feature representation using recurrent neural network for speech emotion recognition","authors":"Tashev ; Lee, Jinkyu Lee, Ivan Tashev","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lian et al., 2018] Zheng Lian, Ya Li, Jianhua Tao, and Jian Huang. Improving speech emotion recognition via transformer-based predictive coding through transfer learning. CoRR, abs/1811.07691, 2018.","title":"Improving speech emotion recognition via transformer-based predictive coding through transfer learning","authors":"Zheng Lian, Ya Li, Jianhua Tao, Jian Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2017] Jun Liu, Gang Wang, Ping Hu, Ling-Yu Duan, and Alex C Kot. Global context-aware attention lstm networks for 3d action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 7, page 43, 2017.","title":"Global context-aware attention lstm networks for 3d action recognition","authors":"Jun Liu, Gang Wang, Ping Hu, Ling-Yu Duan, Alex C Kot","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mirsamadi et al., 2017] Seyedmahdad Mirsamadi, Emad Barsoum, and Cha Zhang. Automatic speech emotion recognition using recurrent neural networks with local at- tention. In Proceedings of the IEEE International Confer- ence on Acoustics, Speech and Signal Processing, pages 2227-2231. IEEE, 2017.","title":"Automatic speech emotion recognition using recurrent neural networks with local attention","authors":"Mirsamadi","year":"2017","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Poria et al., 2016] Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. Convolutional mkl based multimodal emotion recognition and sentiment analysis. In Proceedings of the IEEE International Conference on Data Mining, pages 439-448. IEEE, 2016.","title":"Convolutional mkl based multimodal emotion recognition and sentiment analysis","authors":"Poria","year":"2016","venue":"Proceedings of the IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Poria et al., 2017] Soujanya Poria, Erik Cambria, Deva- manyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 873-883, 2017.","title":"Context-dependent sentiment analysis in user-generated videos","authors":"Poria","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Powers, 2011] David Martin Powers. Evaluation: from pre- cision, recall and f-measure to roc, informedness, marked- ness and correlation. 2011.","title":"Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation","authors":"David Martin, Powers","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rozgic et al., 2012] Viktor Rozgic, Sankaranarayanan Ananthakrishnan, Shirin Saleem, Rohit Kumar, and Rohit Prasad. Ensemble of svm trees for multimodal emotion recognition. In Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1-4. IEEE, 2012.","title":"Ensemble of svm trees for multimodal emotion recognition","authors":"Sankaranarayanan Viktor Rozgic, Shirin Ananthakrishnan, Rohit Saleem, Rohit Kumar, Prasad","year":"2012","venue":"Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schirmer and Adolphs, 2017] Annett Schirmer and Ralph Adolphs. Emotion perception from face, voice, and touch: comparisons and convergence. Trends in cognitive sci- ences, 21(3):216-228, 2017.","title":"Emotion perception from face, voice, and touch: comparisons and convergence","authors":"Adolphs Schirmer, Annett Schirmer, Ralph Adolphs","year":"2017","venue":"Trends in cognitive sciences","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schuller et al., 2017] Björn Schuller, Stefan Steidl, Anton Batliner, Elika Bergelson, et al. The interspeech 2017 computational paralinguistics challenge: Addressee, cold & snoring. In Computational Paralinguistics Challenge (ComParE), Interspeech 2017, pages 3442-3446, 2017.","title":"The interspeech 2017 computational paralinguistics challenge: Addressee, cold & snoring","authors":"Schuller","year":"2017","venue":"Computational Paralinguistics Challenge (ComParE), Interspeech 2017","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Steinschneider et al., 2013] Mitchell Steinschneider, Kir- ill V Nourski, and Yonatan I Fishman. Representation of speech in human auditory cortex: is it special? Hearing research, 305:57-73, 2013.","title":"Representation of speech in human auditory cortex: is it special?","authors":"Mitchell Steinschneider, V Kirill, Yonatan I Nourski, Fishman","year":"2013","venue":"Hearing research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strapparava and Mihalcea, 2008] Carlo Strapparava and Rada Mihalcea. Learning to identify emotions in text. In Proceedings of the 2008 ACM symposium on Applied computing, pages 1556-1560. ACM, 2008.","title":"Carlo Strapparava and Rada Mihalcea. Learning to identify emotions in text","authors":"Mihalcea Strapparava","year":"2008","venue":"Proceedings of the 2008 ACM symposium on Applied computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Stuhlsatz et al., 2011] André Stuhlsatz, Christine Meyer, Florian Eyben, Thomas Zielke, Günter Meier, and Björn Schuller. Deep neural networks for acoustic emotion recognition: raising the benchmarks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5688-5691. IEEE, 2011.","title":"Deep neural networks for acoustic emotion recognition: raising the benchmarks","authors":"Stuhlsatz","year":"2011","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Trigeorgis et al., 2016] George Trigeorgis, Fabien Ringeval, et al. Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5200-5204. IEEE, 2016.","title":"Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network","authors":"George Trigeorgis, Fabien Ringeval","year":"2016","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia and Liu, 2017] Rui Xia and Yang Liu. A multi-task learning framework for emotion recognition using 2d con- tinuous space. IEEE Transactions on Affective Computing, (1):3-14, 2017.","title":"A multi-task learning framework for emotion recognition using 2d continuous space","authors":"Rui Xia, Yang Liu","year":"2017","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}