{"type":"ScholarlyArticle","title":"CTL-MTNet: A Novel CapsNet and Transfer Learning-Based Mixed Task Net for Single-Corpus and Cross-Corpus Speech Emotion Recognition","authors":[{"name":"Xin-Cheng Wen","slug":"xin-cheng-wen","given":"Xin-Cheng","family":"Wen","affiliation":"School of Informatics , Xiamen University , Xiamen , China"},{"name":"JiaXin Ye","slug":"jiaxin-ye","given":"JiaXin","family":"Ye","affiliation":"School of Informatics , Xiamen University , Xiamen , China"},{"name":"Yan Luo","slug":"yan-luo","given":"Yan","family":"Luo","affiliation":"School of Informatics , Xiamen University , Xiamen , China"},{"name":"Yong Xu","slug":"yong-xu","given":"Yong","family":"Xu","affiliation":"School of Computer Science and Mathematics , Fujian University of Technology , Fuzhou , China"},{"name":"Xuan-Ze Wang","slug":"xuan-ze-wang","given":"Xuan-Ze","family":"Wang","affiliation":"School of Informatics , Xiamen University , Xiamen , China"},{"name":"Chang-Li Wu","slug":"chang-li-wu","given":"Chang-Li","family":"Wu","affiliation":"School of Informatics , Xiamen University , Xiamen , China"},{"name":"Kun-Hong Liu","slug":"kun-hong-liu","given":"Kun-Hong","family":"Liu","affiliation":"School of Film , Xiamen University , Xiamen , China; School of Informatics , Xiamen University , Xiamen , China"}],"abstract":"Speech Emotion Recognition (SER) has become a growing focus of research in human-computer interaction. An essential challenge in SER is to extract common attributes from different speakers or languages, especially when a specific source corpus has to be trained to recognize the unknown data coming from another speech corpus. To address this challenge, a Capsule Network (CapsNet) and Transfer Learning based Mixed Task Net (CTL-MTNet) are proposed to deal with both the single-corpus and cross-corpus SER tasks simultaneously in this paper. For the single-corpus task, the combination of Convolution-Pooling and Attention CapsNet module (CPAC) is designed by embedding the self-attention mechanism to the CapsNet, guiding the module to focus on the important features that can be fed into different capsules. The extracted high-level features by CPAC provide sufficient discriminative ability. Furthermore, to handle the cross-corpus task, CTL-MTNet employs a Corpus Adaptation Adversarial Module (CAAM) by combining CPAC with Margin Disparity Discrepancy (MDD), which can learn the domain-invariant emotion representations through extracting the strong emotion commonness. Experiments including ablation studies and visualizations on both single- and cross-corpus tasks using four well-known SER datasets in different languages are conducted for performance evaluation and comparison. The results indicate that in both tasks the CTL-MTNet showed better performance in all cases compared to a number of state-of-the-art methods. The source code and the supplementary materials are available at: https://github.com/MLDMXM2017/CTLMTNet.","doi":"10.24963/ijcai.2022/320","url":"https://ijcai.org/proceedings/2022/320","pdf":"https://ijcai.org/proceedings/2022/0320.pdf","markdown":"https://ijcai.org/proceedings/2022/320.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/320","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":320,"pages":{"first":2305,"last":2311},"keywords":["Humans and AI: Cognitive Modeling","Data Mining: Applications","Natural Language Processing: Speech","Humans and AI: Human-Computer Interaction","Machine Learning: Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986650","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984793","length_sec":334,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abdelwahab and Busso, 2018] Mohammed Abdelwahab and Carlos Busso. Domain adversarial for acoustic emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(12):2423-2435, 2018.","title":"Domain adversarial for acoustic emotion recognition","authors":"Busso Abdelwahab, Mohammed Abdelwahab, Carlos Busso","year":"2018","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akc ¸ay and Oguz, 2020] Mehmet Berkehan Akc ¸ay and Kaya Oguz. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communi- cation, 116:56-76, 2020.","title":"Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers","authors":"Akc ¸ay, ; ] Mehmet Oguz, Akc Berkehan, Kaya ¸ay, Oguz","year":"2020","venue":"Speech Communication","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2019] Mengna Gao, Jing Dong, Dongsheng Zhou, et al. End-to-end speech emotion recognition based on one-dimensional convolutional neural network. In Pro- ceedings of the 2019 3rd International Conference on In- novation in Artificial Intelligence, pages 78-82, 2019.","title":"End-to-end speech emotion recognition based on one-dimensional convolutional neural network","authors":"Gao","year":"2019","venue":"Proceedings of the 2019 3rd International Conference on Innovation in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hajarolasvadi and Demirel, 2019] Noushin Hajarolasvadi and Hasan Demirel. 3d cnn-based speech emotion recognition using k-means clustering and spectrograms. Entropy, 21(5):479, 2019.","title":"Noushin Hajarolasvadi and Hasan Demirel. 3d cnn-based speech emotion recognition using k-means clustering and spectrograms","authors":"Demirel Hajarolasvadi","year":"2019","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hong et al., 2020] Zhaojin Hong, Chenyang Wei, Yuan Zhuang, et al. Research on psychological counseling and personality analysis algorithm based on speech emotion. In International Conference on Artificial Intelligence and Security, pages 725-737. Springer, 2020.","title":"Research on psychological counseling and personality analysis algorithm based on speech emotion","authors":"Hong","year":"2020","venue":"International Conference on Artificial Intelligence and Security","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ibrahim et al., 2021] Hemin Ibrahim, Chu Kiong Loo, and Fady Alnajjar. Speech emotion recognition by late fusion for bidirectional reservoir computing with random projec- tion. IEEE Access, 9:122855-122871, 2021.","title":"Speech emotion recognition by late fusion for bidirectional reservoir computing with random projection","authors":"Hemin Ibrahim, Chu Kiong Loo, Fady Alnajjar","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ilyas, 2021] Ozer Ilyas. Pseudo-colored rate map represen- tation for speech emotion recognition. Biomedical Signal Processing and Control, 66:102502, 2021.","title":"Pseudo-colored rate map representation for speech emotion recognition","authors":"Ilyas Ozer","year":"2021","venue":"Biomedical Signal Processing and Control","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kwon, 2021] Soonil Kwon. Optimal feature selection based speech emotion recognition using two-stream deep convo- lutional neural network. International Journal of Intelli- gent Systems, 36(9):5116-5135, 2021.","title":"Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network","authors":"Soonil Kwon, Kwon","year":"2021","venue":"International Journal of Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020] Jiaxing Liu, Zhilei Liu, Longbiao Wang, et al. Speech emotion recognition with local-global aware deep representation learning. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7174-7178. IEEE, 2020.","title":"Speech emotion recognition with local-global aware deep representation learning","authors":"Jiaxing Liu, Zhilei Liu, Longbiao Wang","year":"2020","venue":"International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2017] Mingsheng Long, Zhangjie Cao, Jianmin Wang, et al. Conditional adversarial domain adaptation. arXiv preprint arXiv:1705.10667, 2017.","title":"Conditional adversarial domain adaptation","authors":"Mingsheng Long, Zhangjie Cao, Jianmin Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo and Han, 2020] Hui Luo and Jiqing Han. Nonnega- tive matrix factorization based transfer subspace learning for cross-corpus speech emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2047-2060, 2020.","title":"Nonnegative matrix factorization based transfer subspace learning for cross-corpus speech emotion recognition","authors":"Han ; ] Luo, Luo Hui, Jiqing Han","year":"2020","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, et al. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science con- ference, volume 8, pages 18-25. Citeseer, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mustaqeem and Kwon, 2021] Mustaqeem Mustaqeem and Soonil Kwon. Speech emotion recognition based on deep networks: A review. In Proceedings of the Korea Informa- tion Processing Society Conference, pages 331-334. Ko- rea Information Processing Society, 2021.","title":"Speech emotion recognition based on deep networks: A review","authors":"Kwon Mustaqeem, Mustaqeem Mustaqeem, Soonil Kwon","year":"2021","venue":"Proceedings of the Korea Information Processing Society Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sabour et al., 2017] Sara Sabour, Nicholas Frosst, and Ge- offrey E Hinton. Dynamic routing between capsules. arXiv preprint arXiv:1710.09829, 2017.","title":"Dynamic routing between capsules","authors":"Sara Sabour, Nicholas Frosst, Geoffrey E Hinton","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schuller et al., 2010] Björn Schuller, Stefan Steidl, Anton Batliner, et al. The interspeech 2010 paralinguistic chal- lenge. In Proc. INTERSPEECH 2010, Makuhari, Japan, pages 2794-2797, 2010.","title":"The interspeech 2010 paralinguistic challenge","authors":"Schuller","year":"2010","venue":"Proc. INTERSPEECH 2010","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2019] Linhui Sun, Sheng Fu, and Fu Wang. De- cision tree svm model with fisher feature selection for speech emotion recognition. EURASIP Journal on Audio, Speech, and Music Processing, 2019(1):1-14, 2019.","title":"Decision tree svm model with fisher feature selection for speech emotion recognition","authors":"Sun","year":"2019","venue":"EURASIP Journal on Audio, Speech, and Music Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tuncer et al., 2021] Turker Tuncer, Sengul Dogan, and U Rajendra Acharya. Automated accurate speech emo- tion recognition system using twine shuffle pattern and iterative neighborhood component analysis techniques. Knowledge-Based Systems, 211:106547, 2021.","title":"Automated accurate speech emotion recognition system using twine shuffle pattern and iterative neighborhood component analysis techniques","authors":"Turker Tuncer, Sengul Dogan, U Rajendra, Acharya","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Verbitskiy et al., 2021] Sergey Verbitskiy, Vladimir Berikov, Vyshegorodtsev, et al. Eranns: Efficient residual audio neural networks for audio pattern recognition. arXiv preprint arXiv:2106.01621, 2021.","title":"Eranns: Efficient residual audio neural networks for audio pattern recognition","authors":"Sergey Verbitskiy, Vladimir Berikov, Vyshegorodtsev","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2019] Xixin Wu, Songxiang Liu, Yuewen Cao, et al. Speech emotion recognition using capsule networks. In International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pages 6695-6699, 2019.","title":"Speech emotion recognition using capsule networks","authors":"Wu","year":"2019","venue":"International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yildirim et al., 2021] Serdar Yildirim, Yasin Kaya, and Fatih Kılıc ¸. A modified feature selection method based on metaheuristic algorithms for speech emotion recognition. Applied Acoustics, 173:107721, 2021.","title":"A modified feature selection method based on metaheuristic algorithms for speech emotion recognition","authors":"Serdar Yildirim, Yasin Kaya, Fatih Kılıc, ¸","year":"2021","venue":"Applied Acoustics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 7404-7413. PMLR, 2019.","title":"Bridging theory and algorithm for domain adaptation","authors":"Yuchen Zhang, Tianle Liu, Mingsheng Long, Michael Jordan","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2021] Shiqing Zhang, Ruixin Liu, et al. Deep cross-corpus speech emotion recognition: Recent ad- vances and perspectives. Frontiers in Neurorobotics, 15, 2021.","title":"Deep cross-corpus speech emotion recognition: Recent advances and perspectives","authors":"Shiqing Zhang, Ruixin Liu","year":"2021","venue":"Frontiers in Neurorobotics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2021] Kun Zhou, Berrak Sisman, Rui Liu, et al. Seen and unseen emotional style transfer for voice conver- sion with a new emotional speech dataset. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 920-924. IEEE, 2021.","title":"Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset","authors":"Kun Zhou, Berrak Sisman, Rui Liu","year":"2021","venue":"International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}