{"type":"ScholarlyArticle","title":"Personality-Aware Personalized Emotion Recognition from Physiological Signals","authors":[{"name":"Sicheng Zhao","slug":"sicheng-zhao","given":"Sicheng","family":"Zhao","affiliation":"† School of Software , Tsinghua University , China; ‡ Department of Electrical Engineering and Computer Sciences , University of California , Berkeley , USA"},{"name":"Guiguang Ding","slug":"guiguang-ding","given":"Guiguang","family":"Ding","affiliation":"† School of Software , Tsinghua University , China"},{"name":"Jungong Han","slug":"jungong-han","given":"Jungong","family":"Han","affiliation":"School of Computing and Communications , Lancaster University , UK"},{"name":"Yue Gao","slug":"yue-gao","given":"Yue","family":"Gao","affiliation":"† School of Software , Tsinghua University , China"}],"abstract":"Emotion recognition methodologies from physiological signals are increasingly becoming personalized, due to the subjective responses of different subjects to physical stimuli. Existing works mainly focused on modelling the involved physiological corpus of each subject, without considering the psychological factors. The latent correlation among different subjects has also been rarely examined. We propose to investigate the influence of personality on emotional behavior in a hypergraph learning framework. Assuming that each vertex is a compound tuple (subject, stimuli), multi-modal hypergraphs can be constructed based on the personality correlation among different subjects and on the physiological correlation among corresponding stimuli. To reveal the different importance of vertices, hyperedges, and modalities, we assign each of them with weights. The emotion relevance learned on the vertex-weighted multi-modal multi-task hypergraphs is employed for emotion recognition. We carry out extensive experiments on the ASCERTAIN dataset and the results demonstrate the superiority of the proposed method.","doi":"10.24963/ijcai.2018/230","url":"https://ijcai.org/proceedings/2018/230","pdf":"https://ijcai.org/proceedings/2018/0230.pdf","markdown":"https://ijcai.org/proceedings/2018/230.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/230","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":230,"pages":{"first":1660,"last":1667},"keywords":["Uncertainty in AI: Uncertainty in AI","Humans and AI: Personalization and User Modeling","Machine Learning Applications: Other Applications"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015a] Mojtaba Khomami Abadi, Juan Abdón Mi- randa Correa, Julia Wache, Heng Yang, Ioannis Patras, and Nicu Sebe. Inference of personality traits and affect schedule by anal- ysis of spontaneous reactions to affective videos. In IEEE FGR, volume 1, pages 1-8, 2015.","title":"Inference of personality traits and affect schedule by analysis of spontaneous reactions to affective videos","authors":"Abadi","year":"2015","venue":"IEEE FGR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abadi et al., 2015b] Mojtaba Khomami Abadi, Ramanathan Sub- ramanian, Seyed Mostafa Kia, Paolo Avesani, Ioannis Patras, and Nicu Sebe. Decaf: Meg-based multimodal database for decoding affective physiological responses. IEEE TAFFC, 6(3):209-222, 2015.","title":"Decaf: Meg-based multimodal database for decoding affective physiological responses","authors":"Abadi","year":"2015","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Atrey et al., 2010] Pradeep K Atrey, M Anwar Hossain, Abdul- motaleb El Saddik, and Mohan S Kankanhalli. Multimodal fu- sion for multimedia analysis: a survey. Multimedia Systems, 16(6):345-379, 2010.","title":"Multimodal fusion for multimedia analysis: a survey","authors":"M Pradeep K Atrey, Abdulmotaleb El Anwar Hossain, Mohan S Saddik, Kankanhalli","year":"2010","venue":"Multimedia Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bu et al., 2010] Jiajun Bu, Shulong Tan, Chun Chen, Can Wang, Hao Wu, Lijun Zhang, and Xiaofei He. Music recommendation by unified hypergraph: combining social media information and music content. In ACM MM, pages 391-400, 2010.","title":"Music recommendation by unified hypergraph: combining social media information and music content","authors":"Jiajun Bu, Shulong Tan, Chun Chen, Can Wang, Hao Wu, Lijun Zhang, Xiaofei He","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Camilleri et al., 2017] Elizabeth Camilleri, Georgios N Yan- nakakis, and Antonios Liapis. Towards general models of player affect. In ACII, pages 333-339, 2017.","title":"Towards general models of player affect","authors":"Elizabeth Camilleri, Georgios N Yannakakis, Antonios Liapis","year":"2017","venue":"ACII","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Costa and MacCrae, 1992] Paul T Costa and Robert R MacCrae. Revised NEO personality inventory (NEO PI-R) and NEO five- factor inventory (NEO-FFI): Professional manual. Psychological Assessment Resources, Incorporated, 1992.","title":"Revised NEO personality inventory (NEO PI-R) and NEO fivefactor inventory (NEO-FFI): Professional manual","authors":"T Paul, Robert R Costa, Maccrae","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[D'mello and Kory, 2015] Sidney K D'mello and Jacqueline Kory. A review and meta-analysis of multimodal affect detection sys- tems. ACM CSUR, 47(3):43, 2015.","title":"A review and meta-analysis of multimodal affect detection systems","authors":"Kory ; Sidney, K, Jacqueline Kory","year":"2015","venue":"ACM CSUR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frijda, 1986] Nico H Frijda. The emotions. Cambridge University Press, 1986.","title":"The emotions","authors":"H Nico, Frijda","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2012] Yue Gao, Meng Wang, Dacheng Tao, Rongrong Ji, and Qionghai Dai. 3-d object retrieval and recognition with hypergraph analysis. IEEE TIP, 21(9):4290-4303, 2012.","title":"3-d object retrieval and recognition with hypergraph analysis","authors":"Yue Gao, Meng Wang, Dacheng Tao, Rongrong Ji, Qionghai Dai","year":"2012","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2013] Yue Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, and Xindong Wu. Visual-textual joint rel- evance learning for tag-based social image search. IEEE TIP, 22(1):363-376, 2013.","title":"Visual-textual joint relevance learning for tag-based social image search","authors":"Yue Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, Xindong Wu","year":"2013","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Juslin and Laukka, 2004] Patrik N Juslin and Petri Laukka. Ex- pression, perception, and induction of musical emotions: A re- view and a questionnaire study of everyday listening. Journal of New Music Research, 33(3):217-238, 2004.","title":"Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening","authors":"N Patrik, Petri Juslin, Laukka","year":"2004","venue":"Journal of New Music Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kehoe et al., 2012] Elizabeth G Kehoe, John M Toomey, Joshua H Balsters, and Arun LW Bokde. Personality modulates the effect- s of emotional arousal and valence on brain activation. Social Cognitive and Affective Neuroscience, 7(7):858-870, 2012.","title":"Personality modulates the effects of emotional arousal and valence on brain activation","authors":"John M Elizabeth G Kehoe, Joshua H Toomey, Arun Lw Balsters, Bokde","year":"2012","venue":"Social Cognitive and Affective Neuroscience","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim and André, 2008] Jonghwa Kim and Elisabeth André. Emo- tion recognition based on physiological changes in music listen- ing. IEEE TPAMI, 30(12):2067-2083, 2008.","title":"Emotion recognition based on physiological changes in music listening","authors":"André ; Kim, Jonghwa Kim, Elisabeth André","year":"2008","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koelstra et al., 2012] Sander Koelstra, Christian Muhl, Moham- mad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj E- brahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals. IEEE TAFFC, 3(1):18-31, 2012.","title":"Deap: A database for emotion analysis; using physiological signals","authors":"Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Anton Thierry Pun, Ioannis Nijholt, Patras","year":"2012","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lisetti and Nasoz, 2004] Christine Lætitia Lisetti and Fatma Na- soz. Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Ad- vances in Signal Processing, 2004(11):929414, 2004.","title":"Using noninvasive wearable computers to recognize human emotions from physiological signals","authors":"Christine Lætitia, Lisetti, Fatma Nasoz","year":"2004","venue":"EURASIP Journal on Advances in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Martinez et al., 2013] Hector P Martinez, Yoshua Bengio, and Georgios N Yannakakis. Learning deep physiological models of affect. IEEE CIM, 8(2):20-33, 2013.","title":"Learning deep physiological models of affect","authors":"Yoshua Hector P Martinez, Georgios N Bengio, Yannakakis","year":"2013","venue":"IEEE CIM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ngiam et al., 2011] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, pages 689-696, 2011.","title":"Multimodal deep learning","authors":"Ngiam","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perugini and Di Blas, 2002] Marco Perugini and Lisa Di Blas. An- alyzing personality related adjectives from an eticemic perspec- tive: the big five marker scales (bfms) and the italian ab5c taxon- omy. Big Five Assessment, pages 281-304, 2002.","title":"Analyzing personality related adjectives from an eticemic perspective: the big five marker scales (bfms) and the italian ab5c taxonomy. Big Five Assessment","authors":"Di Perugini, ; Blas, Marco Perugini, Lisa Di Blas","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Purkait et al., 2017] Pulak Purkait, Tat-Jun Chin, Alireza Sadri, and David Suter. Clustering with hypergraphs: the case for large hyperedges. IEEE TPAMI, 39(9):1697-1711, 2017.","title":"Clustering with hypergraphs: the case for large hyperedges","authors":"Pulak Purkait, Tat-Jun Chin, Alireza Sadri, David Suter","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shu and Wang, 2017] Yangyang Shu and Shangfei Wang. Emotion recognition through integrating eeg and peripheral signals. In ICASSP, pages 2871-2875, 2017.","title":"Emotion recognition through integrating eeg and peripheral signals","authors":"Wang ; Shu, Yangyang Shu, Shangfei Wang","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Soleymani et al., 2012] Mohammad Soleymani, Jeroen Lichte- nauer, Thierry Pun, and Maja Pantic. A multimodal database for affect recognition and implicit tagging. IEEE TAFFC, 3(1):42- 55, 2012.","title":"A multimodal database for affect recognition and implicit tagging","authors":"Mohammad Soleymani, Jeroen Lichtenauer, Maja Thierry Pun, Pantic","year":"2012","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Su et al., 2017] Lifan Su, Yue Gao, Xibin Zhao, Hai Wan, Ming Gu, and Jiaguang Sun. Vertex-weighted hypergraph learning for multi-view object classification. In IJCAI, pages 2779-2785, 2017.","title":"Vertex-weighted hypergraph learning for multi-view object classification","authors":"Lifan Su, Yue Gao, Xibin Zhao, Hai Wan, Ming Gu, Jiaguang Sun","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/387"},{"position":23,"raw":"[Subramanian et al., 2014] Ramanathan Subramanian, Divya Shankar, Nicu Sebe, and David Melcher. Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. Journal of Vision, 14(3):31:1-31:18, 2014.","title":"Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes","authors":"Ramanathan Subramanian, Divya Shankar, Nicu Sebe, David Melcher","year":"2014","venue":"Journal of Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Subramanian et al., 2016] Ramanathan Subramanian, Julia Wache, Mojtaba Abadi, Radu Vieriu, Stefan Winkler, and Nicu Sebe. Ascertain: Emotion and personality recognition using commercial sensors. IEEE TAFFC, 2016.","title":"Ascertain: Emotion and personality recognition using commercial sensors","authors":"Ramanathan Subramanian, Julia Wache, Mojtaba Abadi, Radu Vieriu, Stefan Winkler, Nicu Sebe","year":"2016","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tognetti et al., 2010] Simone Tognetti, Maurizio Garbarino, An- drea Bonarini, and Matteo Matteucci. Modeling enjoyment pref- erence from physiological responses in a car racing game. In IEEE CIG, pages 321-328, 2010.","title":"Modeling enjoyment preference from physiological responses in a car racing game","authors":"Simone Tognetti, Maurizio Garbarino, Andrea Bonarini, Matteo Matteucci","year":"2010","venue":"IEEE CIG","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van Lankveld et al., 2011] Giel Van Lankveld, Pieter Spronck, Jaap Van den Herik, and Arnoud Arntz. Games as personality profiling tools. In IEEE CIG, pages 197-202, 2011.","title":"Games as personality profiling tools","authors":"Giel Van Lankveld, Pieter Spronck, Jaap Van Den, Arnoud Herik, Arntz","year":"2011","venue":"IEEE CIG","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinciarelli and Mohammadi, 2014] Alessandro Vinciarelli and Gelareh Mohammadi. A survey of personality computing. IEEE TAFFC, 5(3):273-291, 2014.","title":"A survey of personality computing","authors":"Alessandro Vinciarelli, Gelareh Mohammadi","year":"2014","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wagner et al., 2011] Johannes Wagner, Elisabeth Andre, Florian Lingenfelser, and Jonghwa Kim. Exploring fusion methods for multimodal emotion recognition with missing data. IEEE TAFFC, 2(4):206-218, 2011.","title":"Exploring fusion methods for multimodal emotion recognition with missing data","authors":"Wagner","year":"2011","venue":"IEEE TAFFC","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2009] Meng Wang, Xian-Sheng Hua, Richang Hong, Jinhui Tang, Guo-Jun Qi, and Yan Song. Unified video annota- tion via multigraph learning. IEEE TCSVT, 19(5):733-746, 2009.","title":"Unified video annotation via multigraph learning","authors":"Meng Wang, Xian-Sheng Hua, Richang Hong, Jinhui Tang, Guo-Jun Qi, Yan Song","year":"2009","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wen et al., 2014] Longyin Wen, Wenbo Li, Junjie Yan, Zhen Lei, Dong Yi, and Stan Z Li. Multiple target tracking based on undi- rected hierarchical relation hypergraph. In CVPR, pages 1282- 1289, 2014.","title":"Multiple target tracking based on undirected hierarchical relation hypergraph","authors":"Longyin Wen, Wenbo Li, Junjie Yan, Zhen Lei, Dong Yi, Stan Z Li","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Winter and Kuiper, 1997] Kathy A Winter and Nicholas A Kuiper. Individual differences in the experience of emotions. Clinical Psychology Review, 17(7):791-821, 1997.","title":"Individual differences in the experience of emotions","authors":"A Kathy, Nicholas A Winter, Kuiper","year":"1997","venue":"Clinical Psychology Review","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2014] Yang Yang, Jia Jia, Shumei Zhang, Boya Wu, Qicong Chen, Juanzi Li, Chunxiao Xing, and Jie Tang. How do your friends on social media disclose your emotions? In AAAI, pages 306-312, 2014.","title":"How do your friends on social media disclose your emotions?","authors":"Yang Yang, Jia Jia, Shumei Zhang, Boya Wu, Qicong Chen, Juanzi Li, Chunxiao Xing, Jie Tang","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yannakakis et al., 2017] Georgios N Yannakakis, Roddy Cowie, and Carlos Busso. The ordinal nature of emotions. In ACII, pages 248-255, 2017.","title":"The ordinal nature of emotions","authors":"Roddy Georgios N Yannakakis, Carlos Cowie, Busso","year":"2017","venue":"ACII","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yao et al., 2016] Chao Yao, Jimin Xiao, Tammam Tillo, Yao Zhao, Chunyu Lin, and Huihui Bai. Depth map down-sampling and coding based on synthesized view distortion. IEEE TMM, 18(10):2015-2022, 2016.","title":"Depth map down-sampling and coding based on synthesized view distortion","authors":"Chao Yao, Jimin Xiao, Tammam Tillo, Yao Zhao, Chunyu Lin, Huihui Bai","year":"2016","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2016] Sicheng Zhao, Hongxun Yao, Yue Gao, Ron- grong Ji, Wenlong Xie, Xiaolei Jiang, and Tat-Seng Chua. Pre- dicting personalized emotion perceptions of social images. In ACM MM, pages 1385-1394, 2016.","title":"Predicting personalized emotion perceptions of social images","authors":"Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, Tat-Seng Chua","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao et al., 2017a] Sicheng Zhao, Guiguang Ding, Yue Gao, and Jungong Han. Approximating discrete probability distribution of image emotions by multi-modal features fusion. In IJCAI, pages 466-4675, 2017.","title":"Approximating discrete probability distribution of image emotions by multi-modal features fusion","authors":"Zhao","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/651"},{"position":37,"raw":"[Zhao et al., 2017b] Sicheng Zhao, Yue Gao, Guiguang Ding, and Tat-Seng Chua. Real-time multimedia social event detection in microblog. IEEE TCYB, 2017.","title":"Real-time multimedia social event detection in microblog","authors":"Zhao","year":"2017","venue":"IEEE TCYB","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhao et al., 2017c] Sicheng Zhao, Hongxun Yao, Yue Gao, Ron- grong Ji, and Guiguang Ding. Continuous probability distribution prediction of image emotions via multitask shared sparse regres- sion. IEEE TMM, 19(3):632-645, 2017.","title":"Continuous probability distribution prediction of image emotions via multitask shared sparse regression","authors":"Zhao","year":"2017","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2006] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clustering, classifica- tion, and embedding. In NIPS, pages 1601-1608, 2006.","title":"Learning with hypergraphs: Clustering, classification, and embedding","authors":"Zhou","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}