{"type":"ScholarlyArticle","title":"Automatic Recognition of Emotional Subgroups in Images","authors":[{"name":"Emmeke Veltmeijer","slug":"emmeke-veltmeijer","given":"Emmeke","family":"Veltmeijer","affiliation":"Department of Computer Science , Vrije Universiteit Amsterdam"},{"name":"Charlotte Gerritsen","slug":"charlotte-gerritsen","given":"Charlotte","family":"Gerritsen","affiliation":"Department of Computer Science , Vrije Universiteit Amsterdam"},{"name":"Koen Hindriks","slug":"koen-hindriks","given":"Koen","family":"Hindriks","affiliation":"Department of Computer Science , Vrije Universiteit Amsterdam"}],"abstract":"Both social group detection and group emotion recognition in images are growing fields of interest, but never before have they been combined. In this work we aim to detect emotional subgroups in images, which can be of great importance for crowd surveillance or event analysis. To this end, human annotators are instructed to label a set of 171 images, and their recognition strategies are analysed. Three main strategies for labeling images are identified, with each strategy assigning either 1) more weight to emotions (emotion-based fusion), 2) more weight to spatial structures (group-based fusion), or 3) equal weight to both (summation strategy). Based on these strategies, algorithms are developed to automatically recognize emotional subgroups. In particular, K-means and hierarchical clustering are used with location and emotion features derived from a fine-tuned VGG network. Additionally, we experiment with face size and gaze direction as extra input features. The best performance comes from hierarchical clustering with emotion, location and gaze direction as input.","doi":"10.24963/ijcai.2022/190","url":"https://ijcai.org/proceedings/2022/190","pdf":"https://ijcai.org/proceedings/2022/0190.pdf","markdown":"https://ijcai.org/proceedings/2022/190.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/190","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":190,"pages":{"first":1363,"last":1370},"keywords":["Machine Learning: Clustering","Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Machine Learning: Classification","Machine Learning: Applications","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986879","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985022","length_sec":317,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Badr et al., 2015] Hamada Badr, Benjamin Zaitchik, and Amin Dezfuli. A tool for hierarchical climate regional- ization. Earth Sci. Inform., 8(4):949-958, 2015.","title":"A tool for hierarchical climate regionalization","authors":"Hamada Badr, Benjamin Zaitchik, Amin Dezfuli","year":"2015","venue":"Earth Sci. Inform","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cerekovic, 2016] Aleksandra Cerekovic. A deep look into group happiness prediction from images. In ICMI 2016 - Proc. 2016 ACM Int. Conf. Multim. Interact., pages 437- 444, 2016.","title":"A deep look into group happiness prediction from images","authors":"Aleksandra Cerekovic","year":"2016","venue":"ICMI 2016 -Proc. 2016 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dhall et al., 2012] Abhinav Dhall, Jyoti Joshi, Ibrahim Rad- wan, and Roland Goecke. Finding happiest moments in a social context. In Asian Conference on Computer Vision, pages 613-626. Springer, 2012.","title":"Finding happiest moments in a social context","authors":"Abhinav Dhall, Jyoti Joshi, Ibrahim Radwan, Roland Goecke","year":"2012","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dhall et al., 2016] Abhinav Dhall, Roland Goecke, Jyoti Joshi, Jesse Hoey, and Tom Gedeon. Emotiw 2016: Video and group-level emotion recognition challenges. In ICMI 2016 -Proc. 2016 ACM Int. Conf. Multim. Interact., pages 427-432, 2016.","title":"Emotiw 2016: Video and group-level emotion recognition challenges","authors":"Abhinav Dhall, Roland Goecke, Jyoti Joshi, Jesse Hoey, Tom Gedeon","year":"2016","venue":"ICMI 2016 -Proc. 2016 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhall et al., 2018] Abhinav Dhall, Amanjot Kaur, Roland Goecke, and Tom Gedeon. Emotiw 2018: Audio-video, student engagement and group-level affect prediction. In ICMI 2018 -Proc. 2018 ACM Int. Conf. Multim. Interact., pages 653-656, 2018.","title":"Emotiw 2018: Audio-video, student engagement and group-level affect prediction","authors":"Abhinav Dhall, Amanjot Kaur, Roland Goecke, Tom Gedeon","year":"2018","venue":"ICMI 2018 -Proc. 2018 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frank, 1995] Kenneth Frank. Identifying cohesive sub- groups. Social networks, 17(1):27-56, 1995.","title":"Identifying cohesive subgroups","authors":"Kenneth Frank","year":"1995","venue":"Social networks","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2018] Xin Guo, Bin Zhu, Luisa Polanía, Charles Boncelet, and Kenneth Barner. Group-level emotion recognition using hybrid deep models based on faces, scenes, skeletons and visual attentions. In ICMI 2018 - Proc. 2018 ACM Int. Conf. Multim. Interact., pages 635- 639, 2018.","title":"Group-level emotion recognition using hybrid deep models based on faces, scenes, skeletons and visual attentions","authors":"Xin Guo, Bin Zhu, Luisa Polanía, Charles Boncelet, Kenneth Barner","year":"2018","venue":"ICMI 2018 -Proc. 2018 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Japar et al., 2021] Nurul Japar, Ven Jyn Kok, and Chee Seng Chan. Coherent group detection in still image. Multimed. Tools Appl., pages 1-20, 2021.","title":"Coherent group detection in still image","authors":"Nurul Japar, Jyn Ven, Chee Kok, Chan Seng","year":"2021","venue":"Multimed. Tools Appl","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kosti et al., 2019] Ronak Kosti, Jose Alvarez, Adria Re- casens, and Agata Lapedriza. Context based emotion recognition using emotic dataset. IEEE Trans. Pattern Analys. Mach. Intell., 2019. [Labelbox, 2021] Labelbox. https://labelbox.com, 2021. Ac- cessed: 2022-06-09.","title":"Context based emotion recognition using emotic dataset","authors":"Ronak Kosti, Jose Alvarez, Adria Recasens, Agata Lapedriza","year":"2019","venue":"IEEE Trans. Pattern Analys. Mach. Intell","doi":null,"url":"https://labelbox.com,2021.Ac-cessed:2022-06-09","internal_url":null},{"position":10,"raw":"[Leach et al., 2014] Michael Leach, Rolf Baxter, Neil Robertson, and Ed Sparks. Detecting social groups in crowded surveillance videos using visual attention. In Proc. IEEE Conf. Comput. Comput. Vis. Pattern Recognit. Workshops, pages 461-467, 2014.","title":"Detecting social groups in crowded surveillance videos using visual attention","authors":"Michael Leach, Rolf Baxter, Neil Robertson, Ed Sparks","year":"2014","venue":"Proc. IEEE Conf. Comput. Comput. Vis. Pattern Recognit. Workshops","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Jianshu Li, Sujoy Roy, Jiashi Feng, and Ter- ence Sim. Happiness level prediction with sequential in- puts via multiple regressions. In ICMI 2016 -Proc. 2016 ACM Int. Conf. Multim. Interact., pages 487-493, 2016.","title":"Happiness level prediction with sequential inputs via multiple regressions","authors":"Li","year":"2016","venue":"ICMI 2016 -Proc. 2016 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pages 2852- 2861, 2017.","title":"Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild","authors":"Shan Li, Weihong Deng, Junping Du","year":"2017","venue":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Light, 1971] Richard Light. Measures of response agree- ment for qualitative data: some generalizations and alter- natives. Psychological bulletin, 76(5):365, 1971.","title":"Measures of response agreement for qualitative data: some generalizations and alternatives","authors":"Richard Light","year":"1971","venue":"Psychological bulletin","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[MacQueen and others, 1967] James MacQueen et al. Some methods for classification and analysis of multivariate ob- servations. In Proc. 5th Berkeley Symp. Math. Stat. Prob., volume 1, pages 281-297.","title":"Some methods for classification and analysis of multivariate observations","authors":"James Macqueen","year":null,"venue":"Proc. 5th Berkeley Symp. Math. Stat. Prob","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mou et al., 2019] Wenxuan Mou, Hatice Gunes, and Ioan- nis Patras. Your fellows matter: Affect analysis across subjects in group videos. In 2019 IEEE Int. Conf. Aut. Face Gest. Rec. FG 2019, pages 1-5. IEEE, 2019.","title":"Your fellows matter: Affect analysis across subjects in group videos","authors":"Wenxuan Mou, Hatice Gunes, Ioannis Patras","year":"2019","venue":"IEEE Int. Conf. Aut. Face Gest. Rec. FG 2019","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Noordewier and Breugelmans, 2013] Marret Noordewier and Seger Breugelmans. On the valence of surprise. Cognition & emotion, 27(7):1326-1334, 2013.","title":"Marret Noordewier and Seger Breugelmans. On the valence of surprise","authors":"Breugelmans Noordewier","year":"2013","venue":"Cognition & emotion","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parkhi et al., 2015] Omkar Parkhi, Andrea Vedaldi, and An- drew Zisserman. Deep face recognition. In Proc. Brit. Mach. Vis. Conf. (BMVC), pages 41.1-41.12. BMVA Press, 2015.","title":"Deep face recognition","authors":"Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman","year":"2015","venue":"Proc. Brit. Mach. Vis. Conf. (BMVC)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Pas- sos, David Cournapeau, Matthieu Brucher, Matthieu Per- rot, and Édouard Duchesnay. Scikit-learn: Machine learn- ing in Python. J. Mach. Learn. Res., 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Édouard Duchesnay","year":"2011","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ruiz et al., 2018] Nataniel Ruiz, Eunji Chong, and James Rehg. Fine-grained head pose estimation without key- points. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) Workshops, June 2018.","title":"Fine-grained head pose estimation without keypoints","authors":"Nataniel Ruiz, Eunji Chong, James Rehg","year":"2018","venue":"IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) Workshops","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan et al., 2017] Lianzhi Tan, Kaipeng Zhang, Kai Wang, Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao. Group emo- tion recognition with individual facial emotion cnns and global image based cnns. In ICMI 2017 -Proc. 2017 ACM Int. Conf. Multim. Interact., pages 549-552, 2017.","title":"Group emotion recognition with individual facial emotion cnns and global image based cnns","authors":"Tan","year":"2017","venue":"ICMI 2017 -Proc. 2017 ACM Int. Conf. Multim. Interact","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tarasov and Savchenko, 2018] Alexander Tarasov and An- drey Savchenko. Emotion recognition of a group of people in video analytics using deep off-the-shelf image embed- dings. In Int. Conf. Analys. Images Soc. Netw. Texts, pages 191-198. Springer, 2018.","title":"Alexander Tarasov and Andrey Savchenko. Emotion recognition of a group of people in video analytics using deep off-the-shelf image embeddings","authors":"Savchenko Tarasov","year":"2018","venue":"Int. Conf. Analys. Images Soc. Netw. Texts","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thorndike, 1953] Robert Thorndike. Who belongs in the family? Psychometrika, 18(4):267-276, 1953.","title":"Robert Thorndike. Who belongs in the family?","authors":"Thorndike","year":"1953","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vascon et al., 2016] Sebastiano Vascon, Eyasu Mequanint, Marco Cristani, Hayley Hung, Marcello Pelillo, and Vit- torio Murino. Detecting conversational groups in images and sequences: A robust game-theoretic approach. Com- put. Vis. Image Underst., 143:11-24, 2016.","title":"Detecting conversational groups in images and sequences: A robust game-theoretic approach","authors":"Sebastiano Vascon, Eyasu Mequanint, Marco Cristani, Hayley Hung, Marcello Pelillo, Vittorio Murino","year":"2016","venue":"Comput. Vis. Image Underst","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Veltmeijer et al., 2021] Emmeke Veltmeijer, Charlotte Ger- ritsen, and Koen Hindriks. Automatic emotion recognition for groups: a review. IEEE Transactions on Affective Com- puting, 2021.","title":"Automatic emotion recognition for groups: a review","authors":"Veltmeijer","year":"2021","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Qi Wang, Mulin Chen, Feiping Nie, and Xuelong Li. Detecting coherent groups in crowd scenes by multiview clustering. IEEE Trans. Pattern Analys. Mach. Intell., 42(1):46-58, 2018.","title":"Detecting coherent groups in crowd scenes by multiview clustering","authors":"Qi Wang, Mulin Chen, Feiping Nie, Xuelong Li","year":"2018","venue":"IEEE Trans. Pattern Analys. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ward Jr, 1963] Joe Ward Jr. Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc., 58(301):236-244, 1963.","title":"Hierarchical grouping to optimize an objective function","authors":"Joe Ward","year":"1963","venue":"J. Am. Stat. Assoc","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yücel et al., 2013] Zeynep Yücel, Francesco Zanlungo, Tetsushi Ikeda, Takahiro Miyashita, and Norihiro Hagita. Deciphering the crowd: Modeling and identification of pedestrian group motion. Sensors, 13(1):875-897, 2013.","title":"Deciphering the crowd: Modeling and identification of pedestrian group motion","authors":"Zeynep Yücel, Francesco Zanlungo, Tetsushi Ikeda, Takahiro Miyashita, Norihiro Hagita","year":"2013","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019] Meng Zhang, Xinchen Liu, Wu Liu, Anfu Zhou, Huadong Ma, and Tao Mei. Multi-granularity reasoning for social relation recognition from images. In 2019 IEEE Int. Conf. Multim. Expo (ICME), pages 1618- 1623. IEEE, 2019.","title":"Multi-granularity reasoning for social relation recognition from images","authors":"Meng Zhang, Xinchen Liu, Wu Liu, Anfu Zhou, Huadong Ma, Tao Mei","year":"2019","venue":"IEEE Int. Conf. Multim. Expo (ICME)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}