{"type":"ScholarlyArticle","title":"Visual Emotion Representation Learning via Emotion-Aware Pre-training","authors":[{"name":"Yue Zhang","slug":"yue-zhang","given":"Yue","family":"Zhang","affiliation":"Drexel University , College of Computing & Informatics , Philadelphia , PA , USA"},{"name":"Wanying Ding","slug":"wanying-ding","given":"Wanying","family":"Ding","affiliation":"JPMorgan Chase & Co. , Palo Alto , CA , USA"},{"name":"Ran Xu","slug":"ran-xu","given":"Ran","family":"Xu","affiliation":"Salesforce Research , Palo Alto , CA , USA"},{"name":"Xiaohua Hu","slug":"xiaohua-hu","given":"Xiaohua","family":"Hu","affiliation":"Drexel University , College of Computing & Informatics , Philadelphia , PA , USA"}],"abstract":"Despite recent progress in deep learning, visual emotion recognition remains a challenging problem due to ambiguity of emotion perception, diverse concepts related to visual emotion and lack of large-scale annotated dataset. In this paper, we present a large-scale multimodal pre-training method to learn visual emotion representation by aligning emotion, object, attribute triplet with a contrastive loss. We conduct our pre-training on a large web dataset with noisy tags and fine-tune on visual emotion classification datasets. Our method achieves state-of-the-art performance for visual emotion classification.","doi":"10.24963/ijcai.2022/234","url":"https://ijcai.org/proceedings/2022/234","pdf":"https://ijcai.org/proceedings/2022/0234.pdf","markdown":"https://ijcai.org/proceedings/2022/234.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/234","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":234,"pages":{"first":1679,"last":1685},"keywords":["Computer Vision: Vision and language","Natural Language Processing: Sentiment Analysis and Text Mining","Humans and AI: Cognitive Modeling"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986950","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985093","length_sec":336,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.","title":"Uniter: Universal image-text representation learning","authors":"Chen","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gan et al., 2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS, 2020.","title":"Large-scale adversarial training for vision-and-language representation learning","authors":"Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Han et al., 2021] Xiaotian Han, Jianwei Yang, Houdong Hu, Lei Zhang, Jianfeng Gao, and Pengchuan Zhang. Image scene graph generation (sgg) benchmark. arXiv preprint arXiv:2107.12604, 2021.","title":"Image scene graph generation (sgg) benchmark","authors":"Xiaotian Han, Jianwei Yang, Houdong Hu, Lei Zhang, Jianfeng Gao, Pengchuan Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jia et al., 2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. ICML, 2021.","title":"Scaling up visual and vision-language representation learning with noisy text supervision","authors":"Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, Tom Duerig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kosti et al., 2017] Ronak Kosti, Jose M. Alvarez, Adria Re- casens, and Agata Lapedriza. Emotion recognition in con- text. In CVPR, 2017.","title":"Emotion recognition in context","authors":"Ronak Kosti, Jose M Alvarez, Adria Recasens, Agata Lapedriza","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kosti et al., 2020] Ronak Kosti, Jose M. Alvarez, Adria Re- casens, and Agata Lapedriza. Context based emotion recognition using emotic dataset. In TPAMI, 2020.","title":"Context based emotion recognition using emotic dataset","authors":"Ronak Kosti, Jose M Alvarez, Adria Recasens, Agata Lapedriza","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. In IJCV, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Fei-Fei Li","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krylov et al., 2021] Ilya Krylov, Sergei Nosov, and Vladislav Sovrasov. Open images v5 text annotation and yet another mask text spotter. In ACML. PMLR, 2021.","title":"Open images v5 text annotation and yet another mask text spotter","authors":"Krylov","year":"2021","venue":"ACML. PMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision- language tasks. In ECCV, 2020.","title":"Oscar: Object-semantics aligned pre-training for visionlanguage tasks","authors":"Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.","title":"Align before fuse: Vision and language representation learning with momentum distillation","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visi- olinguistic representations for vision-and-language tasks.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"NeurIPS, 2019.","title":null,"authors":"Neurips","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Machajdik and Hanbury, 2010] Jana Machajdik and Allan Hanbury. Affective image classification using features in- spired by psychology and art theory. In ACM-MM, 2010. Panda et al., 2018] Rameswar Panda, Jianming Zhang, Haoxiang Li, Joon-Young Lee, Xin Lu, and Amit K. Roy- Chowdhury. Contemplating visual emotions: Understand- ing and overcoming dataset bias. In ECCV, 2018.","title":"Affective image classification using features inspired by psychology and art theory","authors":"Hanbury Machajdik, Jana Machajdik, Allan Hanbury, ; Panda","year":"2010","venue":"ACM-MM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2015] Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew Gallagher. A mixed bag of emo- tions: Model, predict, and transfer emotion distributions. In CVPR, 2015.","title":"A mixed bag of emotions: Model, predict, and transfer emotion distributions","authors":"Kuan-Chuan, Tsuhan Peng, Amir Chen, Andrew Sadovnik, Gallagher","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peng et al., 2016] Kuan-Chuan Peng, Amir Sadovnik, An- drew Gallagher, and Tsuhan Chen. Where do emotions come from? predicting the emotion stimuli map. In ICIP, 2016.","title":"Where do emotions come from? predicting the emotion stimuli map","authors":"Kuan-Chuan, Amir Peng, Andrew Sadovnik, Tsuhan Gallagher, Chen","year":"2016","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Iiya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Iiya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Su et al., 2020] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. ICLR, 2020.","title":"Vl-bert: Pre-training of generic visual-linguistic representations","authors":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2013] Xiaohui Wang, Jia Jia, Jiaming Yin, and Lianhong Cai. Interpretable aesthetic features for affective image classification. In ICIP, 2013.","title":"Interpretable aesthetic features for affective image classification","authors":"Xiaohui Wang, Jia Jia, Jiaming Yin, Lianhong Cai","year":"2013","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2020] Zijun Wei, Jianming Zhang, Zhe Lin, Joon-Young Lee, Niranjan Balasubramanian, Minh Hoai, and Dimitris Samaras. Learning visual emotion represen- tations from web data. In CVPR, 2020.","title":"Learning visual emotion representations from web data","authors":"Zijun Wei, Jianming Zhang, Zhe Lin, Joon-Young Lee, Niranjan Balasubramanian, Minh Hoai, Dimitris Samaras","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2018a] Jufeng Yang, Dongyu She, Yu-kun Lai, Paul L. Rosin, and Ming-Hsuan Yang. Weakly supervised coupled networks for visual sentiment analysis. In CVPR, 2018.","title":"Weakly supervised coupled networks for visual sentiment analysis","authors":"Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2018b] Jufeng Yang, Dongyu She, Ming Sun, Ming-Ming Cheng, Rosin Paul L., and Liang Wang. Vi- sual sentiment prediction based on automatic discovery of affective regions. In IEEE Transactions on Multimedia, 2018.","title":"Visual sentiment prediction based on automatic discovery of affective regions","authors":"Yang","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[You et al., 2015] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Robust image sentiment analysis using progressively trained and domain transferred deep net- works. In AAAI, 2015.","title":"Robust image sentiment analysis using progressively trained and domain transferred deep networks","authors":"You","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[You et al., 2016] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In AAAI, 2016.","title":"Building a large scale dataset for image emotion recognition: The fine print and the benchmark","authors":"You","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021.","title":"Vinvl: Revisiting visual representations in vision-language models","authors":"Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}