{"type":"ScholarlyArticle","title":"Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition","authors":[{"name":"Xinge Zhu","slug":"xinge-zhu","given":"Xinge","family":"Zhu","affiliation":"University of Chinese Academy of Sciences , China"},{"name":"Liang Li","slug":"liang-li","given":"Liang","family":"Li","affiliation":"Key Lab of Intell. Info. Process. , Inst. of Comput. Tech. , CAS , China"},{"name":"Weigang Zhang","slug":"weigang-zhang","given":"Weigang","family":"Zhang","affiliation":"University of Chinese Academy of Sciences , China; Harbin Institute of Technology , Weihai , China"},{"name":"Tianrong Rao","slug":"tianrong-rao","given":"Tianrong","family":"Rao","affiliation":"University of Technology , Sydney , Australia"},{"name":"Min Xu","slug":"min-xu","given":"Min","family":"Xu","affiliation":"University of Technology , Sydney , Australia"},{"name":"Qingming Huang","slug":"qingming-huang","given":"Qingming","family":"Huang","affiliation":"University of Chinese Academy of Sciences , China; Key Lab of Intell. Info. Process. , Inst. of Comput. Tech. , CAS , China"},{"name":"Dong Xu","slug":"dong-xu","given":"Dong","family":"Xu","affiliation":"University of Sydney , Australia"}],"abstract":"Visual emotion recognition aims to associate images with appropriate emotions. There are different visual stimuli that can affect human emotion from low-level to high-level, such as color, texture, part, object, etc. However, most existing methods treat different levels of features as independent entity without having effective method for feature fusion. In this paper, we propose a unified CNN-RNN model to predict the emotion based on the fused features from different levels by exploiting the dependency among them. Our proposed architecture leverages convolutional neural network (CNN) with multiple layers to extract different levels of features with in a multi-task learning framework, in which two related loss functions are introduced to learn the feature representation. Considering the dependencies within the low-level and high-level features, a new bidirectional recurrent neural network (RNN) is proposed to integrate the learned features from different layers in the CNN model. Extensive experiments on both Internet images and art photo datasets demonstrate that our method outperforms the state-of-the-art methods with at least 7% performance improvement.","doi":"10.24963/ijcai.2017/503","url":"https://ijcai.org/proceedings/2017/503","pdf":"https://ijcai.org/proceedings/2017/0503.pdf","markdown":"https://ijcai.org/proceedings/2017/503.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/503","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":503,"pages":{"first":3595,"last":3601},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Classification","Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Caruana, 1998] Rich Caruana. Multitask learning. In Learning to learn, pages 95-133. Springer, 1998.","title":"Multitask learning","authors":"Rich Caruana","year":"1998","venue":"Learning to learn","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung et al., 2015] Junyoung Chung, Kyle Kastner, Lau- rent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In NIPS, pages 2980-2988, 2015.","title":"A recurrent latent variable model for sequential data","authors":"Chung","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collobert et al., 2011] Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab- like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011.","title":"Torch7: A matlablike environment for machine learning","authors":"Ronan Collobert, Koray Kavukcuoglu, Clément Farabet","year":"2011","venue":"BigLearn, NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, pages 647-655, 2014.","title":"Decaf: A deep convolutional activation feature for generic visual recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv:1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Joshi et al., 2011] Dhiraj Joshi, Ritendra Datta, Elena Fe- dorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo. Aesthetics and emotions in images. IEEE Signal Processing Magazine, 28(5):94-115, 2011.","title":"Aesthetics and emotions in images","authors":"Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, Jiebo Luo","year":"2011","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kang, 2003] Hang-Bong Kang. Affective content detection using hmms. In ACM MM, pages 259-262. ACM, 2003.","title":"Affective content detection using hmms","authors":"Hang-Bong Kang, Kang","year":"2003","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lang, 1979] Peter J Lang. A bio-informational theory of emotional imagery. Psychophysiology, 16(6):495-512, 1979.","title":"A bio-informational theory of emotional imagery","authors":"Lang Peter","year":"1979","venue":"Psychophysiology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2012] Liang. Li, Shuqiang. Jiang, and Qingming. Huang. Learning hierarchical semantic description via mixed-norm regularization for image understanding. IEEE Trans. on Multimedia, 14(5), 2012.","title":"Learning hierarchical semantic description via mixed-norm regularization for image understanding","authors":"Liang, Shuqiang Li, Qingming Jiang, Huang","year":"2012","venue":"IEEE Trans. on Multimedia","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2015] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen- Change Loy, and Xiaoou Tang. Semantic image segmenta- tion via deep parsing network. In ICCV, pages 1377-1385, 2015.","title":"Semantic image segmentation via deep parsing network","authors":"Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen-Change Loy, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luong et al., 2015] Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. arXiv:1511.06114, 2015.","title":"Multi-task sequence to sequence learning","authors":"Minh-Thang Luong, V Quoc, Ilya Le, Oriol Sutskever, Lukasz Vinyals, Kaiser","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Machajdik and Hanbury, 2010] Jana Machajdik and Allan Hanbury. Affective image classification using features in- spired by psychology and art theory. In ACM MM, pages 83-92. ACM, 2010.","title":"Affective image classification using features inspired by psychology and art theory","authors":"Hanbury Machajdik, Jana Machajdik, Allan Hanbury","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikels et al., 2005] Joseph A Mikels, Barbara L Fredrick- son, Gregory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz. Emotional category data on images from the international affective picture system. Behavior research methods, 37(4):626-630, 2005.","title":"Emotional category data on images from the international affective picture system","authors":"Barbara L Joseph A Mikels, Fredrickson, Casey M Gregory R Larkin, Sam J Lindberg, Patricia A Maglio, Reuter-Lorenz","year":"2005","venue":"Behavior research methods","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rao et al., 2016a] Tianrong Rao, Min Xu, Huiying Liu, Jin- qiao Wang, and Ian Burnett. Multi-scale blocks based im- age emotion classification using multiple instance learn- ing. In ICIP 2016, pages 634-638. IEEE, 2016.","title":"Multi-scale blocks based image emotion classification using multiple instance learning","authors":"Rao","year":"2016","venue":"ICIP 2016","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rao et al., 2016b] Tianrong Rao, Min Xu, and Dong Xu. Learning multi-level deep representations for image emo- tion classification. arXiv:1611.07145, 2016.","title":"Learning multi-level deep representations for image emotion classification","authors":"Rao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sartori et al., 2015] Andreza Sartori, Dubravko Culibrk, Yan Yan, and Nicu Sebe. Who's afraid of itten: Using the art theory of color combination to analyze emotions in abstract paintings. In ACM MM, pages 311-320. ACM, 2015.","title":"Who's afraid of itten: Using the art theory of color combination to analyze emotions in abstract paintings","authors":"Andreza Sartori, Dubravko Culibrk, Yan Yan, Nicu Sebe","year":"2015","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Siersdorfer et al., 2010] Stefan Siersdorfer, Enrico Minack, Fan Deng, and Jonathon Hare. Analyzing and predicting sentiment of images on the social web. In ACM MM, pages 715-718. ACM, 2010.","title":"Analyzing and predicting sentiment of images on the social web","authors":"Stefan Siersdorfer, Enrico Minack, Fan Deng, Jonathon Hare","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Solli and Lenz, 2009] Martin Solli and Reiner Lenz. Color based bags-of-emotions. In International Conference on Computer Analysis of Images and Patterns, pages 573- 580. Springer, 2009.","title":"Color based bags-of-emotions","authors":"Lenz Solli, Martin Solli, Reiner Lenz","year":"2009","venue":"International Conference on Computer Analysis of Images and Patterns","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4(2), 2012.","title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and He, 2008] Weining Wang and Qianhua He. A survey on emotional semantic image retrieval. In ICIP, pages 117-120, 2008.","title":"A survey on emotional semantic image retrieval","authors":"Weining Wang, Qianhua He","year":"2008","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yanulevskaya et al., 2008] Victoria Yanulevskaya, JC Van Gemert, Katharina Roth, Ann-Katrin Her- bold, Nicu Sebe, and Jan-Mark Geusebroek. Emotional valence categorization using holistic image features. In ICIP, pages 101-104. IEEE, 2008.","title":"Emotional valence categorization using holistic image features","authors":"Victoria Yanulevskaya, J C Van Gemert, Katharina Roth, Ann-Katrin Herbold, Nicu Sebe, Jan-Mark Geusebroek","year":"2008","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. arXiv:1605.02677, 2016.","title":"Building a large scale dataset for image emotion recognition: The fine print and the benchmark","authors":"You","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yue-Hei Ng et al., 2015] Joe Yue-Hei Ng, Fan Yang, and Larry S Davis. Exploiting local features from deep net- works for image retrieval. In CVPR Workshops, pages 53- 61, 2015.","title":"Exploiting local features from deep networks for image retrieval","authors":"Yue-Hei Ng","year":"2015","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2014] Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. Ex- ploring principles-of-art features for image emotion recog- nition. In ACM MM, pages 47-56. ACM, 2014.","title":"Exploring principles-of-art features for image emotion recognition","authors":"Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, Xiaoshuai Sun","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2014] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv:1412.6856, 2014.","title":"Object detectors emerge in deep scene cnns","authors":"Zhou","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}