{"type":"ScholarlyArticle","title":"Approximating Discrete Probability Distribution of Image Emotions by Multi-Modal Features Fusion","authors":[{"name":"Sicheng Zhao","slug":"sicheng-zhao","given":"Sicheng","family":"Zhao","affiliation":"† School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Guiguang Ding","slug":"guiguang-ding","given":"Guiguang","family":"Ding","affiliation":"† School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Yue Gao","slug":"yue-gao","given":"Yue","family":"Gao","affiliation":"† School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Jungong Han","slug":"jungong-han","given":"Jungong","family":"Han","affiliation":"‡ School of Computing & Communications , Lancaster University , UK"}],"abstract":"Existing works on image emotion recognition mainly assigned the dominant emotion category or average dimension values to an image based on the assumption that viewers can reach a consensus on the emotion of images. However, the image emotions perceived by viewers are subjective by nature and highly related to the personal and situational factors. On the other hand, image emotions can be conveyed by different features, such as semantics and aesthetics. In this paper, we propose a novel machine learning approach that formulates the categorical image emotions as a discrete probability distribution (DPD). To associate emotions with the extracted visual features, we present a weighted multi-modal shared sparse leaning to learn the combination coefficients, with which the DPD of an unseen image can be predicted by linearly integrating the DPDs of the training images. The representation abilities of different modalities are jointly explored and the optimal weight of each modality is automatically learned. Extensive experiments on three datasets verify the superiority of the proposed method, as compared to the state-of-the-art.","doi":"10.24963/ijcai.2017/651","url":"https://ijcai.org/proceedings/2017/651","pdf":"https://ijcai.org/proceedings/2017/0651.pdf","markdown":"https://ijcai.org/proceedings/2017/651.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/651","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Uncertainty in AI","track":"Main track","number":651,"pages":{"first":4669,"last":4675},"keywords":["Uncertainty in AI: Approximate Probabilistic Inference","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Alameda-Pineda et al., 2016] Xavier Alameda-Pineda, Elisa Ric- ci, Yan Yan, and Nicu Sebe. Recognizing emotions from abstract paintings using non-linear matrix completion. In CVPR, pages 5240-5248, 2016.","title":"Recognizing emotions from abstract paintings using non-linear matrix completion","authors":"Alameda-Pineda","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Borth et al., 2013] Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. Large-scale visual sentiment ontol- ogy and detectors using adjective noun pairs. In ACM MM, pages 223-232, 2013.","title":"Large-scale visual sentiment ontology and detectors using adjective noun pairs","authors":"Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, Shih-Fu Chang","year":"2013","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carney et al., 2005] Michael Carney, Pádraig Cunningham, Jim Dowling, and Ciaran Lee. Predicting probability distributions for surf height using an ensemble of mixture density networks. In ICML, pages 113-120, 2005.","title":"Predicting probability distributions for surf height using an ensemble of mixture density networks","authors":"Michael Carney, Pádraig Cunningham, Jim Dowling, Ciaran Lee","year":"2005","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chartrand and Yin, 2008] Rick Chartrand and Wotao Yin. Itera- tively reweighted algorithms for compressive sensing. In ICAS- SP, pages 3869-3872, 2008.","title":"Iteratively reweighted algorithms for compressive sensing","authors":"Yin Chartrand, Rick Chartrand, Wotao Yin","year":"2008","venue":"ICAS-SP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2014] Tao Chen, Felix X Yu, Jiawei Chen, Yin Cui, Yan-Ying Chen, and Shih-Fu Chang. Object-based visual senti- ment concept analysis and application. In ACM MM, pages 367- 376, 2014.","title":"Object-based visual sentiment concept analysis and application","authors":"Tao Chen, Felix X Yu, Jiawei Chen, Yin Cui, Yan-Ying Chen, Shih-Fu Chang","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2016] Guiguang Ding, Yuchen Guo, Jile Zhou, and Yue Gao. Large-scale cross-modality search via collective matrix factorization hashing. IEEE Transactions on Image Processing, 25(11):5427-5440, 2016.","title":"Large-scale cross-modality search via collective matrix factorization hashing","authors":"Ding","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ekman, 1992] Paul Ekman. An argument for basic emotions. Cog- nition & Emotion, 6(3-4):169-200, 1992.","title":"An argument for basic emotions","authors":"Paul Ekman","year":"1992","venue":"Cognition & Emotion","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[James and Dasarathy, 2014] Alex Pappachen James and Belur V Dasarathy. Medical image fusion: A survey of the state of the art. Information Fusion, 19:4-19, 2014.","title":"Alex Pappachen James and Belur V Dasarathy. Medical image fusion: A survey of the state of the art","authors":"Dasarathy James","year":"2014","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadar- rama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, Trevor Darrell","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Joshi et al., 2011] Dhiraj Joshi, Ritendra Datta, Elena Fe- dorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo. Aesthetics and emotions in images. IEEE Signal Processing Magazine, 28(5):94-115, 2011.","title":"Aesthetics and emotions in images","authors":"Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, Jiebo Luo","year":"2011","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Ge- offrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2013] Haifeng Liu, Zheng Hu, Dian Zhou, and Hui Tian. Cumulative probability distribution model for evaluating user be- havior prediction algorithms. In ICSC, pages 385-390, 2013.","title":"Cumulative probability distribution model for evaluating user behavior prediction algorithms","authors":"Haifeng Liu, Zheng Hu, Dian Zhou, Hui Tian","year":"2013","venue":"ICSC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2012] Xin Lu, Poonam Suryanarayan, Reginald B Adams Jr, Jia Li, Michelle G Newman, and James Z Wang. On shape and the computability of emotions. In ACM MM, pages 229-238, 2012.","title":"On shape and the computability of emotions","authors":"Xin Lu, Poonam Suryanarayan, Reginald B Adams, Jia Li, Michelle G Newman, James Z Wang","year":"2012","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Machajdik and Hanbury, 2010] Jana Machajdik and Allan Han- bury. Affective image classification using features inspired by psychology and art theory. In ACM MM, pages 83-92, 2010.","title":"Affective image classification using features inspired by psychology and art theory","authors":"Jana Machajdik, Allan Hanbury","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikels et al., 2005] Joseph A Mikels, Barbara L Fredrickson, Gre- gory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz. Emotional category data on images from the in- ternational affective picture system. Behavior Research Methods, 37(4):626-630, 2005.","title":"Emotional category data on images from the international affective picture system","authors":"Barbara L Joseph A Mikels, Fredrickson, Casey M Gregory R Larkin, Sam J Lindberg, Patricia A Maglio, Reuter-Lorenz","year":"2005","venue":"Behavior Research Methods","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ngiam et al., 2011] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, pages 689-696, 2011.","title":"Multimodal deep learning","authors":"Ngiam","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Patterson and Hays, 2012] Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, pages 2751-2758, 2012.","title":"Sun attribute database: Discovering, annotating, and recognizing scene attributes","authors":"Hays Patterson, Genevieve Patterson, James Hays","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2015] Kuan-Chuan Peng, Amir Sadovnik, Andrew Gallagher, and Tsuhan Chen. A mixed bag of emotions: Mod- el, predict, and transfer emotion distributions. In CVPR, pages 860-868, 2015.","title":"A mixed bag of emotions: Model, predict, and transfer emotion distributions","authors":"Kuan-Chuan, Amir Peng, Andrew Sadovnik, Tsuhan Gallagher, Chen","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pipa et al., 2013] Gordon Pipa, Sonja Grün, and Carl van Vreeswi- jk. Impact of spike train autostructure on probability distribution of joint spike events. Neural Computation, 25(5):1123-1163, 2013.","title":"Impact of spike train autostructure on probability distribution of joint spike events","authors":"Gordon Pipa, Sonja Grün, Carl Van Vreeswijk","year":"2013","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schlosberg, 1954] H. Schlosberg. Three dimensions of emotion. Psychological Review, 61(2):81, 1954.","title":"Three dimensions of emotion","authors":"H Schlosberg","year":"1954","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] Jingwen Wang, Jianlong Fu, Yong Xu, and Tao Mei. Beyond object recognition: Visual sentiment analysis with deep coupled adjective and noun neural networks. In IJCAI, pages 626-630, 2016.","title":"Beyond object recognition: Visual sentiment analysis with deep coupled adjective and noun neural networks","authors":"Jingwen Wang, Jianlong Fu, Yong Xu, Tao Mei","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wright et al., 2009] John Wright, Allen Y Yang, Arvind Ganesh, Shankar S Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 31(2):210-227, 2009.","title":"Robust face recognition via sparse representation","authors":"John Wright, Allen Y Yang, Arvind Ganesh, S Shankar, Yi Sastry, Ma","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[You et al., 2016a] Quanzeng You, Liangliang Cao, Hailin Jin, and Jiebo Luo. Robust visual-textual sentiment analysis: When at- tention meets tree-structured recursive neural networks. In ACM MM, pages 1008-1017, 2016.","title":"Robust visual-textual sentiment analysis: When attention meets tree-structured recursive neural networks","authors":"You","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016b] Quanzeng You, Jiebo Luo, Hailin Jin, and Jian- chao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In AAAI, pages 308-314, 2016.","title":"Building a large scale dataset for image emotion recognition: The fine print and the benchmark","authors":"You","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yuan et al., 2012] Xiao-Tong Yuan, Xiaobai Liu, and Shuicheng Yan. Visual classification with multitask joint sparse represen- tation. IEEE Transactions on Image Processing, 21(10):4349- 4360, 2012.","title":"Visual classification with multitask joint sparse representation","authors":"Xiao-Tong Yuan, Xiaobai Liu, Shuicheng Yan","year":"2012","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2014a] Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongx- un Yao, Tat-Seng Chua, and Xiaoshuai Sun. Exploring principles-of-art features for image emotion recognition. In ACM MM, pages 47-56, 2014.","title":"Exploring principles-of-art features for image emotion recognition","authors":"Zhao","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2014b] Sicheng Zhao, Hongxun Yao, You Yang, and Yanhao Zhang. Affective image retrieval via multi-graph learn- ing. In ACM MM, pages 1025-1028, 2014.","title":"Affective image retrieval via multi-graph learning","authors":"Zhao","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2015a] Sicheng Zhao, Hongxun Yao, and Xiaolei Jiang. Predicting continuous probability distribution of image emotions in valence-arousal space. In ACM MM, pages 879-882, 2015.","title":"Predicting continuous probability distribution of image emotions in valence-arousal space","authors":"Zhao","year":"2015","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2015b] Sicheng Zhao, Hongxun Yao, Xiaolei Jiang, and Xiaoshuai Sun. Predicting discrete probability distribution of image emotions. In ICIP, pages 2459-2463, 2015.","title":"Predicting discrete probability distribution of image emotions","authors":"Zhao","year":"2015","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2016] Sicheng Zhao, Hongxun Yao, Yue Gao, Ron- grong Ji, Wenlong Xie, Xiaolei Jiang, and Tat-Seng Chua. Pre- dicting personalized emotion perceptions of social images. In ACM MM, pages 1385-1394, 2016.","title":"Predicting personalized emotion perceptions of social images","authors":"Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, Tat-Seng Chua","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2017] Sicheng Zhao, Hongxun Yao, Yue Gao, Ron- grong Ji, and Guiguang Ding. Continuous probability distribu- tion prediction of image emotions via multi-task shared sparse regression. IEEE Transactions on Multimedia, 19(3):632-645, 2017.","title":"Continuous probability distribution prediction of image emotions via multi-task shared sparse regression","authors":"Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Guiguang Ding","year":"2017","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2006] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clustering, classifica- tion, and embedding. In NIPS, pages 1601-1608, 2006.","title":"Learning with hypergraphs: Clustering, classification, and embedding","authors":"Zhou","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}