CausVSR: Causality Inspired Visual Sentiment Recognition

CausVSR: Causality Inspired Visual Sentiment Recognition

Xinyue Zhang, Zhaoxia Wang, Hailing Wang, Jing Xiang, Chunwei Wu, Guitao Cao

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence
Main Track. Pages 3196-3204. https://doi.org/10.24963/ijcai.2024/354

Visual Sentiment Recognition (VSR) is an evolving field that aims to detect emotional tendencies within visual content. Despite its growing significance, detecting emotions depicted in visual content, such as images, faces challenges, notably the emergence of misleading or spurious correlations of the contextual information. In response to these challenges, we propose a causality inspired VSR approach, called CausVSR. CausVSR is rooted in the fundamental principles of Emotional Causality theory, mimicking the human process from receiving emotional stimuli to deriving emotional states. CausVSR takes a deliberate stride toward conquering the VSR challenges. It harnesses the power of a structural causal model, intricately designed to encapsulate the dynamic causal interplay between visual content and their corresponding pseudo sentiment regions. This strategic approach allows for a deep exploration of contextual information, elevating the accuracy of emotional inference. Additionally, CausVSR utilizes a global category elicitation module, strategically employed to execute front-door adjustment techniques, effectively detecting and handling spurious correlations. Experiments, conducted on four widely-used datasets, demonstrate CausVSR's superiority in enhancing emotion perception within VSR, surpassing existing methods.
Keywords:
Humans and AI: HAI: Cognitive modeling
Computer Vision: CV: Machine learning for vision
Computer Vision: CV: Recognition (object detection, categorization)
Machine Learning: ML: Deep learning architectures