Multimodal Emotion Recognition with Large Language Models

1 Background and Challenges

Emotion is an integral component of human daily experiences, significantly influencing individuals' communication, decision-making, and behavior. In real-world settings, emotions are expressed and perceived through multiple modalities, including language, speech, facial expressions, and gestures [Zhao et al., 2023; Wang et al., 2026b]. As a result, Multimodal Emotion Recognition (MER) has emerged as a critical research area, focusing on enabling models to comprehend emotions from modality-compound inputs as humans do. To achieve this, traditional research primarily relies on small-scale, task-specific models [Yang et al., 2025b; Li et al., 2025c]. Despite achieving satisfactory performance, they suffer from strong dependencies on predefined input domains and output spaces, which make them less effective in practical applications that require dynamism and flexibility.

∗Corresponding author.

The emerging Large Language Models (LLMs) present a promising avenue for addressing these limitations [Shou et al., 2025]. Large-scale pre-training equips LLMs with strong instruction-following capabilities, which are inherited by Multimodal LLMs (MLLMs), giving rise to a new MERwith-LLMs paradigm. By formulating MER as an LLMcentric autoregressive process, this paradigm enables unified handling of inputs spanning multiple domains and modalities, as well as producing diverse instruction-conditioned outputs. It even deepens the MER task itself, extending classification toward explanation [Lian et al., 2024a]. Collectively, these advantages have driven a paradigm shift, reflected by a rapidly growing body of recent work on MER-with-LLMs.

Alongside this trend, new challenges have also arrived. A line of studies [Lian et al., 2024b; Wu et al., 2025a] has reached a consensus that, under zero-shot inference, MLLMs often fail to achieve proficiency on MER comparable to that observed in other multimodal tasks. This deficiency is inherently rooted in the complexity of MER, which entails capturing high-level affective cues from multimodal inputs, modeling interactions among heterogeneous modalities, and integrating them to derive emotional conclusions. Following a progressive order, we categorize the currently most pressing challenges into three branches, as illustrated in Fig. 1(a):

Affective Data Scarcity. Data constitute the foundation of modern models. The underperformance of general MLLMs on MER underscores the importance of emotional data, making targeted optimization on such data a direct and intuitive remedy. However, the subjectivity of emotion perception often necessitates collaboration from multiple annotators for reliable annotation, substantially increasing labeling costs and constraining the scale of high-quality datasets. Moreover, datasets collected in different contexts are typically annotated under distinct and often incompatible emotion theories, further aggravating dataset fragmentation. These limitations compel existing approaches to either maximize the utilization of available data or construct new datasets.

Multimodal Affective Gap. The "affective gap" generally refers to the intra-modality misalignment between emotional and factual features [Zhao et al., 2021b], which challenges MLLMs in capturing meaningful affective cues. We extend this concept to cover an inter-modality setting, where it characterizes the heterogeneity and semantic discrepancy of emotional features across different modalities. Heterogeneity reflects inconsistencies in information density and expression patterns; for instance, text tends to be more compact and abstract, whereas images are more dispersed and concrete. Semantic discrepancy refers to the fact that each modality provides unique and sometimes conflicting cues. Together, these factors complicate inter-modality emotional synergy modeling. Overall, these gaps motivate the exploration of dedicated affective cues extraction and interaction modeling.

Affective Interpretation Opacity. Inherited from the habits of small-scale models, mainstream MER methods primarily focus on emotion recognition rather than explanation. This emphasis results in a lack of decision transparency, which not only hinders iterative model improvement but also raises concerns about reliability in high-stakes and cross-domain applications, including healthcare, education, and finance. Benefiting from the autoregressive generation process of LLMs, the MER-with-LLMs paradigm naturally supports step-bystep reasoning or explicit natural language explanations for emotional decisions, thereby giving rise to a new task: explainable MER. In this context, existing approaches aim to enhance model transparency and reliability, for example, by reducing hallucinations, eliciting confidence verbalization, and enabling open-vocabulary MER.

To address these challenges, various optimization strategies with divergent emphases have been proposed, significantly advancing this paradigm, as illustrated in Fig. 1(b). However, their optimization pathways remain intricate and diversified, and a systematic organization is still lacking. To promote a more structured development of this emerging paradigm, this paper provides a comprehensive review of the existing challenges, a taxonomy of current methods, and an in-depth discussion of recent progress and future directions.

2 ProblemFormulation and Taxonomy

Compared with traditional MER [Zhao et al., 2021a], we extend the MER-with-LLMs paradigm to a broader setting. Under this paradigm, LLMs serve as a text-centered orchestrator that processes affective information from multimodal inputs containing at least one non-textual modality.

Based on this definition, we first provide a symbolic formulation before taxonomizing existing methods. In general, a multimodal sample X can consist of various modalities, such as I (image), V (video), A (audio), T (text), or even physiological signals like electroencephalogram (EEG). Although the latter holds critical application value [Cai et al., 2022], this paper primarily focuses on I, V , A, and T modalities, considering their widespread availability. Together with a text instruction Q that specifies the task requirements, the sample is fed into an MLLM πθ. To process these inputs, the MLLM typically employs modality-specific encoders and adapters to transform I, V , and A into multimodal embeddings, which are then combined with the tokenized T and Q to generate a textual response R = [r1, r2, · · · , rn]. Each token is generated by maximizing the conditional likelihood given the multimodal inputs and the preceding responses:

(1)

Here, V denotes the vocabulary of the MLLM. Depending on downstream requirements, the desired responses can take the form of emotion categories or tailored explanations. Separated by input domains and modalities, we divide MER into five mainstream sub-tasks, with samples visualized in Fig. 3:

• Generic Visual Emotion Comprehension (GVEC) aims to predict viewer-side emotional responses elicited by visual stimuli. X = {I}/{V }/{V,A} consists of images or videos from arbitrary domains, such as artworks, landscapes, movies, or human-shared content.
• Visual-Textual Sentiment Analysis (VTSA) aims to identify the emotion embedded in image-text pairs. X = {I, T} is typically image-text posts collected from social media platforms such as Twitter and Tumblr.

Figure 4: Illustration of Affective Data Augmentation methods.

• Speech Emotion Comprehension (SEC) aims to describe emotions conveyed by speeches, commonly through classification or captioning. X = {A} is a speech segment extracted from TV shows or purposely recorded.
• Facial Expression Recognition (FER) aims to infer human emotional states from facial expressions. X = {I}/{V } focuses on facial cues, typically in the form of a face-centered static image or dynamic video.
• Conversation-based Multimodal Emotion Recognition (CMER) extends FER to a conversational setting, aiming to analyze a speaker's emotional state from audiovisual dialogues. X = {V,A, T}/{V, T} is typically sourced from movies or TV shows.

On top of the above formulation, we present the taxonomy of MER-with-LLMs in Fig. 2. We categorize existing methods into three branches according to the challenges they focus on and provide a high-level overview below:

Affective Data Augmentation. Existing methods for handling Affective Data Scarcity are divided into two branches. Training-free Sample Configuration aims to maximize the utilization of available data, typically by augmenting the input context while freezing MLLM parameters. Emotion Data Annotation focuses on augmenting the volume of data, either by engineering training data for fine-tuning or by constructing benchmarks that enable more emotion-aware evaluation.

Multimodal Affective Representation. To handle Multimodal Affective Gap, a line of research focuses on feeding multimodal embeddings with richer affective representations into LLMs. According to their primary emphasis, we further divide these methods into two branches: Perceptual Emotion Mapping, which refines the perception of intra-modality affective cues by optimizing the encoding process, and Multimodal Emotion Coordination, which enhances the modeling of inter-modality coordination through comprehensive fusion of affective cues from multiple sources.

Multimodal Affective Reasoning. To address Affective Interpretation Opacity, some studies encourage MLLMs to generate step-by-step reasoning before producing final emotion predictions, thereby enhancing explainability. Pioneering works primarily focus on the accuracy and reliability of such reasoning, which we categorize as Emotion Explanation and Hallucination. More recent studies further explore accommodating the inherent subjectivity of emotion perception, which we classify as Subjective Emotion Reasoning.

Below, we elaborate on each branch. Methods that fall into overlapping regions are classified by their primary focus.

3 Affective Data Augmentation

Training-free Sample Configuration. To handle Affective Data Scarcity, this line of work emphasizes maximizing the utility of existing samples. This is achieved by carefully configuring the samples provided as input to MLLMs, a process that introduces human-prior-inspired biases or demonstrations. Such strategies are cost-efficient and are particularly useful when additional data is unavailable.

Under zero-shot settings, the biases are injected through extracting key image features, allowing MLLMs to focus on regions that convey the most salient emotional cues. For example, initial explorations [Etesam et al., 2024] utilize CLIP [Radford et al., 2021] to extract emotional information from bounding boxes around individuals in images, generating captions that reflect emotional content. While bounding boxes alone provide useful spatial localization, they often lack the granularity needed for nuanced MER. To overcome this limitation, several approaches [Zhang et al., 2024; Wang et al., 2025c] incorporate multi-level visual cues, including spatial localization, facial geometry, and action units, enabling a more comprehensive analysis of human emotional information. Inspired by the success of In-Context Learning (ICL), Wu et al. [2025a] unleash the emotion perception of MLLMs in a few-shot setting. By optimizing the retrieval, presentation, and distribution of demonstrations, this method allows MLLMs to generalize effectively from minimal labeled data, further improving emotion inference in multimodal contexts. We illustrate them in Fig. 4(a).

Emotion Data Annotation. When additional data is permitted, annotating the desired emotion-oriented datasets serves as a more straightforward approach. In this line of work, we separately introduce the training datasets and evaluation benchmarks, both of which are summarized in Tab. 1.

Dataset Engineering. This branch enhances the intrinsic capabilities of MLLMs by constructing training datasets, as illustrated in Fig. 4(b). For the GVEC task, EmoVIT [Xie et al., 2024] leverages advanced proprietary models to annotate the first emotion-centric visual instruction tuning dataset, and VEC-CoT [Wu et al., 2026a] subsequently contributes a larger-scale structured affective reasoning dataset. For the FER task, FABA-Instruct [Li et al., 2024] considers both action units and emotions, introducing the first dataset of its kind. AMT [Li et al., 2025a] further extends to a multi-task scenario. The CMER task has received the most attention, likely due to its highly complex input modalities, with different methods exploring complementary scenarios. Specifically, DEEMO [Li et al., 2025b] is designed to preserve privacy; EmoCause [Huang et al., 2025] and MEC4 [Wu et al., 2025b] focus on understanding emotion cause under multi-attribute and multilingual settings; MESC [Chu et al., 2025] targets therapeutic and counseling scenarios; and MERR [Cheng et al., 2024] and MER-Caption [Lian et al., 2025a] address regular CMER tasks, characterized by their high quality and large scale. These datasets provide essential prerequisites for subsequent emotion-oriented optimization. Benchmark Construction. Beyond training, evaluation also plays an indispensable role in the development of MER-withLLMs, whose evolution exhibits unexpected consistency across MER sub-tasks. Pioneering efforts focus on systematically integrating existing datasets, such as VECBench [Wu et al., 2026a], MM-BigBench [Yang et al., 2023], and MER-UniBench [Lian et al., 2025a]. Subsequent works aim to shed light on deeper insights. For example, EEmoBench [Gao et al., 2025] unifies evaluation of perception, description, ranking, and assessment; FaceBench [Wang et al., 2025b] encompasses perception for fine-grained facial details; EIBench [Lin et al., 2025] delves into both explicit and implicit factors that drive emotional responses; Hi-EF [Wang et al., 2026a] proposes forecasting future emotional trajectories; and MTMEUR [Hu et al., 2025a] probes MLLMs' understanding of multi-turn emotional dialogues.

More recent efforts push further toward deeper emotional dynamics. OV-MERD [Lian et al., 2025b] annotates samples with multi-label, open-vocabulary emotion descriptions, and MVEI [Wu et al., 2026b] introduces the emotion statement judgment task, both of which incorporate the previously overlooked issue of subjectivity in evaluation. CA-MER [Han et al., 2025] examines how models handle emotional conflicts across modalities, which is a practical yet underexplored problem. EmotionHallucer [Xing et al., 2026] conducts an in-depth investigation of hallucinations in emotional reasoning, advancing reliability. Collectively, these datasets and benchmarks act as guiding beacons for the MER-with-LLMs paradigm, providing anchors and directions.

4 Multimodal Affective Representation

As illustrated in Fig. 5, to tackle Multimodal Affective Gap, current methods fall into two categories: one bridges intramodality gaps by refining the encoding process, while the other handles inter-modality gaps by modeling interactions. Perceptual Emotion Mapping. Multimodal encoders are typically pre-trained on general-purpose data, which makes them less effective when directly applied to capturing affective cues. Existing solutions are largely shared across different subtasks. An intuitive one is curating data and arranging training schedules, thereby adjusting the parameters of encoders or adapters to a more emotion-aligned direction. For instance, EmoDETective [Huang et al., 2025] in- Encoder

(b). Multimodal Emotion Coordination more emphasizes the inter-modality interaction process.

Figure 5: Differences in methodological focuses of the two types of Multimodal Affective Representation methods.

troduces a progressive training strategy that transitions from factual detection to emotional conclusion; ExpLLM [Lan et al., 2025] constructs an Exp-CoT engine to synthesize data and improves FER through multi-turn conversations; BLSPEmo [Wang et al., 2024] first teaches models to understand emotional transcripts before incorporating audio signals; and EmoVerse [Li et al., 2025a] jointly optimizes multiple tasks and ensures smooth training by controlling the task ratios.

In parallel, directly modifying the encoded embeddings through sharpening or augmentation serves as another established solution. Sharpening promotes the capture of affective cues by filtering out redundant features before the LLM. For instance, SEPM [Fang et al., 2025] applies coarse-to-fine inference to remove emotion-irrelevant visual tokens; StimuVAR [Guo et al., 2025] proposes event-driven frame sampling and emotion-triggered tube selection strategies; and SECap [Xu et al., 2024] employs a learnable Q-former to compress audio embeddings. By contrast, augmentation complements affective cues by introducing additional features: MERMAID [Yang et al., 2025c] treats MLLMs as agents and augments the original image through self-reflection; EmoLA [Li et al., 2024], Emotion-LLaMA [Cheng et al., 2024], FaceInsight [Li et al., 2025d], and EmoChat [Xu et al., 2025] implement augmentation more straightforwardly by incorporating expert encoders, with the latter two further introducing consistency constraints to facilitate feature alignment. These methods are collectively organized in Tab. 2.

Multimodal Emotion Coordination. To model the synergy of inter-modality affective cues, current works rely on modality fusion to enable thorough interactions before feeding the representations into the LLM. We summarize these methods in Tab. 3. Overall, they primarily adopt concatenation, Qformer, and attention-based mechanisms, often accompanied by the encoder augmentation strategies discussed earlier.

Specifically, Omni-Emotion [Yang et al., 2025a] finds that video-level concatenation outperforms frame-level interactions. FEALLM [Hu et al., 2025b] and AffectGPT [Lian et al., 2025a] employ vanilla attention mechanisms, where the former focuses on fusion across different hierarchical levels and scales, while the latter emphasizes holistic interaction. EmoVIT [Xie et al., 2024] and M3F [Wu et al., 2025b] favor Q-former-based designs, leveraging learnable queries to extract instruction-aligned cues or retrieve memory fragments. EMO-LLaMA [Xing et al., 2024] combines attention with Qformer mechanisms to inject rich facial knowledge into image embeddings. Notably, Omni-SILA [Luo et al., 2025] adopts an alternative fusion paradigm based on Mixture-of-Experts for effective multi-source integration. In general, these diversified successes highlight the inherent complexity and future potential of bridging inter-modality affective gaps.

5 Multimodal Affective Reasoning

As illustrated in Fig. 6, directly predicting emotions may lead to user confusion regarding model decisions, which has motivated the emergence and development of the explainable MER task. This line of research brings diverse benefits: it not only enhances user trust but also helps uncover potential model deficiencies and optimization pathways.

Emotion Explanation and Hallucination. Methods in this branch primarily focus on the usability and reliability of explanations. To address potential inconsistencies between reasoning and predictions, Facial-R1 [Wu et al., 2026c] proposes a data-centric solution, which is also supported by an RL reward that encourages factual observations. ERV [Rha et al., 2026] trains a lightweight auxiliary model specifically to verify whether the answers are faithful to the reasoning. To mitigate hallucinations, PEP-MEK [Xing et al., 2026] introduces a predict-explain-predict inference pipeline, inserting an additional stage to extract modality-specific and emotionrelated knowledge. AlignCap [Liang et al., 2024] incorporates guidance from teacher models and proprietary LLMs to enforce consistency and rationality. EmotionReasoner [Zhao et al., 2026] enhances emotional interpretability by introducing an emotion-polarity token weighting mechanism that guides the reasoning process to focus on emotionally salient tokens.

Some other critical directions have also been explored. MulCoT-RD [Shangguan et al., 2025] improves computational efficiency through distilling knowledge from heavy MLLMs to lightweight ones. FIRES [Wang et al., 2025a] equips the model with a flexible thinking pattern that dynamically incorporates "visual scene", "emotion", "situation", and "response strategy" on demand. MultiMood [Le et al., 2026] reinforces user trust in emotional support dialogue by formulating psychology-based criteria for guidance.

Subjective Emotion Reasoning. The inherent subjectivity of emotion perception allows different observers to produce different responses to the same content, posing a long-standing challenge to the MER field. Leveraging the MER-with-LLMs paradigm, several meaningful attempts have been made.

Some methods reformulate pre-defined emotion sets into open-vocabulary ones and allow multiple labels per sample, enabling models to recognize as many plausible emotions as possible. Along this direction, Agent-MER [Lai et al., 2025] emulates human cognition by converting singlestep emotion prediction into a structured, knowledge-driven process, achieving a balance between coverage and accuracy through self-consistent voting. AffectGPT-R1 [Lian et al., 2025c] focuses on targeted reward design during RL, promoting more effective optimization. Beyond reformulating the task itself, another line of work augments existing ones with a confidence dimension, allowing models to self-assess the acceptability of their predictions under potential subjective scenarios. EmoCaliber [Wu et al., 2026a] progressively equips models with the ability to express and calibrate confidence, establishing a solid baseline for this direction. The methods discussed above are summarized in Tab. 4.

6 Model Performance Quantification

Following a qualitative review of methods within the MERwith-LLMs paradigm, this section presents a quantitative perspective to derive further findings and insights. As shown in Tab. 5, we observe a consistent correlation between performance and techniques across sub-tasks, revealing an evolutionary trend from freezing to fine-tuning model parameters and from SFT to RL. However, RL also introduces an observable performance fluctuation in CMER, highlighting potential room for optimization. On the other hand, methods under the Emotion Explanation and Hallucination branch tend to achieve relatively strong performance, underscoring the value of continued research into affective reasoning.

Yet these quantitative gains may obscure several underlying concerns: many existing methods still suffer from weak zero-shot generalization, limited long-tail recognition, fragile cross-cultural transfer, sentiment hallucination, and ethical risks. Although less salient than the three major challenges discussed earlier, these issues are no less important and warrant dedicated attention in future work.

7 Conclusion and Future Directions

This paper presents a comprehensive review of the emerging MER-with-LLMs paradigm. We systematically identify the key challenges and categorize existing approaches into three corresponding branches. For each branch, we discuss its motivation and representative methodologies, and compare selected works in terms of their focus, techniques, and achieved outcomes. By summarizing current progress, we aim to facilitate the future development of this paradigm. We conclude the paper by outlining the following open research directions:

Unified and Generalized MER. Existing MER research is scattered across multiple sub-tasks, all of which play indispensable roles in general emotional intelligence. Therefore, research that unifies these sub-tasks is warranted, as leveraging their complementary strengths may lead to unforeseen synergies and fuel substantial vitality.

Mechanism-level Exploration for MER. Understanding why often matters more than knowing what. The methodological progress of MER has left substantial gaps in mechanistic understanding, such as which model parameters critically influence emotion perception, and the underlying reasons why affective reasoning facilitates recognition.

Subjectivity-embraced Framework. Perception subjectivity is non-negligible in the future applications of MER systems. While greatly inspiring, current frameworks still leave substantial room for refinement. For example, it remains underexplored how to establish theoretically grounded openvocabulary emotions, as well as how to evaluate open-ended interpretations. Alternative frameworks also merit investigation, such as determining how subjectivity should be incorporated into models, and how to design subjectivity-enriched datasets and corresponding training strategies.

Agentic Emotion Understanding. Direct interaction with the real world is an inevitable path for future MER development. To bridge perception with actionable emotional intelligence, it is crucial to establish training and evaluation frameworks that support observation, planning, reasoning, tool invocation, and real-time feedback.

Safety, Bias, and Cultural Adaptation in MER. Ensuring safety in MER requires addressing risks related to cultural variation, social bias, and individual differences. Current research largely remains at the stage of dataset construction and empirical analysis of general-purpose MLLMs, with limited exploration of concrete solutions. Advancing this area demands principled model designs and training strategies that enable culturally adaptive, bias-aware, and personalized emotion understanding for robust and responsible deployment.

Acknowledgments

This work is supported by Beijing Natural Science Foundation (No. L252009), the National Natural Science Foundation of China (Nos. 62571294, 62441614), the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (No. GML-KF-26-09), and the Open Project of Xinjiang Key Laboratory of Multimodal Intelligent Computing and Large Models, Kashi University.

Contribution Statement

Hongrui Zhang and Daiqing Wu contribute equally to this work.