ElderMTL: Multi-Task Affect Monitoring for Elderly Care

1 Introduction

The global population is rapidly aging, increasing care demands while intensifying caregiver shortages and pressure on healthcare systems, particularly in geriatric facilities where continuous monitoring and emotional support are essential. Current monitoring approaches-such as periodic assessments or wearable sensors-are often intrusive, laborintensive, and unsuitable for long-term, continuous observation of older adults.

Research in Human-Robot Interaction (HRI) and Human-Computer Interaction (HCI) has consistently emphasized the critical role of affective perception in facilitating effective human-machine interaction. In this context, affective computing aims to give computer systems the ability to automatically recognize, understand, and respond properly to human emotional cues [G¨unes and Churamani, 2023]. In broader HRI settings, real-time facial emotion recognition systems have been used to improve interaction by helping robots recognize common facial expressions and adjust their behavior accordingly. [Epifanic, 2023]. Furthermore, research has explored on-the-fly detection of engagement in spontaneous human-robot interactions, highlighting the relevance of temporal facial and behavioral dynamics during interaction [Ben Youssef et al., 2020].

Despite these advances, most affect recognition methods are evaluated in controlled settings, trained on generalpopulation data, or lack the latency and robustness required for real-time deployment in care environments. Facial expression recognition in real time remains challenging due to uncontrolled lighting, pose variations, and demographic biases in training data [Rawal and Stock-Homburg, 2022]. Parallel efforts in HRI, including multimodal affective datasets such as AFFECT-HRI, emphasize the need to better understand affective responses to robot behavior, but have not focused on continuous, video-only affect monitoring in elderly populations [Heinisch et al., 2024]. Moreover, companion robot research underscores the importance of addressing the emotional needs of older adults, revealing the potential of socially assistive platforms to support well-being and engagement [Zeng et al., 2025].

Motivated by this gap, we present ElderMTL (Elder Multitask Learning), a real-time, video-only affect recognition system tailored for geriatric care. ElderMTL is designed to monitor emotional states, engagement, and distress indicators during live interaction using standard RGB cameras, enabling scalable and accessible affective monitoring in realworld care environments. The overview of the proposed system is presented in Figure 1.

1.1 FAU for Affective Monitoring

Facial affect recognition in elderly populations poses unique challenges not addressed by models trained on general adult datasets. Aging alters facial morphology and expression dynamics (e.g., wrinkles, reduced muscle tone, slower activations), causing appearance-based models to misinterpret agerelated features as emotional cues and introduce systematic bias [Morey, 2025; Fan et al., 2022; Park et al., 2022].

Most existing approaches map facial appearance directly to emotion labels or affect dimensions. Although effective in controlled settings, such black-box models lack robustness and interpretability in real-world geriatric care environments characterized by lighting variation, pose changes, occlusions, and subtle expressions [G¨unes and Churamani, 2023].

Facial Action Units (FAUs), defined by the Facial Action Coding System (FACS), provide a physiologically grounded representation of facial muscle activity [Ekman and Friesen, 1978]. Because FAUs encode low-level muscle activations independently of emotion categories, they offer a more robust, interpretable, and generalizable description of facial behavior, supporting recognition of emotions, engagement, and distress-related states [Kollias and Zafeiriou, 2021; Ben Youssef et al., 2020; Lucey et al., 2011; Fan et al., 2022].

Accordingly, we adopt an FAU-centered modeling framework for live human-robot and human-computer interaction, leveraging FAU dynamics to enable interpretable and reliable affect monitoring tailored to elderly users in real-world care settings.

2 Methodology

We propose ElderMTL, an age-aware multi-task framework for affect recognition in elderly care. As illustrated in Figure 2, the architecture comprises a transformer-based visual encoder, AU-specific spatial feature extraction, temporal aggregation, age-aware modulation, structured AU graph reasoning, and task-specific prediction heads. The model is trained using masked multi-task losses to jointly predict Facial Action Units (AUs), categorical facial expressions (FER), and Valence-Arousal (VA) from video sequences, while explicitly accounting for age-related facial variations.

Given an input frame xt, a shared representation is extracted using Video-MAE expanded with lightweight adapter layers to mitigate domain shifts across datasets.

To obtain AU-specific spatial features, we introduce learnable AU queries qa ∈ RD that attend to spatial tokens. For AU a at time t, cross-attention produces ht,a = Attention(qa, zt), enabling focus on discriminative facial regions. Temporal dynamics are modeled per AU by processing h1,a, . . . , hT,a with a temporal transformer, yielding an aggregated embedding Ha that captures subtle elderly-specific motion patterns. To account for baseline facial changes due to aging, age adaptation is introduced via a small learnable network fage(·), ˜Ha =Ha + fage(age).

Structured dependencies among AUs are modeled using a symmetric learnable adjacency matrix A. Age modulation is incorporated by defining the effective adjacency as Amod ij = σ(Aij) · gi(age) · gj(age), where σ is a sigmoid function and gi(·) are age-conditioned scaling functions. Graph-enhanced embeddings are then computed as Hgraph a = AmodHaW, where W is a learnable projection matrix. An L1 regularization term ∥A∥1 is applied to encourage sparsity and interpretable AU relationships.

The shared AU representations are fed into three taskspecific heads. The AU head produces per-AU logits and is trained only when AU annotations are available. The FER head predicts categorical emotions using cross-entropy loss. The VA head outputs continuous valence and arousal values and is optimized using the Concordance Correlation Coefficient loss, defined as

During training on multiple datasets with incomplete annotations, each loss is applied only when the corresponding labels are present. Masked losses and a multi-dataset curriculum are used to maintain balanced optimization. The total objective

isLtotal = λAULAU+λFERLFER +λVALCCC +λsparse∥A∥1 (2) where the coefficients λ control the contribution of each component.

3 Experimental Setup

We evaluate ElderMTL in a unified multi-task setting (FER, AU detection, and VA regression) using heterogeneous datasets with partial supervision and automatically estimated age metadata.

ElderReact [Ma et al., 2019] contains elderly facial expressions with categorical emotion and valence annotations, supporting age-relevant affect modeling.

AffWild2 [Kollias and Zafeiriou, 2020] provides largescale in-the-wild annotations for FER, VA, and multiple AUs, enabling robust multi-task training under unconstrained conditions.

DISFA [Mavadati et al., 2013] offers high-quality framelevel AU annotations in controlled settings, strengthening structural AU supervision.

Since demographic annotations are incomplete or inconsistent across datasets, we estimate age for all videos using MiVOLO [Gonc¸alves et al., 2024], a state-of-the-art visionlanguage age estimator. Age is normalized to [0, 1] and injected into (1) AU-level muscle bias modeling, (2) AU graph modulation, and (3) global affect feature conditioning.

All AU annotations are aligned into a shared 12-AU representation; missing AUs are handled via binary masking during loss computation. Datasets lacking FER or VA labels contribute only to the corresponding supervised objectives, enabling partial-label multi-task learning within a shared backbone.

Video clips are uniformly sampled to 16 frames and resized to 224 × 224. We use AdamW with weight decay and a learning rate equal to 3e-5.

To assess age robustness, we partition samples into Young (< 40), Middle (40-59), and Elderly (≥ 60) groups and report Macro-F1 for FER, mean AU-F1, CCC for valence and arousal, and absolute performance gaps between age groups.

3.1 Demonstration

In our live demonstration, ElderMTL processes video streams and visualizes predictions for Action Units, Valence-Arousal, and Emotion. To illustrate interpretability and applicability, we integrate a frozen local instruction-tuned language model (Llama 3.1 8B Instruct [Huang et al., 2024]). Importantly, it is not trained or fine-tuned on our data, and it is explicitly instructed to reason strictly from detected signals. Its purpose is to show how structured affect outputs can be translated into natural language insights for caregiver support. A recording of our demonstration is available at the following link: https://youtu.be/ZIZS9w760BA

4 Results

We evaluate ElderMTL in a unified multi-task setting (FER, AU detection, VA regression) on ElderReact, AffWild2, and DISFA. As shown in Table 1, ElderMTL achieves the strongest FER and AU detection performance across datasets, reaching 0.74 FER F1 on ElderReact and 0.55 on AffWild2, while obtaining 0.46/0.51 AU F1 (DISFA/AffWild2) and 0.217 VA CCC. These results highlight the benefit of joint multi-task learning with age-aware conditioning.

Cross-age evaluation (Table 2) shows controlled degradation with age (∆Y-M = 0.03, ∆Y-E = 0.06 for FER). Removing the age-aware graph module consistently reduces performance, particularly for elderly subjects, confirming the importance of structured, age-modulated AU modeling.

Overall, the results demonstrate that incorporating ageconditioned modulation and structured AU reasoning improves affect recognition accuracy and mitigates age-related performance bias.

5 Conclusion

We present ElderMTL, an age-aware multi-task framework for affect recognition in elderly care that jointly models Facial Action Units, Valence-Arousal, and categorical emotions with age-conditioned modulation and structured AU dependencies. The approach addresses age-specific facial dynamics and achieves competitive, age-robust performance across tasks. A live demonstration illustrates its applicability for affect monitoring in assistive elderly care systems.

Ethical Statement

ElderMTL is intended to support, not replace, human caregivers, and its predictions should be treated as decisionsupport signals warranting human verification rather than clinical assessments. Since the system continuously monitors a vulnerable population via video, deployment requires informed consent and appropriate safeguards for sensitive data.

Acknowledgments

The work was supported by a grant, provided by the Ministry of Economic Development of the Russian Federation (agreement dated June 20, 2025 No. 139-15-2025-011, identifier 000000C313925P4G0002).