MemoVAD: Resource-Efficient Video Anomaly Detection via Dynamic Semantic Memory in Edge Computing Scenarios

1 Introduction

The widespread deployment of surveillance cameras in public spaces has resulted in an explosive growth of video data, creating an urgent demand for automated Video Anomaly Detection (VAD) [Shathik, 2025]. The primary objective of VAD is to identify abnormal events, such as accidents or criminal activities, within long sequences of normal activities. While recent advancements in deep learning have significantly improved detection accuracy, deploying the models in real-world scenarios remains a significant challenge. The difficulty arises primarily from the inherent conflict between the need for sophisticated semantic reasoning and the constrained computational resources available on edge devices [Ghasemi et al., 2024; Guo et al., 2025].

∗ Corresponding Author

Traditional VAD approaches typically rely on reconstruction or prediction paradigms that operate under unsupervised or weakly supervised settings [Wu et al., 2024b; Li et al., 2025a]. The methods generally learn the distribution of normal patterns and treat deviations as anomalies. Although computationally efficient enough for some edge applications, they often struggle to distinguish between genuine anomalies and benign distributional shifts, such as dynamic background changes or camera jitter [Sultani et al., 2018; Tian et al., 2021]. Furthermore, the methods lack highlevel semantic understanding, which restricts their ability to interpret complex scenes. Conversely, the emergence of Vision-Language Models (VLMs) has introduced a new paradigm where visual data is aligned with rich textual semantics through CLIP-based encoding [Deng et al., 2025; Li et al., 2025a]. VLMs demonstrate remarkable capability in zero-shot anomaly detection and semantic reasoning [Zanella et al., 2024; Ye et al., 2025; Shao et al., 2025]. However, their massive parameter counts and heavy computational requirements render them unsuitable for direct deployment on resource-constrained edge hardware.

To leverage the semantic power of VLMs without overwhelming edge resources, edge-cloud collaboration offers a promising direction [Zhang et al., 2025]. A naive solution is to transmit all video data to a cloud-based VLM for processing [Kumar et al., 2024; Wang et al., 2025; Sharshar et al., 2025]. Nevertheless, it incurs huge bandwidth costs and high latency, which are unacceptable for real-time surveillance applications [Khan et al., 2022; Mondal et al., 2024; Li et al., 2025b]. Consequently, a critical research gap exists in developing a framework that can maintain the low latency of edge processing while selectively integrating the high-level semantic capabilities of cloud-resident large models.

To bridge the gap, we propose MemoVAD, a resourceefficient framework for video anomaly detection via dynamic semantic memory in edge computing scenarios. As shown in Fig. 1, anomalies typically exhibit recurring semantic patterns and manifest as temporally continuous segments. Therefore, continuous reliance on a computationally expensive VLM is unnecessary. Instead, we introduce a novel collaborative mechanism where a lightweight edge detector handles the majority of the inference workload and queries the remote VLM only when necessary. Specifically, MemoVAD introduces three components to balance semantic capability and edge efficiency. First, it adopts a lightweight detector that uses a frozen VideoMAE [Tong et al., 2022] backbone for motion feature extraction, together with a causal Temporal Context Encoder (TCE) to model local temporal dependencies. Second, we propose an Uncertainty-Aware Gating (UAG) mechanism grounded in Subjective Logic [Sensoy et al., 2018], which estimates perceived uncertainty and triggers a VLM query when the edge model lacks sufficient evidence for a reliable decision. Finally, we develop a Dynamic Semantic Memory (DSM) that caches VLM-verified semantic prototypes on the edge. For semantically similar events, the model retrieves relevant knowledge from the memory instead of contacting the remote server, enabling efficient online knowledge distillation and progressively transferring VLM-level semantics to the edge model.

In summary, our main contributions are as follows:

• We propose MemoVAD, a resource-efficient collaborative framework that harmonizes the speed of edge computing with the semantic reasoning of large VLMs.
• We introduce an uncertainty-aware subjective-logic gate to query VLMs only under high perceived uncertainty, and a dynamic semantic memory caching VLM insights to reduce communication overhead while preserving accuracy.
• Our MemoVAD demonstrates remarkable improvements compared to various benchmarks across two public datasets, validating its superior performance.

2 RelatedWorks

2.1 Weakly Supervised Video Anomaly Detection with VLMs

Due to the huge cost of frame-level annotations, current research predominantly focuses on Weakly Supervised Video Anomaly Detection (WSVAD), where only video-level labels are available [Mishra et al., 2024; Abdalla et al., 2025;

Liu et al., 2024; Wu et al., 2024a; Karim et al., 2024; Zhang et al., 2023; Pu et al., 2024]. Multiple Instance Learning (MIL) serves as the dominant framework in the domain, treating videos as bags of clips to distinguish anomalous bags from normal ones [Zhang et al., 2022; Tang et al., 2023; Fang et al., 2024]. However, standard MIL-based detectors typically rely on discriminative feature embeddings that lack explicit semantic interpretability. To mitigate the issue, recent studies have integrated Vision-Language Models (VLMs), leveraging the zero-shot capabilities of models like CLIP [Radford et al., 2021] and GPT-4V to identify anomalies via textual prompting [Wu et al., 2024b; Ye et al., 2025; Yang et al., 2024; Zanella et al., 2024; Shao et al., 2025]. While VLMs demonstrate superior performance in recognizing semantically complex events, their substantial computational overhead poses a significant barrier to real-time deployment. Our work builds upon the efficient MIL formulation but enhances it by integrating explicit semantic knowledge. Specifically, to mitigate the computational cost of VLMs, we adopt an online knowledge distillation strategy that continuously updates the edge model via a dynamic memory mechanism.

2.2 Efficient Edge-Cloud Collaboration

Deploying deep learning models on edge devices requires careful optimization to balance accuracy and efficiency [Guo et al., 2025; Ghasemi et al., 2024]. Common techniques include model compression methods such as quantization, pruning, and lightweight architecture design. While the techniques reduce inference latency, they often degrade model capacity and performance. Edge-cloud collaborative intelligence seeks to mitigate the issue by offloading heavy computation to the cloud [Zhang et al., 2025; Shathik, 2025]. Traditional offloading strategies decide which parts of a model to execute locally and which to transmit based on bandwidth and battery constraints. However, most existing collaborative frameworks focus on partition points within a fixed network architecture rather than dynamic interaction based on sample difficulty. Our approach differs by employing an uncertaintydriven policy that dynamically determines the necessity of cloud interaction. By utilizing Subjective Logic to quantify evidence sufficiency, MemoVAD ensures that communication resources are expended only on hard and novel samples, thereby achieving a superior balance between resource efficiency and detection performance.

3 Methodology

3.1 ProblemDefinition

We formulate WSVAD as a Multiple Instance Learning (MIL) task. Let X = {(Vi, Yi)}|X| i=1 denote the training set, where Vi is an untrimmed video and Yi ∈ {0, 1} is the videolevel label. Yi = 0 indicates a normal video, while Yi = 1 implies the video contains at least one anomalous event. Each video Vi is divided into a sequence of T non-overlapping clips {c1, c2, . . . , cT }.

The objective is to learn a clip-level anomaly scoring function fθ(ct) → yt ∈ [0, 1], such that yt is high for anomalous clips and low for normal ones. Under the MIL assumption, the relationship between clip scores and the video label is defined as:

Specifically, for a normal bag Yi = 0, all clips are normal yt ≈ 0; for an anomalous bag Yi = 1, at least one clip is anomalous yt ≈ 1.

Unlike traditional centralized VAD, we consider an edgecloud collaborative scenario. The local edge device has limited computational resources and must minimize communication with the remote VLM. The goal is to maximize detection performance while adhering to a strict communication budget:

where I(·) is the indicator function for triggering a remote VLM query, and δ is the desired performance threshold.

3.2 Overview

We propose MemoVAD, a resource-efficient framework designed to bridge the gap between lightweight edge detection and semantic reasoning. As illustrated in Fig. 2, the framework operates under an edge-cloud collaborative paradigm to identify anomalies from the input video clips defined in Sec. 3.1. MemoVAD comprises three key components:

1. Edge-Resident Detector: A lightweight network utilizing a frozen VideoMAE backbone for initial feature extraction and temporal modeling.
2. Uncertainty-Aware Gating (UAG): A policy module that determines whether to query the VLM based on perceived uncertainty.
3. Dynamic Semantic Memory (DSM): A continuously updating memory bank that stores VLM-verified prototypes, enabling the edge model to retrieve high-level semantic knowledge without recurring communication costs.

3.3 Edge-Resident Network

Feature Extraction

To guarantee real-time inference on resource-constrained edge devices, we adopt VideoMAE-Small [Tong et al., 2022]

as our visual backbone. We freeze the pre-trained weights to preserve its generalizable motion features and prevent catastrophic forgetting during domain adaptation. Given an input video clip ct ∈ RC×F×H×W , the backbone extracts a compact feature vector xt:

where Dstu denotes the feature embedding dimension. For the input preprocessing, we resize frames to a standard spatial resolution of H ×W and sample F frames with a fixed temporal stride τ.

Causal Temporal Context Encoder

Since the backbone processes clips independently, the extracted feature xt lacks temporal context required for detecting complex anomalies. To bridge the gap, we introduce a lightweight Temporal Context Encoder (TCE) implemented as a 2-layer causal Transformer Encoder, which aggregates information from a fixed-length history window of L clips. To retain sequential order information, learnable positional embeddings are added to the input sequence before encoding:

where P denotes learnable positional embeddings. The output hpre t ∈ RDstu is the pre-enrichment contextual student feature for clip t. In deployment, we implement the causal Transformer with KV-cache for streaming inference, enabling amortized constant-time updates per incoming clip.

3.4 Dynamic Semantic Memory (DSM)

DSM is designed to realize online knowledge distillation by caching VLM reasoning results and enabling semantic retrieval on the edge.

Memory Structure

We structure the memory as a set of key-value pairs M = {(ki, vi)}Ni=1 with memory size N.

• Key ki ∈ RDstu stores the pre-enrichment student feature hpre for a VLM-confirmed event. It ensures that keys share the same distribution as incoming queries.
• Value vi ∈ RDvlm stores the corresponding semantic embedding produced by the teacher VLM, serving as highlevel knowledge to be distilled.

Memory Retrieval

For each incoming clip feature hpre t , we retrieve relevant semantic information fromM using multi-head cross-attention. The query is derived from the student feature, while the memory keys and values serve as K and V :

The retrieved semantic information mt is computed via the attention mechanism:

Simultaneously, to quantify retrieval confidence for gating, we compute the maximum cosine similarity between hpre t and stored keys:

The metric sretrieval measures how well the current input matches the known anomalous prototypes in the memory.

Semantic Adapter Module (SAM)

Directly fusing semantic prototypes into the student space may cause distribution mismatch. We introduce a lightweight Semantic Adapter Module (SAM) to align and fuse mt into the student feature space.

We first project the retrieved semantic prototype to the student dimension via a lightweight projection ϕ : RDvlm → RDstu (e.g., a single linear layer). Then we concatenate and pass through a bottleneck adapter:

where MLPdown compresses by ratio r = 4, σ is GELU, and MLPup restores to Dstu. The enriched student feature is obtained by a gated residual connection:

where α is a learnable scalar initialized to 0 to prevent negative transfer in early training.

3.5 Uncertainty-Aware Gating (UAG)

Continuous VLM querying is extremely expensive. UAG requests VLM assistance only for hard or novel samples, using perceived uncertainty and memory confidence.

Gating Policy

Softmax confidence cannot reliably separate aleatoric and perceived uncertainty. We adopt Subjective Logic [Sensoy et al., 2018] to estimate perceived uncertainty via evidential learning.

Given the pre-enrichment feature hpre t , the gating head predicts non-negative evidence:

for the Normal and Anomalous classes. Evidence defines a Dirichlet distribution Dir(αt):

where St represents evidence strength. The perceived uncertainty is defined as inverse total evidence:

To improve robustness under domain shift where models can be over-confident yet wrong, we trigger VLM queries when the case is uncertain and novel. The triggering condition is:

where τunc, τsim are hyper-parameters trading off accuracy and communication cost.

VLM Inference and Memory Update

When triggered, the raw clip ct is transmitted to the remote VLM. We uniformly sample F frames from ct and provide a lightweight prompt for anomaly verification. The VLM outputs: (1) Anomaly Verification, deciding whether the clip truly contains an anomaly; and (2) Semantic Extraction, producing a text-aligned embedding zvlm ∈ RDvlm .

If the VLM confirms an anomaly, we write the new knowledge into memory:

Online Semantic-Diversity Replacement. To keep a fixed memory budget Nmax while avoiding quadratic recomputation, we maintain a redundancy score for each key:

When inserting a new key knew, we compute cos(knew, ki) for all stored keys and update affected ρi incrementally. If the memory is full, we remove the most redundant key argmaxi ρi, yielding an online replacement with amortized O(NDstu) per insertion.

3.6 Objective Function

The model is trained with weak video-level labels. Let S(·) denote the anomaly classifier head followed by a Sigmoid. The predicted anomaly score for the t-th clip is:

MIL Ranking Loss

We formulate weakly supervised VAD as a Multiple-Instance Learning task. Normal videos are negative bags Bn and anomalous videos are positive bags Ba. Using the standard MIL assumption, the video-level score is represented by the maximum clip score. The ranking loss is:

where m ∈ (0, 1) is the margin hyperparameter.

Semantic Distillation Loss

To distill semantic knowledge from the VLM, we minimize the distance between student and teacher representations. We introduce a projection head ψ : RDstu → RDvlm (two-layer MLP) and apply supervision only when teacher signals are available. The distillation loss is:

where ϵ is a small constant for numerical stability. Here, Mt = 1 if the VLM is triggered for clip t, and 0 otherwise. The stop-gradient operator treats the VLM as a fixed semantic anchor.

Temporal Smoothness Loss

We impose temporal continuity and sparsity priors:

where the first term enforces smoothness, and the second encourages sparsity of anomaly predictions. λsp is a sparsity hyper-parameter that balances the two constraints. It prevents the trivial solution where the model predicts high scores for all clips.

Gating Head Supervision

We supervise the evidential gating head following Evidential Deep Learning [Sensoy et al., 2018]. Let Ybag ∈ {0, 1} be the video-level label. To reduce label noise under MIL, we apply the loss only to the top-scoring clip(s) in each bag:

Total Loss

The final objective function is a weighted sum of the components:

where λ1, λ2, and λ3 are pre-defined hyperparameters that balance the trade-off between anomaly detection, semantic alignment, and uncertainty estimation.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluate MemoVAD on two large-scale weakly supervised video anomaly detection benchmarks. (1) UCF-Crime [Sultani et al., 2018] contains 1,900 real-world surveillance videos spanning 13 anomaly categories and normal activities. (2) XD-Violence [Wu et al., 2020] consists of 4,754 videos collected from movies and games, providing audio-visual signals. In our experiments, we use only the visual modality.

Metrics. Following standard protocols, we employ the Area Under the Receiver Operating Characteristic Curve (AUC)

for UCF-Crime and Average Precision (AP) for XD-Violence to evaluate detection performance. In addition to detection performance, we strictly evaluate resource efficiency in edge scenarios. Specifically, we report Throughput (FPS) and Latency (s) to verify real-time capabilities, and define Communication Rate (CR) as the percentage of clips that trigger a VLM query.

Baselines. For comparison, Sultani et al. [Sultani et al., 2018], RTFM [Tian et al., 2021], CRFD [Wu and Liu, 2021], MSL [Li et al., 2022], MGFN [Chen et al., 2023], URDMU [Zhou et al., 2023], CLIP-TSA [Joo et al., 2023], TPWNG [Yang et al., 2024], VadCLIP [Wu et al., 2024b], OVVAD [Li et al., 2025a], and EventVAD [Shao et al., 2025] are chosen as baselines.

4.2 Comparison with State-of-the-Art Methods

Main Results. As presented in Table 1, MemoVAD achieves 89.45% AUC on UCF-Crime and 85.97% AP on XDViolence, consistently outperforming prior approaches with C3D/I3D or CLIP-based feature backbones. Such consistent gains across two widely used benchmarks indicate the strong effectiveness and robustness of our method under different anomaly categories and evaluation metrics. In particular, MemoVAD improves the best baselines TPWNG and VadCLIP by an absolute margin of 1.66% and 1.43% AUC on UCF-Crime, and 2.29% and 1.46% AP on XD-Violence, respectively. Notably, the gains are achieved by utilizing the frozen VideoMAE-S as a resource-efficient student backbone, demonstrating that accurate anomaly detection does not strictly require computationally intensive foundation models. Efficiency Results. MemoVAD is explicitly designed for real-time anomaly detection under an edge-collaborative setting, where both on-device computation and communication costs are critical. As detailed in Table 2, MemoVAD maintains real-time capability on the Jetson AGX Orin by achieving a throughput of 36.2 FPS. The performance represents a substantial improvement over the I3D baseline, which operates exclusively on the edge at 23.7 FPS. Furthermore, MemoVAD significantly outpaces the CLIP-based baseline that relies continuously on remote VLM support, running at only 11.5 FPS. The throughput improvement is also reflected in the latency metrics, where MemoVAD achieves a rapid response time of 0.475 seconds, markedly outperforming both the CLIP-based baseline and the I3D-based baseline. Such a performance gain validates the efficacy of utilizing the lightweight VideoMAE-S architecture, which avoids the computational bottlenecks often associated with the heavier semantic backbones employed by competing methods. Beyond local computational efficiency, MemoVAD significantly minimizes the dependency on remote VLM access. While the CLIP-based baseline necessitates a 100% query rate to function, our method initiates VLM queries for merely 8.63% and 15.72% of the clips on the UCF-Crime and XD-Violence datasets respectively. By processing the vast majority of clips locally, MemoVAD effectively circumvents the bandwidth limitations and latency penalties often associated with expensive remote inference.

4.3 Ablation Studies

Effectiveness of Key Components. Table 3 summarizes the incremental contributions of each module. Starting with the baseline in Row 1, the frozen VideoMAE backbone achieves 72.33% AUC at 40.6 FPS, but is constrained by the lack of temporal modeling. Incorporating the causal TCE as shown in Row 2 effectively aggregates historical motion cues and boosts AUC to 79.56% with a negligible latency overhead resulting in 39.8 FPS. To gauge the maximum potential of semantic reasoning, we evaluate a teacher forcing variant with DSM in Row 3 that queries the VLM for every clip. While the configuration yields a peak AUC of 90.10%, it incurs huge costs including a 100% communication rate and a throughput drop to 12.8 FPS with a high latency of 1.481 seconds, which renders it impractical for edge deployment. To mitigate the issue, the UAG mechanism in Row 4 selectively triggers queries only for clips with high-uncertainty and novelty, reducing communication to 8.63% and restoring a real-time throughput of 37.5 FPS with 0.462 seconds, while incurring only a modest AUC drop to 87.85%. Finally, the introduced SAM alleviates the feature distribution mismatch between the student and teacher. By aligning retrieved semantics, it recovers the AUC to 89.45% and closely matches the upper bound while maintaining a high efficiency of 36.2 FPS and a marginal latency of 0.475 seconds.

Analysis of Gating Policies. To validate the design of our Uncertainty-Aware Gating, we compare the proposed method against standard baselines as detailed in Table 4. The baseline utilizing solely Softmax confidence yields suboptimal performance of 86.12%, primarily due to its inability to discriminate between hard samples and out-of-distribution data. While employing evidential uncertainty alone improves precision, it fails to capture novel semantic events that are statistically confident yet semantically unfamiliar. Consequently, our hybrid approach achieves the optimal synergy by triggering VLM queries exclusively when the model exhibits perceived uncertainty or when the input remains semantically distinct from existing memory prototypes.

Effectiveness of Memory Dynamics. We further investigate the impact of memory management on long-term learning stability in Table 5. Naive strategies, such as FIFO or Random replacement, result in the catastrophic forgetting of rare anomaly prototypes, leading to a performance degradation exceeding 2%. In contrast, our Online SemanticDiversity Replacement ensures the retention of a spanning set of diverse abnormal information, thereby maximizing the utility of the fixed storage budget of 2,048 slots.

Efficiency-Accuracy Trade-off. Fig. 3 illustrates the balance between computational efficiency and detection accuracy. To characterize the operational envelope of MemoVAD, we systematically modulate the gating hyperparameters, specifically the uncertainty threshold τunc and the similarity threshold τsim. Fig. 3a reveals a smooth Pareto frontier where the proposed method maintains an AUC exceeding 89% even when the communication overhead is constrained to less than 10%. Furthermore, we provide a comparative analysis against the baseline deployment paradigms in Fig. 3b. Specifically, while exhibiting a marginal degradation in AUC compared to the Cloud-only approach, our method significantly reduces the inference latency. Notably, the total runtime of MemoVAD is approximately one-third of that required by the Cloud-only paradigm, demonstrating its practicality for real-time edge computing scenarios. Such performance significantly surpasses the baselines and underscores the adaptability of our framework to fluctuating network bandwidths in real-world deployments.

4.4 Qualitative Analysis

Qualitative Visualization. Fig. 4 details the inference dynamics of MemoVAD across diverse scenarios. The blue curves denote the predicted anomaly scores and the black vertical stems indicate VLM query triggers. As presented in Fig. 4a, the query frequency spikes immediately at the anomaly onset to resolve high perceived uncertainty. Upon the termination of the event, the system rapidly validates the return to a normal state, resulting in a rapid decay of the anomaly score. Similarly, in Fig. 4b, where the anomaly persists until the end of the video, the system maintains robust recognition throughout the abnormal event duration. Notably, in Fig. 4c, the MemoVAD successfully validates the interval between events to confirm the temporary restoration of normality, demonstrating its ability to capture complex temporal dependencies. In the normal video depicted in Fig. 4d, our system maintains consistently low anomaly scores with extremely sparse updates, effectively minimizing computational costs in the absence of semantic shifts. Overall, MemoVAD precisely localizes events matching ground-truth red borders, while meeting rate limits via content-adaptive resources.

Feature DiscriminationVisualization. To intuitively evaluate the quality of learned representations, we visualize the feature distributions on the UCF-Crime dataset using t-SNE, as depicted in Fig. 5. The visualization in Fig. 5a reveals that the baseline VideoMAE exhibits a high degree of feature entanglement, where anomalous samples exhibit heavily overlap with normal patterns due to the lack of discriminative semantic guidance. In contrast, as illustrated in Fig. 5b, benefiting from specific optimization and semantic enrichment of MemoVAD, the learned features form distinct and compact clusters with clear decision boundaries. It further demonstrates that MemoVAD effectively disentangles anomalous features away from the normal distribution.

Figure 5: Feature visualization via t-SNE on UCF-Crime dataset.

5 Conclusion

In this work, we proposed MemoVAD, a framework that strategically bridges a lightweight edge detector with a powerful remote VLM. Instead of continuously relying on the cloud, our system learns to seek remote guidance only when uncertain, caching the insights locally to improve over time. Our experiments confirm that the uncertainty-driven collaboration yields state-of-the-art performance with minimal bandwidth usage, demonstrating a practical path for deploying foundation model capabilities in real-world surveillance without being overwhelmed by their computational weight.

Future work will explore extending the paradigm to collaborative edge computing scenarios, where distributed devices can share semantic knowledge and jointly maintain memory. We will also investigate long-term adaptation under nonstationary environments, including scene shifts and evolving anomaly patterns. In addition, more robust memory update and validation mechanisms will be studied to reduce the influence of noisy VLM feedback while preserving the efficiency benefits of selective remote querying.

Acknowledgments

The above work was supported in part by the Joint Funds of the National Natural Science Foundation of China under Grant U25A20436, Guangxi Key Research & Development Program (FN2504240036, 2025FN96441087), the National Natural Science Foundation of China (NSFC) (62372047, 62302049), Guangdong S&T Programme (No. 2025B0101120006), the Natural Science Foundation of Guangdong Province (2024A1515011323), the Supplemental Funds for Major Scientific Research Projects of Beijing Normal University, Zhuhai (ZHPT2023002), the Fundamental Research Funds for the Central Universities, Guangdong Province Educational Science Planning Research Project under 2025JKZG069, Higher Education Research Topics of Guangdong Association of Higher Education in the 14th Five-Year Plan under 24GYB207, Natural Science Foundation of Sichuan under Grant 2026NSFSC1459 and support from the Interdisciplinary Intelligence Super Computer Center of Beijing Normal University at Zhuhai.