LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning

1 Introduction

Recent advances in multimodal large language models (MLLMs) have greatly improved visual-to-text generation by encoding visual inputs as tokens processed by LLMs [Wu et al., 2023]. Compared with image-related tasks, video understanding tasks requires jointly modeling spatial content and temporal dynamics [Zheng et al., 2023; Song et al., 2024; Chen et al., 2024], and are commonly categorized into video question answering (QA) [Tian et al., 2025], short captioning [He et al., 2025], and detailed captioning [Chai et al., 2025]. Among them, the latter targets fine-grained and temporally coherent descriptions, providing richer information than the former two and better supporting downstream tasks such as video retrieval and multimodal QA. Accordingly, this work focuses on detailed video captioning, and Fig. 1 demonstrates consistent improvements over baselines across nine benchmarks spanning both detailed captioning and QA tasks.

*Corresponding authors.

Due to computational constraints, most video-LLMs adopt uniform sampling to select a fixed number of frames [Maaz et al., 2024; Li et al., 2024; Lin et al., 2023]. While this strategy ensures temporal coverage, it ignores the uneven distribution of informative events and often underrepresents short but critical actions. In contrast, query-driven frame selection methods for video QA retrieve frames relevant to a given question [Tang et al., 2025b; Zhu et al., 2025; Zhang et al., 2025], but are unsuitable for query-agnostic tasks such as detailed video captioning.

An effective frame selection strategy for detailed captioning should satisfy two requirements: (1) event awareness, capturing salient actions and scene changes, and (2) temporal diversity, avoiding redundant frames concentrated in short intervals. However, existing approaches typically fail to meet both: Top-K selection leads to temporal clustering, while reinforcement learning-based methods introduce high variance and are difficult to integrate with frozen video-LLMs.

To address these limitations, we propose a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS integrates an event-aware temporal scoring network with a stratified Top-K selection mechanism to balance event relevance and temporal diversity. Furthermore, LFS leverages caption feedback from frozen videoLLMs as supervision, directly optimizing frame selection for caption quality rather than proxy objectives, and can be seamlessly integrated into existing pipelines without modifying language model parameters.

In the evaluation of detailed video captioning, representative benchmarks such as VDC and Dream-1K report relatively low performance. Prior studies further show that even experienced human annotators achieve accuracies below 50% [Wang et al., 2024; Chai et al., 2025], highlighting a substantial gap between benchmark evaluation and human understanding. To address this gap, we introduce ICH-CC, a benchmark for detailed video captioning on intangible cultural heritage cuisine videos, constructed with fully human-authored captions and carefully designed question-answer pairs.

Experiments demonstrate that LFS significantly improves performance on ICH-CC, boosting Qwen3-VL by 3.18% and 4.47% on the Chinese and English subsets, respectively. On community benchmarks such as VDC, LFS consistently improves multiple video-LLM backbones under saturated evaluation settings. Moreover, gains in detailed captioning achieved by LFS reliably transfer to downstream video QA.

In summary, frame selection is a critical bottleneck for detailed video captioning. By explicitly modeling event relevance and temporal diversity, LFS provides an effective and easily deployable solution that improves captioning quality. Our main contributions are:

• We propose LFS, a learnable frame selector that bal-
• ances event awareness and temporal diversity via stratified Top-K selection and caption-guided supervision.
• We introduce ICH-CC, a benchmark aligned with human cognition for evaluating detailed video captioning in Chinese and English.
• We conduct extensive experiments and show that LFS consistently improves detailed video captioning and zero-shot video QA with multiple video-LLMs and benchmarks.

2 RelatedWork

2.1 Video-LLMs for Question Answering

Early Video-LLMs, such as Video-Chat [Li et al., 2024] and Video-LLaMA [Zhang et al., 2023], concatenate frame features with text prompts and fine-tune LLMs on instructiontuning datasets, enabling open-ended dialogue and basic temporal reasoning but suffering from long-context inefficiency. Subsequent works improve scalability through spatiotemporal pooling, Q-former adapters [Qasim et al., 2025; Diba et al., 2023; Kim et al., 2024], and token dropping, while specialized pre-training objectives enhance motion and event understanding. Despite these advances, most VideoLLMs remain optimized for query-driven reasoning, leaving frame selection for query-agnostic tasks relatively underexplored.

2.2 Video-LLMs for Detailed Captioning

Recent work extends video captioning from brief summaries to detailed descriptions [Kim et al., 2025; Wu et al., 2025; Ge et al., 2024]. Models such as Video-LLaVA [Lin et al., 2023], VideoChat2 [Li et al., 2024], and PLLaVA [Xu et al., 2024] are fine-tuned on large-scale video-text datasets and evaluated using standard captioning metrics, but often generalize poorly to fine-grained and temporally structured descriptions. To mitigate this issue, benchmarks such as Dream1K [Wang et al., 2024] and VDC [Chai et al., 2025] have been introduced, along with methods like AuroraCap that reduce visual tokens via token merging. Nevertheless, most approaches still rely on uniform frame sampling, which frequently misses short yet informative events in long videos.

2.3 Frame Selection before Video-LLMs

To reduce computational cost, prior frame selection methods aim to retain task-relevant frames. Rule-based approaches use uniform subsampling or shot boundary detection, while model-based methods typically select Top-K frames based on query relevance using CLIP-like models [Guo et al., 2025;

Hu et al., 2025; Tang et al., 2025a]. Token-level selectors further prune redundant visual tokens within frames. While effective for query-driven tasks such as video QA, these methods often overlook temporal diversity and are not designed for query-agnostic settings [Li et al., 2025a; Buch et al., 2025]. In contrast, detailed video captioning requires selecting event-aware and temporally-diverse frames that capture distinct events across the entire video, a requirement largely unaddressed by existing frame selection approaches.

3 Method

3.1 Overall Framework

We propose Learnable Frame Selector (LFS), a lightweight and plug-and-play module that selects a small set of informative and temporally diverse frames for detailed video captioning, as shown in Fig.2.

Given frozen frame embeddings extracted by a vision encoder Long-CLIP [Zhang et al., 2024], LFS predicts a continuous temporal importance field over all frames using a lightweight temporal scoring network (TSNet). Unlike uniform sampling or query-dependent retrieval, LFS models frame importance in a query-agnostic manner. At inference, a stratified Top-K strategy enforces temporal coverage and avoids frame clustering. During training, LFS is optimized with caption-guided supervision from a frozen video-LLM as captioner, directly aligning frame selection with caption quality.

3.2 Temporal Importance Modeling

Let X = {xt}Nt=1 , xt ∈ Rd denote frame embeddings from the frozen Long-CLIP. LFS predicts an importance logit s(t) for each frame using TSNet shown in Fig.3.

TSNet first applies a temporal convolution to capture local transitions:

A global summary g = Mean(H1) is used for gated modulation:

which adaptively rescales temporal features while preserving near-identity behavior at initialization.

A second temporal convolution aggregates local responses:

followed by a pointwise projection:

Finally, logits are normalized within each video:

ˆ

s(t) = s(t) - µs

√σ2

s + ϵ

(4)

3.3 Stratified Top-K Frame Selection

The normalized logits ˆ s(t) are converted into a soft importance distribution:

where τ is a temperature parameter annealed during training.

To avoid premature collapse of the importance distribution during early training, we include an entropy regularization termH(p) that encourages exploration over frames when the selector is still uncertain.

At inference time, LFS selects exactly K frames using a stratified Top-K strategy. The video timeline is divided into K equal temporal segments, and one frame is selected from each segment:

This strategy guarantees full temporal coverage and prevents temporal clustering. When applicable, the first and last frames are always retained.

During training, gradients are propagated through the soft distribution p(t), while the stratified Top-K operator is applied only for hard frame selection at inference.

3.4 Caption-Guided Optimization

To align frame importance with detailed captioning quality, LFS is trained under the guidance of a frozen video captioner. Given temporal importance scores p(t), we select a truncated set of top-ranked frames to control computation and normalize their weights to obtain w with ∑t wt = 1. This truncatedand-renormalized design enables efficient optimization under a fixed frame budget while preserving differentiability.

To encourage compact frame selection, we impose an ℓ1 regularization on w. Although w is normalized, the ℓ1 penalty operates on the truncated candidate set prior to renormalization, discouraging mass spreading and promoting concentration on representative frames. In addition, an entropy regularizer is applied to p(t) to encourage early exploration and is gradually annealed during training.

We inject the frame weights w into the captioner through a differentiable weighting hook at a designated visual module. Let Ft ∈ RL×D denote per-frame visual features at the hooked layer. The hook computes a fused representation:

which is broadcast back to match the original tensor shape. This operation acts as a global modulation signal, while the captioner's internal temporal and linguistic modeling remains unchanged.

The captioner input consists of a textual prompt concatenated with the ground-truth caption. We compute token-level cross-entropy only on caption tokens, masking prompt and padding tokens:

where C indexes caption tokens. The captioner parameters θ are frozen, so gradients propagate only to the temporal scoring network through w.

To reduce captioner bias and stabilize optimization, we adopt a relative caption objective by subtracting a uniform baseline computed over the same truncated frame set:

where wuni assigns equal weights to the selected frames.

The final training objective is:

where the ℓ1 term promotes compact frame weighting, the entropy regularizer encourages exploration, and the relative caption loss aligns frame selection with captioning quality.

3.5 Training Details

LFS is trained for five epochs using AdamW optimizer with a learning rate of 1 × 10-4 and weight decay of 1 × 10-4 . We use mixed-precision training with a batch size of 1. The temperature parameter is annealed from τstart = 2.0 to τend =

1.0. Sparsity and entropy regularization weights are set to λ0 = 0.01 and λent = 0.01, respectively, and the number of selected frames is capped at 16 per video. The frozen captioning model used for caption-guided supervision is Qwen3VL-8B. In the training process of LFS, we employed three NVIDIA A800 GPUs.

The training set consists of 1,588 videos of 2-3 minutes collected fromWebVid-10 [Bain et al., 2021], TGIF [Li et al., 2016], Charades [Sigurdsson et al., 2016], YouCook2 [Zhou et al., 2018], and TREC-VTT [Awad et al., 2023]. Training captions are obtained via distillation from Qwen3-VL-253BA22B.

4 ICH-CC Benchmark

Existing video understanding benchmarks, such as VDC and Dream-1K, primarily prioritize evaluation challenges and thus leads to misalignment with human cognitive understanding ability. Therefore, we present a ICH-CC benchmark to reflect human cognitive ability.

As shown in Fig.4, five experienced annotators each carefully labeled 20 videos only, producing the detailed captions and 100 QA counterparts, and conducted five rounds of crosschecking to ensure that every item was reviewed through all annotators. All videos are sourced from real-world business

scenarios in the context of intangible cultural heritage Chi-

nese cuisine.

ICH-CC consists of two subsets, ICH-CC-en for English and ICH-CC-zh for Chinese, with each containing 500 evaluation questions designed from 100 videos (2-3 minutes each). ICH-CC has a balanced composition of question types: detail-related, entity-relation, event-evolution, and background questions account for 20%, 20%, 40%, and 20%, respectively.

Models are tasked with generating a detailed caption for each video, which is then assessed by answering associated objective questions derived from human-authored descriptions. A question is deemed correct if the caption provides sufficient evidence to support the correct answer. The final score is calculated based on the percentage of correctly answered questions, with a focus on both semantic coverage and temporal consistency.

5 Experimental Results

5.1 Analysis of ICH-CC benchmark

Quantitative Analysis. Table 1 reports results on ICH-CC, where ICH-CC-en/ICH-CC-zh denote accuracy on English and Chinese subsets. Strong baselines like Qwen2.5-VL-7B and Qwen3-VL-8B show competitive performance. Integrating LFS yields consistent gains under a 16-frame budget. LFS + Qwen2.5-VL-7B improves overall accuracy from 68.46% to 71.77%. LFS + Qwen3-VL-8B achieves 75.05%, with +4.47% on ICH-CC-en and +3.14% on ICH-CC-zh. These results show LFS with event awareness and temporal diversity effectively enhances detailed video captioning.

Qualitative Analysis. Figure 5 compares uniform sampling and LFS on an ICH-CC video (Nai-Lao-Wei) with 16 frames. Uniform sampling captures only five event-aware frames and misses key short steps, while LFS retrieves eight covering the main procedure. Bar plots show uniform sampling wastes frames on redundant intervals, whereas the stratified Top-K

strategy distributes samples across the timeline and concentrates on high-importance segments, preserving both early context and late-stage frames. Consequently, captions from LFS-selected frames recover more detailed procedures (boiling, cooling, adding rice wine, stirring, garnishing, and baking) than uniform sampling.

5.2 Analysis of VDC Benchmark

We evaluate on VDC. Following AuroraCap, we prompt Llama-3.1-8B to answer based on predicted captions. Table 2 shows integrating LFS with AuroraCap-7B and Qwen3-VL8B yields consistent improvements across categories. The largest gains appear in Detailed, where accuracy improves from 41.30% to 43.04% for AuroraCap-7B and 55.56% to 57.58% for Qwen3-VL-8B. These results indicate that eventaware scoring plus stratified selection better captures critical moments.

5.3 Analysis of Dream-1KBenchmark

Table 3 summarizes results on Dream-1K. Compared with vanilla Tarsier2-7B using uniform sampling, LFS + Tarsier27B achieves the same F1 (0.40), with slight recall improvement (0.48 vs. 0.47) and marginal precision decrease (0.34 vs. 0.35). This similarity mainly arises from dataset characteristics: Dream-1K consists of short clips (< 10s), where 8 frames provide near-complete coverage, leaving limited headroom. The modest recall gain indicates LFS can still recover additional cues, reflecting a recall-precision tradeoff. Dream-1K serves as a short-video robustness test, showing LFS integrates into Tarsier2-7B without degrading performance, while advantages are more evident on longer videos (VDC and ICH-CC).

5.4 Zero-Shot Video Question-Answering

Table 4 reports video QA results where models answer questions solely from video descriptions, without access to raw frames, directly testing whether detailed captions support downstream reasoning. LFS consistently improves Qwen3VL-8B across all benchmarks. On MVBench, accuracy increases from 68.7% to 70.8%, indicating improved capture of complex actions. On VideoMME without subtitles, LFS yields a +1.2% gain, suggesting stronger visual grounding from captions alone. We also observe improvements on MLYU-MCQ (79.0%) and VideoMMMU (66.8%), both of which require fine-grained event understanding.

5.5 Ablation Study

All ablations are conducted using Qwen3-VL-8B as the fixed captioning backbone to isolate the contribution of individual LFS components. Table 5 reports results with a fixed frame budget of K=16. The full model achieves the best performance across all benchmarks (72.67% on ICH-CC-en, 77.43% on ICH-CC-zh, and 57.58% on VDC Detailed).

Removing the stratified mechanism causes the largest performance drop (e.g., ICH-CC-zh: 77.43 → 71.23; VDC: 57.58 → 56.20), underscoring its importance for temporal coverage and preventing clustering. Omitting eventlevel modeling (H2) or caption-guided supervision (Lcap) degrades performance, confirming their necessity for coherent event modeling and quality optimization. Removing global gating or normalization yields smaller drops, showing their stabilizing effect. Overall, LFS relies on combining these components for accurate, temporally diverse detailed captioning.

6 Conclusion

We developed a learnable frame selector (LFS) before videoLLMs by integrating event-aware temporal modeling, stratified Top-K selection, and caption-guided supervision for video captioning. Experiments show LFS improves videoLLM backbones and produces more informative descriptions, demonstrating the value of frame selection for detailed video captioning.

Acknowledgements

We thank the National Library of China for providing realworld business scenarios and cultural context.