Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

1 Introduction

Video-to-Audio (V2A) Generation [Luo et al., 2024; Wang et al., 2024b; Wang et al., 2024a; Du et al., 2023], which generates corresponding audio directly from silent videos, has significant applications in film and video post-production.

During live filming, capturing clean sound is often challenging due to ambient noise interference, the faintness of certain sounds, and other factors. As a result, most sounds must be recreated in post-production. As illustrated in Figure 1, the process of adding relevant and synchronized sound effects to silent videos is known as Foley [Ament, 2014]. Traditional Foley requires skilled Foley artists to reproduce different sounds by manipulating various objects. The success of this process heavily depends on the artist's expertise and experience. Moreover, due to the vast array of sound categories and manipulable objects, this process is labor-intensive and time-consuming. This complexity hinders large-scale replication and individual video creation. In contrast, V2A Generation [Luo et al., 2024; Wang et al., 2024b;

∗Corresponding author.

Wang et al., 2024a; Du et al., 2023] offers an appealing alternative by automatically generating corresponding audio directly from silent videos. This innovative approach alleviates the burden on human labor and offers a more scalable and faster solution for both individuals and companies.

Significant progress has been made in V2A Generation. SpecVQGAN [Iashin and Rahtu, 2021] leads the way in audio generation for open-domain videos. Diff-Foley [Luo et al., 2024] stands out for its focus on addressing the challenge of audio-visual temporal synchronization, sparking a wave of subsequent research with impressive advancements in model performance [Wang et al., 2024b; Zhang et al., 2024a; Ren et al., 2024], precise temporal alignment [Pascual et al., 2025; Viertola et al., 2024], lightweight designs [Wang et al., 2024a], and text control integration [Du et al., 2023; Xie et al., 2024; Jeong et al., 2024]. However, current methods overlook the role of cinematic language [Mercado, 2019], a cornerstone of artistic expression in film and video.

Cinematic language [Mercado, 2019] has the expressive power of storytelling, enabling directors to convey subjective intentions effectively. Camera techniques such as close-ups and camera movements are common to cinematic language. For instance, close-ups highlight specific features of characters or objects by zooming in, while camera movements dynamically introduce or remove them from the frame. These camera techniques aim to portray characters as shown in Figure 2, such as characters playing instruments or engaging in sports, where Foley targets may only be partially visible in the spatial frame or temporal sequence. We refer to these situations where Foley targets are partially visible spatially or temporally as partial visibility. In these situations, humans can infer sound based on other visual clues and temporal clues, even with incomplete information. However, current V2A methods struggle to handle the challenges posed by partial visibility, leading to poor performance in such scenarios.

State-of-the-art (SOTA) V2A models [Luo et al., 2024; Wang et al., 2024b; Zhang et al., 2024a] typically employ a two-stage training process: large-scale pretraining on audiovisual datasets to learn a robust video encoder, followed by training an audio generator conditioned on the extracted video features. In cinematic language scenarios, partial visibility of Foley targets causes the video encoder to extract inaccurate video features. This misrepresentation results in incorrect audio-visual associations, hindering the generation of corresponding audio. To directly adapt these models for cinematic scenarios, a straightforward approach would be collecting cinematic videos to retrain both the video encoder and the audio generator. However, this approach faces substantial obstacles. High-quality cinematic video clips are not only scarce but also restricted by copyright. Furthermore, training models directly on cinematic videos with incomplete visual information can be problematic. Such outliers may cause the model puzzled, rather than enabling it to effectively learn from partial visual information, ultimately disrupting optimization and resulting in unexpectedly poor performance. Interestingly, current V2A models exhibit strong performance on non-cinematic videos. This suggests that the pre-trained knowledge can be leveraged to bridge the performance gap between non-cinematic and cinematic scenarios.

In this paper, we propose a simple teacher-student framework to capture partial audio-visual clues by constructing paired supervision video, focusing on addressing the challenges of partial visibility in cinematic scenarios. First, we simulate fundamental cinematic language variations to create paired training videos: one with cinematic variations and the other without while preserving consistent audio-visual correspondences. The latter serves as a supervision signal, subtly guiding the model to transfer its prior knowledge to understand the partial visual information. These paired videos are more effective than directly training on collected cinematic videos. Next, we adopt a teacher-student framework to align the video features from these paired training videos, benefiting from the supervision signals. This not only enables the student model to learn the associations between sounds and partial visual clues but also preserves its original performance. Our approach is both efficient and general, requiring neither additional cinematic data nor modifications to the subsequent audio generative model. It achieves impressive improvements under partial visibility across all evaluation metrics and enhances performance on the large-scale V2A dataset, VGGSound [Luo et al., 2024] compared to the baseline. Our contributions are as follows:

• We are the first to focus on cinematic language in the V2A generation area, where the Foley targets are only partially visible.
• We propose an efficient and general teacher-student framework that captures partial audio-visual clues by creating paired training videos while maintaining its original performance.

2 RelatedWork

2.1 V2AGeneration

The progress in V2A Generation research attracts a lot of attention. Early methods [Chen et al., 2020b] typically train separate models for each video category to generate more relevant and higher-fidelity audio, which limits the generalization abilities of models. SpecVQGAN [Iashin and Rahtu, 2021] makes a pioneering effort in audio generation for opendomain videos. Diff-Foley [Luo et al., 2024] stands out by addressing the challenge of audio-visual temporal synchronization, inspiring a wave of subsequent studies with impressive results in enhanced model performance [Wang et al., 2024b; Zhang et al., 2024a; Ren et al., 2024], precise temporal alignment [Pascual et al., 2025; Viertola et al., 2024], lightweight designs [Wang et al., 2024a], and the integration of text control [Du et al., 2023; Xie et al., 2024; Jeong et al., 2024]. Despite these advancements, current methods all neglect cinematic language and perform poorly in such scenarios, which is the focus of our work.

2.2 Partial Visual Clues Perception

Partial visual clues often arise when the predicted targets are partially occluded or out of sight. The ability to perceive partial visual information is a common challenge across various visual tasks. In traditional computer vision tasks, such as image classification and segmentation, models [He et al., 2016; Huang et al., 2017] need to make inferences based on visible portions. For person re-identification, models [Miao et al., 2019; Li et al., 2021] are required to identify the same individual despite variations caused by camera perspectives and occlusions. Similarly, robots need to predict human motion directions and trajectories based on visible human body parts [Zhao et al., 2024]. Scene understanding tasks [Zhan et al., 2020; Lee and Park, 2022] also face this challenge, as models must infer semantic information from incomplete visual clues, especially in applications like autonomous driving. In our research, we encounter similar difficulties related to partial visibility in V2A generation.

2.3 Teacher-Student Methodology

The Teacher-Student methodology, also known as knowledge distillation [Hinton, 2015], involves training a student model to learn from a teacher model. This approach aims to achieve model compression or enhance performance. Selfdistillation [Zhu et al., 2018; Xu and Liu, 2019; Zhang et al., 2019; Yun et al., 2019] is a unique variant of the TeacherStudent methodology, where the teacher and student models are different versions or stages of the same model. This characteristic enables the model to learn from itself. In our work, we adopt this practical training methodology, leveraging the prior knowledge of a pre-trained video encoder to guide the learning of audio-visual correlation under partial visibility.

2.4 Data Augmentation

Data augmentation is widely applied to improve model performance and generalization abilities. In image-related fields, data augmentation techniques are widely used in traditional computer vision tasks [He et al., 2016; Huang et al., 2017], such as flipping, rotation, translation, and noise injection. Moreover, video-based data augmentation methods [Gowda et al., 2022] also gain attention in video action recognition. These methods concentrate more on actions and temporal information by altering color [Zhang et al., 2024b] or blending foregrounds and backgrounds [Ding et al., 2022; Li et al., 2023; Wang et al., 2021] from different videos. Our method simulates cinematic language variations to emphasize audio-visual alignment using partial visual clues.

3 Method

An overview of our proposed method is shown in Figure 3. First, we introduce the motivation behind our method in Section 3.1. Next, we propose cinematic language variations f in Section 3.2 and adopt the teacher-student methodology to learn partial visibility in Section 3.3. Finally, in Section 3.4, we describe the latent diffusion models for V2A Generation with cinematic language.

3.1 Motivation

Our method targets the challenge of partial visibility in cinematic language scenarios. To do so, we aim to improve the model's ability to capture audio-visual corrections from partial visual information.

As illustrated in Figure 1, most Video-to-Audio (V2A) models follow a two-stage process: first, a feature extractor E extracts visual features c = E(xv) from the video xv. Subsequently, using these visual features c as a condition, a generative model G generates the corresponding audio xa = G(c). Indeed, the performance of the generative model G relies heavily on the semantic and temporal information embedded in the condition c. Therefore, the quality of the condition c is critical for generating relevant and synchronized audio.

For normal video clips xv, i.e., without cinematic language, the generative model G performs well, indicating that the extracted visual features c = E(xv) provide accurate audio information, making them ideal generation conditions. However, when x′v with cinematic language, the quality of the generated audio x′a deteriorates significantly. Given the strong performance of G on non-cinematic videos, we reasonably infer that this degradation stems from the poor generation conditions c′ = E(x′v ), rather than the generative model G itself. In other words, when the Foley targets are only partially visible, the video encoder E struggles to capture meaningful audio-visual associations from partial visual clues, thereby failing to provide a suitable generation prior.

Consequently, assisting the video encoder in learning the associations between sounds and partial visual clues and providing better conditions for the generative model G emerges as a natural approach.

3.2 Cinematic Language Variations

We propose Cinematic Language Variations f to simulate partial visibility in cinematic language [Mercado, 2019], helping to establish audio-visual associations in such scenarios. This is more effective than directly training on collected cinematic videos. In cinematic language, close-ups and camera movements are two common camera techniques that cause situations where the Foley targets are partially visible in the spatial or temporal dimension. The cinematic language variations f simulates the process of these two techniques as examples.

Given that close-ups emphasize specific local details of characters or objects, fcu uniformly crops the video frames. The cropping sizes H′ and W ′ are randomly selected within a reasonable range, which are determined as follows:

where H and W are the original height and width of the video frames, and rh, rw ∼ U(a1, a2). The range [a1, a2] should be chosen to ensure that the Foley targets are partially visible while retaining sufficient visual information.

To simulate camera movements, fcm utilizes the same cropping size above and shifts the shot along the central axis of the video frames. The shift direction is:

• WhenW >H, shift left or right at random.
• When H > W, shift up or down at random.

3.3 Partial Visibility Learning by Self-distillation

We adopt the teacher-student methodology to learn the partial visibility simulated by Cinematic Language Variations f . Given that the pre-trained video encoder contains rich audiovisual priors, we utilize it as the teacher model T to guide the student model S in learning the association between sounds and partial visual clues, as illustrated in Figure 3.

Given video data xv, by simulating cinematic language variations f , we obtain video xfv . Derived from xv, xfv retains the same semantic and temporal information of the Foley target with xv. They constitute training video pairs (xv, xfv ) with the same audio-visual correspondence. The only difference between training video pairs is the partial visibility of the Foley target: it is fully visible in video xv while partially visible in video xfv . Since the Foley target in video xv is fully visible, the visual features ct = T(xv) extracted by the teacher model T accurately encompass sufficient audio information for conditioning. Thus, aligning the features csf = S(xfv ) extracted by the student model S with feature ct can facilitate the student model S in learning the audiovisual connection between sounds and partial visual clues. During training, we also align cs = S(xv) with the feature ct to maintain the performance on the original dataset. The optimization loss Lp is defined as follows:

We introduce k as the proportion of training video clips with cinematic language variations. By aligning the visual features of training video pairs, we can effectively guide the student model S to map videos with partial visibility into the original feature space of the teacher model. This approach not only learns a better generation condition but also requires no modifications to the subsequent generative model.

3.4 Latent DiffusionModels

Latent Diffusion Models (LDMs) [Rombach et al., 2022] are probabilistic generative models that map the data distribution into a low-dimensional latent space, consisting of an autoencoder and a U-Net denoiser. In the V2A task, the latent encoder E encodes the Mel-spectrogram x ∼ p(x) into the latent representation z = E(x), and the UNet denoiser ϵθ is then trained to reverse the noise addition to generate new latent representations. Under a given condition c, the optimization [Ho et al., 2020; Song et al., 2021] process of LDMs can be defined as follows:

where ϵ represents Gaussian noise, and zt denotes the latent representation at time step t.

For conditional LDMs, classifier-free guidance (CFG) [Ho and Salimans, 2022] is a widely used alternative to classifier guidance (CG) [Dhariwal and Nichol, 2021]. CFG jointly trains conditional ϵθ(xt, c, t) and unconditional ϵθ(xt, t) diffusion models by randomly dropping the condition c. During sampling, the noise prediction is calculated as:

where w is the guidance scale.

In our study, we build upon the open-source DiffFoley [Luo et al., 2024], an LDM conditioned on video features extracted from a frozen CAVP video encoder, to achieve Foley in cinematic scenarios.

4 Experiments

Datasets. We conduct our experiments using VGGSound [Chen et al., 2020a], a large-scale audio-visual dataset containing over 200,000 video clips across 309 distinct sound categories. Existing V2A methods utilize VGGSound for both training and evaluation. We follow the original VGGSound train/test split.

Baseline. Given our choice of Diff-Foley [Luo et al., 2024] as the foundation model for our study, we select Diff-Foley as our baseline, which is a leading open-source V2A model. For simplicity, we adopt the CFG configuration for both generation and comparison, keeping all other settings unchanged.

Evaluation Metrics. For evaluation, we adopt evaluation metrics FAD, FD, KID, KL, and ISc from audioLDM [Liu et al., 2023], alongside Align Acc from Diff-Foley [Luo et al., 2024]. FAD, FD, and KID measure the similarity between real and generated audio, while KL assesses the paired similarity in probability distributions. ISc measures the diversity and quality of generated audio, and Align Acc assesses the audio-visual synchronization.

Implementation Details. For model configuration, we employ a pre-trained video encoder from CAVP [Luo et al., 2024] as the teacher model. The student model adopts the same architecture, with weights initialized from the teacher model's pre-trained parameters. The input video clips are sampled at 4 frames per second (FPS), resulting in T = 4N frames for each N-second video clip. Then the input video xv ∈ RT×3×H×W is extracted by the video encoder into video feature Ev ∈ RT×C , with the feature dimension C = 512. For training, we only apply cinematic language variation fcu on VGGSound [Chen et al., 2020a] training set with k = 75%, where a1 = 0.4 and a2 = 0.6. The student model is trained for 25 epochs on 4 NVIDIA 4090 GPUs, using the AdamW optimizer with a learning rate of 5×10-4 and a total batch size of 32.

To evaluate performance under partial visibility, we create two modified test sets by applying cinematic language variations to the VGGSound [Chen et al., 2020a] test set. Specifically, fcu is used to create VGG-CU (close-ups) test set, and fcm is used to create VGG-CM (camera movements) test set. Following Diff-Foley [Luo et al., 2024], we generate 10 samples per video to ensure reliable evaluation. For simplicity, we use only CFG [Ho and Salimans, 2022] configuration in Diff-Foley, keeping all other experimental settings unchanged, including the DPM-Solver [Lu et al., 2022] Sampler with 25 inference steps and CFG scale ω = 4.5.

4.1 V2AGeneration with Cinematic Language

Simulated Cinematic Scene on VGGSound. Table 1 presents the quantitative results across three test sets: the original VGGSound test set, VGG-CU test set (created using fcu), and VGG-CM test set (created using fcm). In cinematic language scenarios, we observe that Diff-Foley [Luo et al., 2024] baseline shows a significant decline in all evaluation metrics on both VGG-CU and VGG-CM test sets compared to its performance on the original VGGSound test set. In contrast, our method outperforms the baseline by a substantial margin across all evaluation metrics on these two test sets. Notably, even when trained only on VGG-CU training set, our model exhibits considerable improvement on VGGCM test set. These results suggest that our method effectively learns the associations between audio and partial visual information. Furthermore, by learning partial visual information, our method achieves improved performance on the original VGGSound test set compared to the baseline.

Figure 4 presents the qualitative results for V2A generation in cinematic language scenarios involving close-ups and camera movements. Taking the machine gun shooting video as an example, partial visual information is highlighted with pink dashed boxes and pink text, while the corresponding auditory signals are represented in blue. As depicted in Figure 4, we can easily identify the machine gun and the moment of the sound from the video frames, even when the gun is partially visible. However, Diff-Foley fails to generate the appropriate gunshot and other sounds at the correct moment. In contrast, our method successfully generates synchronized gunshot audio when the Foley target is partially visible. Notably, in camera movements scenarios shown in Figure 4(b), the visual clues in the 0s and 1s video frames are lost. Our method does not generate sound when the visual clues are completely missing. While in the following video frames, our method correctly generates relevant and synchronized audio. These results suggest that our method correctly learns the audio-visual associations under partial visibility.

Real Cinematic Scene. To evaluate performance in realworld cinematic scenarios, we collect a new set of 31 YouTube videos that feature a wide range of cinematic techniques, ensuring no overlap with the original VGGSound test set. These videos include characteristics such as close-ups, camera movements, zooms, scene transitions, as well as nonpartial factors like color and lighting variations. We conducted a human study with 20 participants and the results confirm the superiority of our method as shown in Table 2,

4.2 Analysis of Partial Visibility Learning

Visualization Analysis. To better understand what the model learns, we visualize the model's attention areas along the video frames in close-up scenarios. As shown in Figure 6, we observe that the four regions highlighted by the model's attention (denoted by pink dashed lines) are semantically relevant to the partial visual clues. Compared to the ground truth audio, these highlighted regions are also temporally aligned with the corresponding sound. In contrast, the highlighted regions of Diff-Foley's attention are misaligned with the partial visual clues and are not temporally synchronized with the audio. These visualized results demonstrate that our model accurately captures the semantic and temporal association between sound and partial visual clues.

Improvements in Different Video Categories. To further understand where our approach is superior, We evaluate the Align Acc metric for each video category on VGG-CU test set and present the top 25 video categories (309 in total) with the most significant improvements. As shown in Figure 5, these categories primarily include instrumental sounds, ball impact sounds, and vehicle movement sounds that are commonly found in cinematic scenarios. In these categories, DiffFoley exhibits notably lower audio-visual consistency compared to its average performance. In contrast, our method achieves significantly better results, demonstrating its effectiveness in addressing the challenges of partial visibility in cinematic scenarios.

4.3 Ablation Study

The Impact of Different Variations. As shown in Table 3, we introduce three cinematic language variations during training to assess their impact: fcu, fcm, and fcu&cm. The fcu variation simulates spatial partial visibility, while fcm simulates temporal partial visibility. According to FAD, FD, and KID metrics, only using fcu yields the best improvements across all three test scenarios, suggesting that the generated audio is semantically more aligned with the video. This indicates that simulating spatial partial visibility (fcu) is more effective than simulating temporal partial visibility (fcm) for learning the semantic association between sounds and partial visual clues. On the other hand, in terms of Align ACC, fcu&cm achieves the best improvements in VGG-CU and VGG-CM test sets, with the generated audio showing better temporal alignment with the videos. This suggests that simulating both spatial and temporal partial visibility (fcu&cm) enhances the model's ability to learn the temporal synchronization between sounds and partial visual clues.

The Proportion of Variation. Table 4 demonstrates the effect of the proportion k of training video clips with cinematic language variation fcu. In VGG-CU and VGG-CM test sets, model performance improves as k increases from 50% to 75%, showing the effectiveness of fcu. When k reaches 100%, where no data from the original dataset is used, the model's performance shows a decline compared to 75%, emphasizing the importance of training with the paired data.

5 Limitations and Broader Impact

Limitations. Our method excels in scenarios with partial visibility such as camera movements and close-ups, but its adaptability and performance across other diverse cinematic languages still require further exploration.

Broader Impact. V2A boosts video production efficiency, offering substantial benefits to creators. However, vigilance and regulations are still needed to prevent potential misuse.

6 Conclusion

We present a simple yet effective method to address the challenge of partial visibility in cinematic language scenarios by providing a better condition for the audio generator. By simulating the cinematic language variations, the student model aligns the video features from training video pairs with the same audio-visual correspondences, which helps learn the associations between sounds and partial visual clues. Experimental results demonstrate the impressive improvements of our approach, not only in partial visibility scenarios but also on the original V2A dataset, VGGSound.

Acknowledgments

This work was partially supported by the Key Research and Development Program of Yunnan Province under Grant 202403AA080002, partially supported by the National Natural Science Foundation of China under Grant 62471344, and partially supported by the National Natural Science Foundation of China under Grant 62372341.