{"type":"ScholarlyArticle","title":"Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation","authors":[{"name":"Feizhen Huang","slug":"feizhen-huang","given":"Feizhen","family":"Huang","affiliation":"School of Computer Science , Wuhan University"},{"name":"Yu Wu","slug":"yu-wu","given":"Yu","family":"Wu","affiliation":"School of Computer Science , Wuhan University"},{"name":"Yutian Lin","slug":"yutian-lin","given":"Yutian","family":"Lin","affiliation":"School of Computer Science , Wuhan University"},{"name":"Bo Du","slug":"bo-du","given":"Bo","family":"Du","affiliation":"School of Computer Science , Wuhan University"}],"abstract":"Video-to-Audio (V2A) Generation achieves significant progress and plays a crucial role in film and video post-production. However, current methods overlook the cinematic language, a critical component of artistic expression in filmmaking. As a result, their performance deteriorates in scenarios where Foley targets are only partially visible. To address this challenge, we propose a simple self-distillation approach to extend V2A models to cinematic language scenarios. By simulating the cinematic language variations, the student model learns to align the video features of training pairs with the same audio-visual correspondences, enabling it to effectively capture the associations between sounds and partial visual information. Our method not only achieves impressive improvements under partial visibility across all evaluation metrics, but also enhances performance on the large-scale V2A dataset, VGGSound.","doi":"10.24963/ijcai.2025/131","url":"https://ijcai.org/proceedings/2025/131","pdf":"https://ijcai.org/proceedings/2025/0131.pdf","markdown":"https://ijcai.org/proceedings/2025/131.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/131","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":131,"pages":{"first":1170,"last":1178},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Video analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Ament, 2014] Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation. Routledge, 2014.","title":"The Foley grail: The art of performing sound for film, games, and animation","authors":"Vanessa Theme, Ament","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020a] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 721-725. IEEE, 2020.","title":"Vggsound: A large-scale audio-visual dataset","authors":"Chen","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020b] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 29:8292-8302, 2020.","title":"Generating visually aligned sound from videos","authors":"Chen","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. Advances in neural information processing systems, 34:8780-8794, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol Dhariwal","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2022] Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion-aware contrastive video rep- resentation learning via foreground-background merging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9716-9726, 2022.","title":"Motion-aware contrastive video representation learning via foreground-background merging","authors":"Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, Hongkai Xiong","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Du et al., 2023] Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Conditional gener- ation of audio from video via foley analogies. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426-2436, 2023.","title":"Conditional generation of audio from video via foley analogies","authors":"Du","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gowda et al., 2022] Shreyank N Gowda, Marcus Rohrbach, Frank Keller, and Laura Sevilla-Lara. Learn2augment: learning to composite videos for data augmentation in ac- tion recognition. In European conference on computer vi- sion, pages 242-259. Springer, 2022.","title":"Learn2augment: learning to composite videos for data augmentation in action recognition","authors":"Gowda","year":"2022","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton, 2015] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho and Salimans, 2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.","title":"Classifier-free diffusion guidance","authors":"Jonathan Ho, Tim Salimans","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 4700-4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Iashin and Rahtu, 2021] Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, page 2. BMVA Press, 2021.","title":"Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation","authors":"Rahtu Iashin","year":"2021","venue":"32nd British Machine Vision Conference 2021, BMVC 2021","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jeong et al., 2024] Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. arXiv preprint arXiv:2407.05551, 2024.","title":"Read, watch and scream! sound generation from text and video","authors":"Yujin Jeong, Yunji Kim, Sanghyuk Chun, Jiyoung Lee","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee and Park, 2022] Hyunmin Lee and Jaesik Park. Instance-wise occlusion and depth orders in natural scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21210-21221, 2022.","title":"Hyunmin Lee and Jaesik Park. Instance-wise occlusion and depth orders in natural scenes","authors":"Park Lee","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021] Yulin Li, Jianfeng He, Tianzhu Zhang, Xi- ang Liu, Yongdong Zhang, and Feng Wu. Diverse part dis- covery: Occluded person re-identification with part-aware transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2898- 2907, 2021.","title":"Diverse part discovery: Occluded person re-identification with part-aware transformer","authors":"Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, Feng Wu","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2023] Haoxin Li, Yuan Liu, Hanwang Zhang, and Boyang Li. Mitigating and evaluating static bias of ac- tion representations in the background and the foreground. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19911-19923, 2023.","title":"Mitigating and evaluating static bias of action representations in the background and the foreground","authors":"Haoxin Li, Yuan Liu, Hanwang Zhang, Boyang Li","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023] Haohe Liu, Zehua Chen, Yi Yuan, Xin- hao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the Inter- national Conference on Machine Learning, pages 21450- 21474, 2023.","title":"AudioLDM: Text-to-audio generation with latent diffusion models","authors":"Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, Mark D Plumbley","year":"2023","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic mod- els. arXiv preprint arXiv:2211.01095, 2022.","title":"Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models","authors":"Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2024] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.","title":"Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models","authors":"Luo","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mercado, 2019] Gustavo Mercado. The filmmaker's eye: The language of the lens: The power of lenses and the expressive cinematic image. Routledge, 2019.","title":"The filmmaker's eye: The language of the lens: The power of lenses and the expressive cinematic image","authors":"Gustavo Mercado","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Miao et al., 2019] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 542-551, 2019.","title":"Pose-guided feature alignment for occluded person re-identification","authors":"Miao","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pascual et al., 2025] Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serrà. Masked generative video-to-audio transformers with enhanced synchronicity. In European Conference on Computer Vision, pages 247- 264. Springer, 2025.","title":"Masked generative video-to-audio transformers with enhanced synchronicity","authors":"Pascual","year":"2025","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ren et al., 2024] Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu. Sta-v2a: Video- to-audio generation with semantic and temporal align- ment. arXiv preprint arXiv:2409.08601, 2024.","title":"Sta-v2a: Videoto-audio generation with semantic and temporal alignment","authors":"Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, Dong Yu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on com- puter vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2021] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021.","title":"Denoising diffusion implicit models","authors":"Jiaming Song, Chenlin Meng, Stefano Ermon","year":"2021","venue":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Viertola et al., 2024] Ilpo Viertola, Vladimir Iashin, and Esa Rahtu. Temporally aligned audio for video with autore- gression. arXiv preprint arXiv:2409.13689, 2024.","title":"Temporally aligned audio for video with autoregression","authors":"Ilpo Viertola, Vladimir Iashin, Esa Rahtu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self- supervised video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 11804-11813, 2021.","title":"Removing the background by adding the background: Towards background robust selfsupervised video representation learning","authors":"Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, Xing Sun","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2024a] Heng Wang, Jianbo Ma, Santiago Pas- cual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 15492-15501, 2024.","title":"V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models","authors":"Wang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2024b] Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching. arXiv preprint arXiv:2406.00320, 2024.","title":"Frieren: Efficient video-to-audio generation with rectified flow matching","authors":"Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xie et al., 2024] Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Sonicvisionlm: Playing sound with vi- sion language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26866-26875, 2024.","title":"Sonicvisionlm: Playing sound with vision language models","authors":"Zhifeng Xie, Shengye Yu, Qile He, Mengtian Li","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu and Liu, 2019] Ting-Bing Xu and Cheng-Lin Liu. Data- distortion guided self-distillation for deep neural networks. In Proceedings of the AAAI conference on artificial intel- ligence, pages 5565-5572, 2019.","title":"Datadistortion guided self-distillation for deep neural networks","authors":"Bing Ting, Cheng-Lin Xu, Liu","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023- 6032, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhan et al., 2020] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self- supervised scene de-occlusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3784-3792, 2020.","title":"Selfsupervised scene de-occlusion","authors":"Zhan","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2019] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713-3722, 2019.","title":"Be your own teacher: Improve the performance of convolutional neural networks via self distillation","authors":"Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2024a] Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494, 2024.","title":"Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds","authors":"Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2024b] Yitian Zhang, Yue Bai, Huan Wang, Yizhou Wang, and Yun Fu. Don't judge by the look: Towards motion coherent video representation. In The Twelfth International Conference on Learning Represen- tations, 2024.","title":"Don't judge by the look: Towards motion coherent video representation","authors":"Zhang","year":"2024","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhao et al., 2024] Jieting Zhao, Hanjing Ye, Yu Zhan, Hao Luan, and Hong Zhang. Human orientation estimation un- der partial observation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11544-11551. IEEE, 2024.","title":"Human orientation estimation under partial observation","authors":"Zhao","year":"2024","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2018] Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Ad- vances in neural information processing systems, 31, 2018.","title":"Knowledge distillation by on-the-fly native ensemble","authors":"Zhu","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}