{"type":"ScholarlyArticle","title":"Perceptual Learned Video Compression with Recurrent Conditional GAN","authors":[{"name":"Ren Yang","slug":"ren-yang","given":"Ren","family":"Yang","affiliation":"ETH Zürich , Switzerland"},{"name":"Radu Timofte","slug":"radu-timofte","given":"Radu","family":"Timofte","affiliation":"ETH Zürich , Switzerland; Julius Maximilian University of Würzburg , Germany"},{"name":"Luc Van Gool","slug":"luc-van-gool","given":"Luc","family":"Van Gool","affiliation":"ETH Zürich , Switzerland; KU Leuven , Belgium"}],"abstract":"This paper proposes a Perceptual Learned Video Compression (PLVC) approach with recurrent conditional GAN. We employ the recurrent auto-encoder-based compression network as the generator, and most importantly, we propose a recurrent conditional discriminator, which judges raw vs. compressed video conditioned on both spatial and temporal features, including the latent representation, temporal motion and hidden states in recurrent cells. This way, the adversarial training pushes the generated video to be not only spatially photo-realistic but also temporally consistent with the groundtruth and coherent among video frames. The experimental results show that the learned PLVC model compresses video with good perceptual quality at low bit-rate, and that it outperforms the official HEVC test model (HM 16.20) and the existing learned video compression approaches for several perceptual quality metrics and user studies. The project page is available at https://github.com/RenYang-home/PLVC.","doi":"10.24963/ijcai.2022/214","url":"https://ijcai.org/proceedings/2022/214","pdf":"https://ijcai.org/proceedings/2022/0214.pdf","markdown":"https://ijcai.org/proceedings/2022/214.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/214","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":214,"pages":{"first":1537,"last":1544},"keywords":["Computer Vision: Neural generative models, auto encoders, GANs","Computer Vision: Representation Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986959","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985102","length_sec":412,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agustsson et al., 2019] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In ICCV, 2019.","title":"Generative adversarial networks for extreme learned image compression","authors":"Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, Luc Van Gool","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agustsson et al., 2020] Eirikur Agustsson, David Minnen, Nick Johnston, et al. Scale-space flow for end-to-end op- timized video compression. In CVPR, 2020.","title":"Scale-space flow for end-to-end optimized video compression","authors":"Eirikur Agustsson, David Minnen, Nick Johnston","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ballé et al., 2017] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compres- sion. In ICLR, 2017.","title":"End-to-end optimized image compression","authors":"Johannes Ballé, Valero Laparra, Eero P Simoncelli","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ballé et al., 2018] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In ICLR, 2018.","title":"Variational image compression with a scale hyperprior","authors":"Johannes Ballé, David Minnen, Saurabh Singh, Sung, Jin Hwang, Nick Johnston","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bińkowski et al., 2018] Mikołaj Bińkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demysti- fying MMD GANs. In ICLR, 2018.","title":"Demystifying MMD GANs","authors":"Mikołaj Bińkowski, J Dougal, Michael Sutherland, Arthur Arbel, Gretton","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bossen, 2013] Frank Bossen. Common test conditions and software reference configurations. JCTVC-L1100, 2013.","title":"Common test conditions and software reference configurations","authors":"Frank Bossen","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cheng et al., 2019] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learning image and video com- pression through spatial-temporal energy compaction. In CVPR, 2019.","title":"Learning image and video compression through spatial-temporal energy compaction","authors":"Cheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Choi et al., 2019] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Variable rate deep image compression with a conditional autoencoder. In ICCV, 2019.","title":"Variable rate deep image compression with a conditional autoencoder","authors":"Yoojin Choi, Mostafa El-Khamy, Jungwon Lee","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Djelouah et al., 2019] Abdelaziz Djelouah, Joaquim Cam- pos, et al. Neural inter-frame compression for video cod- ing. In ICCV, 2019.","title":"Abdelaziz Djelouah, Joaquim Campos, et al. Neural inter-frame compression for video coding","authors":"Djelouah","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NeurIPS, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.","title":"GANs trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2020a] Yueyu Hu, Wenhan Yang, and Jiaying Liu. Coarse-to-fine hyper-prior modeling for learned image compression. In AAAI, 2020.","title":"Coarse-to-fine hyper-prior modeling for learned image compression","authors":"Hu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2020b] Zhihao Hu, Zhenghao Chen, Dong Xu, et al. Improving deep video compression by resolution- adaptive flow coding. In ECCV, 2020.","title":"Improving deep video compression by resolutionadaptive flow coding","authors":"Hu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2021] Zhihao Hu, Guo Lu, and Dong Xu. FVC: A new framework towards deep video compression in feature space. In CVPR, 2021.","title":"FVC: A new framework towards deep video compression in feature space","authors":"Zhihao Hu, Guo Lu, Dong Xu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Johnston et al., 2018] Nick Johnston, Damien Vincent, David Minnen, et al. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In CVPR, 2018.","title":"Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks","authors":"Nick Johnston, Damien Vincent, David Minnen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2019] Jooyoung Lee, Seunghyun Cho, and Seung-Kwon Beack. Context-adaptive entropy model for end-to-end optimized image compression. In ICLR, 2019.","title":"Context-adaptive entropy model for end-to-end optimized image compression","authors":"Lee","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2020] Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-LVC: multiple frames prediction for learned video compression. In CVPR, 2020.","title":"M-LVC: multiple frames prediction for learned video compression","authors":"Jianping Lin, Dong Liu, Houqiang Li, Feng Wu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Haojie Liu, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma. Learned video compression via joint spatial-temporal correlation exploration. In AAAI, 2020.","title":"Learned video compression via joint spatial-temporal correlation exploration","authors":"Liu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2019] Guo Lu, Wanli Ouyang, Dong Xu, et al. DVC: An end-to-end deep video compression framework. In CVPR, 2019.","title":"DVC: An end-to-end deep video compression framework","authors":"Guo Lu, Wanli Ouyang, Dong Xu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2020] Guo Lu, Chunlei Cai, Xiaoyun Zhang, et al. Content adaptive and error propagation aware deep video compression. In ECCV, 2020.","title":"Content adaptive and error propagation aware deep video compression","authors":"Guo Lu, Chunlei Cai, Xiaoyun Zhang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lucic et al., 2018] Mario Lucic, Karol Kurach, Marcin Michalski, et al. Are GANs created equal? A large-scale study. In NeurIPS, 2018.","title":"Are GANs created equal? A large-scale study","authors":"Mario Lucic, Karol Kurach, Marcin Michalski","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ma et al., 2020] Haichuan Ma, Dong Liu, Ning Yan, et al. End-to-end optimized versatile image compression with wavelet-like transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.","title":"End-to-end optimized versatile image compression with wavelet-like transform","authors":"Ma","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mentzer et al., 2020] Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. NeurIPS, 33, 2020.","title":"High-fidelity generative image compression","authors":"Fabian Mentzer, George D Toderici, Michael Tschannen, Eirikur Agustsson","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mercat et al., 2020] Alexandre Mercat, Marko Viitanen, and Jarno Vanne. UVG dataset: 50/120fps 4K sequences for video codec analysis and development. In ACM MM- Sys, 2020.","title":"UVG dataset: 50/120fps 4K sequences for video codec analysis and development","authors":"Alexandre Mercat, Marko Viitanen, Jarno Vanne","year":"2020","venue":"ACM MM-Sys","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Minnen et al., 2018] David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In NeurIPS, 2018.","title":"Joint autoregressive and hierarchical priors for learned image compression","authors":"David Minnen, Johannes Ballé, George D Toderici","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ranjan and Black, 2017] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In CVPR, 2017.","title":"Optical flow estimation using a spatial pyramid network","authors":"Anurag Ranjan, Michael J Black","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Toderici et al., 2017] George Toderici, Damien Vincent, Nick Johnston, et al. Full resolution image compression with recurrent neural networks. In CVPR, 2017.","title":"Full resolution image compression with recurrent neural networks","authors":"George Toderici, Damien Vincent, Nick Johnston","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Veerabadran et al., 2020] Vijay Veerabadran, Reza Pour- reza, et al. Adversarial distortion for learned video com- pression. In CVPR Workshops, 2020.","title":"Adversarial distortion for learned video compression","authors":"Vijay Veerabadran, Reza Pourreza","year":"2020","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xue et al., 2019] Tianfan Xue, Baian Chen, Jiajun Wu, et al. Video enhancement with task-oriented flow. IJCV, 127(8):1106-1125, 2019.","title":"Video enhancement with task-oriented flow","authors":"Xue","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2020a] Ren Yang, Fabian Mentzer, et al. Learn- ing for video compression with hierarchical quality and recurrent enhancement. In CVPR, 2020.","title":"Learning for video compression with hierarchical quality and recurrent enhancement","authors":"Yang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2020b] Ren Yang, Luc Van Gool, and Radu Timofte. OpenDVC: An open source implementation of the dvc video compression method. arXiv preprint arXiv:2006.15862, 2020.","title":"OpenDVC: An open source implementation of the dvc video compression method","authors":"Yang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2021] Ren Yang, Fabian Mentzer, et al. Learn- ing for video compression with recurrent auto-encoder and recurrent probability model. IEEE J-STSP, 2021.","title":"Learning for video compression with recurrent auto-encoder and recurrent probability model","authors":"Ren Yang, Fabian Mentzer","year":"2021","venue":"IEEE J-STSP","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, et al. The unreasonable effectiveness of deep fea- tures as a perceptual metric. In CVPR, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}