{"type":"ScholarlyArticle","title":"Event-driven Video Deblurring via Spatio-Temporal Relation-Aware Network","authors":[{"name":"Chengzhi Cao","slug":"chengzhi-cao","given":"Chengzhi","family":"Cao","affiliation":"University of Science and Technology of China , China"},{"name":"Xueyang Fu","slug":"xueyang-fu","given":"Xueyang","family":"Fu","affiliation":"University of Science and Technology of China , China"},{"name":"Yurui Zhu","slug":"yurui-zhu","given":"Yurui","family":"Zhu","affiliation":"University of Science and Technology of China , China"},{"name":"Gege Shi","slug":"gege-shi","given":"Gege","family":"Shi","affiliation":"University of Science and Technology of China , China"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China , China"}],"abstract":"Video deblurring with event information has attracted considerable attention. To help deblur each frame, existing methods usually compress a specific event sequence into a feature tensor with the same size as the corresponding video. However, this strategy neither considers the pixel-level spatial brightness changes nor the temporal correlation between events at each time step, resulting in insufficient use of spatio-temporal information. To address this issue, we propose a new Spatio-Temporal Relation-Attention network (STRA), for the specific event-based video deblurring. Concretely, to utilize spatial consistency between the frame and event, we model the brightness changes as an extra prior to aware blurring contexts in each frame; to record temporal relationship among different events, we develop a temporal memory block to restore long-range dependencies of event sequences continuously. In this way, the complementary information contained in the events and frames, as well as the correlation of neighboring events, can be fully utilized to recover spatial texture from events constantly. Experiments show that our STRA significantly outperforms several competing methods, e.g., on the HQF dataset, our network achieves up to 1.3 dB in terms of PSNR over the most advanced method. The code is available at https://github.com/Chengzhi-Cao/STRA.","doi":"10.24963/ijcai.2022/112","url":"https://ijcai.org/proceedings/2022/112","pdf":"https://ijcai.org/proceedings/2022/0112.pdf","markdown":"https://ijcai.org/proceedings/2022/112.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/112","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":112,"pages":{"first":799,"last":805},"keywords":["Computer Vision: Applications","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986973","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985116","length_sec":672,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Brandli et al., 2014] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240 × 180 130 dB 3 M s Latency Global Shutter Spatiotempo- ral Vision Sensor. IEEE Journal of Solid-State Circuits, 49(10):2333-2341, 2014.","title":"A 240 × 180 130 dB 3 M s Latency Global Shutter Spatiotemporal Vision Sensor","authors":"Brandli","year":"2014","venue":"IEEE Journal of Solid-State Circuits","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2021] Sung-Jin Cho, Seo-Won Ji, and Jun-Pyo Hong. Rethinking Coarse-to-Fine Approach in Single Im- age Deblurring. CVPR, 2021.","title":"Rethinking Coarse-to-Fine Approach in Single Image Deblurring","authors":"Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2021] Senyou Deng, Wenqi Ren, and Yanyang Yan. Multi-Scale Separable Network for Ultra-High- Definition Video Deblurring. ICCV, 2021.","title":"Multi-Scale Separable Network for Ultra-High-Definition Video Deblurring","authors":"Deng","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Durand, 2018] Frédo Durand. Burst Image Deblurring Us- ing Permutation Invariant Convolutional Neural Networks. In Vittorio Ferrari and Martial Hebert, editors, ECCV. 2018.","title":"Frédo Durand. Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks","authors":"Durand","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gallego et al., 2021] Guillermo Gallego, Tobi Delbrück, and Garrick Orchard. Event-Based Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2021.","title":"Event-Based Vision: A Survey","authors":"Guillermo Gallego, Tobi Delbrück, Garrick Orchard","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-Excitation Networks. In CVPR, 2018.","title":"Squeezeand-Excitation Networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2021] Yuhuang Hu, Shih-Chii Liu, and Tobi Del- bruck. V2e: From Video Frames to Realistic DVS Events. CVPR, 2021.","title":"V2e: From Video Frames to Realistic DVS Events","authors":"Hu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2017] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. NeurIPS, 2017.","title":"Adam: A Method for Stochastic Optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li and Xu, 2021] Dongxu Li and Chenchen Xu. ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring. CVPR, 2021.","title":"ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring","authors":"Li, Xu, Dongxu Li, Chenchen Xu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Xiang Li, Wenhai Wang, and Xiaolin Hu. Selective Kernel Networks. In CVPR, 2019.","title":"Selective Kernel Networks","authors":"Xiang Li, Wenhai Wang, Xiaolin Hu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Ang Li, Jianzhong Qi, and Rui Zhang. Gen- erative Image Inpainting with Submanifold Alignment. IJ- CAI, 2020.","title":"Generative Image Inpainting with Submanifold Alignment","authors":"Ang Li, Jianzhong Qi, Rui Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/114"},{"position":12,"raw":"[Mou et al., 2021] Chong Mou, Jian Zhang, and Zhuoyuan Wu. Dynamic Attentive Graph Learning for Image Restoration. ICCV, 2021.","title":"Dynamic Attentive Graph Learning for Image Restoration","authors":"Chong Mou, Jian Zhang, Zhuoyuan Wu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nah et al., 2017] Seungjun Nah, Tae Hyun Kim, and Ky- oung Mu Lee. Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring. In CVPR, 2017.","title":"Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring","authors":"Nah","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nikzad et al., 2021] Mohammad Nikzad, Yongsheng Gao, and Jun Zhou. Attention-based Pyramid Dilated Lattice Network for Blind Image Denoising. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021.","title":"Attention-based Pyramid Dilated Lattice Network for Blind Image Denoising","authors":"Mohammad Nikzad, Yongsheng Gao, Jun Zhou","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/129"},{"position":15,"raw":"[Pan et al., 2019] Liyuan Pan, Cedric Scheerlinck, and Yuchao Dai. Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera. In CVPR, 2019.","title":"Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera","authors":"Pan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shang et al., 2021] Wei Shang, Dongwei Ren, Dongqing Zou, Jimmy S Ren, Ping Luo, and Wangmeng Zuo. Bring- ing events into video deblurring with non-consecutively blurry frames. In ICCV, pages 4531-4540, 2021.","title":"Bringing events into video deblurring with non-consecutively blurry frames","authors":"Wei Shang, Dongwei Ren, Dongqing Zou, Jimmy S Ren, Ping Luo, Wangmeng Zuo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stoffregen et al., 2020] Timo Stoffregen, Cedric Scheer- linck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Kleeman, and Robert Mahony. Reducing the Sim- to-Real Gap for Event Cameras. ECCV, 2020.","title":"Reducing the Simto-Real Gap for Event Cameras","authors":"Stoffregen","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, and Ab- hinav Gupta. Non-local Neural Networks. CVPR, 2018.","title":"Non-local Neural Networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Zihao W. Wang, Weixin Jiang, and Kuan He. Event-Driven Video Frame Synthesis. In ICCVW, 2019.","title":"Event-Driven Video Frame Synthesis","authors":"Zihao W Wang, Weixin Jiang, Kuan He","year":"2019","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Bishan Wang, Jingwei He, and Wen Yang. Event Enhanced High-Quality Image Recovery. ECCV, 2020.","title":"Event Enhanced High-Quality Image Recovery","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021] Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, and Angela Yao. Towards Compact Single Image Super-Resolution via Contrastive Self-distillation. IJCAI, 2021.","title":"Towards Compact Single Image Super-Resolution via Contrastive Self-distillation","authors":"Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, Angela Yao","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/155"},{"position":23,"raw":"[Xie et al., 2021] Fei Xie, Wankou Yang, and Kaihua Zhang. Learning Spatio-Appearance Memory Network for High- Performance Visual Tracking. In ICCVW, 2021.","title":"Learning Spatio-Appearance Memory Network for High-Performance Visual Tracking","authors":"Fei Xie, Wankou Yang, Kaihua Zhang","year":"2021","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2021] Fang Xu, Lei Yu, and Bishan Wang. Mo- tion Deblurring With Real Events. ICCV, 2021.","title":"Motion Deblurring With Real Events","authors":"Fang Xu, Lei Yu, Bishan Wang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2021] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-Wise Attention Spiking Neural Net- works for Event Streams Classification. In ICCV, 2021.","title":"Temporal-Wise Attention Spiking Neural Networks for Event Streams Classification","authors":"Yao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Luo, 2019] Kaihao Zhang and Wenhan Luo. Ad- versarial Spatio-Temporal Learning for Video Deblurring. IEEE Trans. on Image Process., 2019.","title":"Adversarial Spatio-Temporal Learning for Video Deblurring","authors":"Luo Zhang, Kaihao Zhang, Wenhan Luo","year":"2019","venue":"IEEE Trans. on Image Process","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Zhu, 2021] Wendong Zhang and Junwei Zhu. Context-Aware Image Inpainting with Learned Semantic Priors. IJCAI, 2021.","title":"Context-Aware Image Inpainting with Learned Semantic Priors","authors":"Zhu Zhang, Wendong Zhang, Junwei Zhu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/183"},{"position":28,"raw":"[Zhou and Teng, 2021] Chu Zhou and Minggui Teng. DeLiEve-Net: Deblurring Low-light Images with Light Streaks and Local Events. In ICCVW, 2021.","title":"DeLiEve-Net: Deblurring Low-light Images with Light Streaks and Local Events","authors":"Chu Zhou, Minggui Teng","year":"2021","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2019] Shangchen Zhou, Jiawei Zhang, and Jin- shan Pan. Spatio-Temporal Filter Adaptive Network for Video Deblurring. In ICCV, 2019.","title":"Spatio-Temporal Filter Adaptive Network for Video Deblurring","authors":"Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2019] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion. In CVPR, 2019.","title":"Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion","authors":"Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, Kostas Daniilidis","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zou et al., 2021] Shihao Zou, Chuan Guo, Xinxin Zuo, Sen Wang, Pengyu Wang, Xiaoqin Hu, Shoushun Chen, Minglun Gong, and Li Cheng. EventHPE: Event-Based 3D Human Pose and Shape Estimation. ICCV, 2021.","title":"EventHPE: Event-Based 3D Human Pose and Shape Estimation","authors":"Shihao Zou, Chuan Guo, Xinxin Zuo, Sen Wang, Pengyu Wang, Xiaoqin Hu, Shoushun Chen, Minglun Gong, Li Cheng","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zou, 2020] Dongqing Zou. Learning Event-Driven Video Deblurring and Interpolation. In Andrea Vedaldi and Horst Bischof, editors, ECCV. 2020.","title":"Learning Event-Driven Video Deblurring and Interpolation","authors":"Dongqing Zou, Zou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}