{"type":"ScholarlyArticle","title":"Self-Supervised Mutual Learning for Dynamic Scene Reconstruction of Spiking Camera","authors":[{"name":"Shiyan Chen","slug":"shiyan-chen","given":"Shiyan","family":"Chen","affiliation":"School of Electronic and Computer Engineering , Peking University"},{"name":"Chaoteng Duan","slug":"chaoteng-duan","given":"Chaoteng","family":"Duan","affiliation":"School of Electronic and Computer Engineering , Peking University"},{"name":"Zhaofei Yu","slug":"zhaofei-yu","given":"Zhaofei","family":"Yu","affiliation":"School of Computer Science , Peking University; Institute for Artificial Intelligence , Peking University"},{"name":"Ruiqin Xiong","slug":"ruiqin-xiong","given":"Ruiqin","family":"Xiong","affiliation":"School of Computer Science , Peking University"},{"name":"Tiejun Huang","slug":"tiejun-huang","given":"Tiejun","family":"Huang","affiliation":"School of Electronic and Computer Engineering , Peking University; School of Computer Science , Peking University; Institute for Artificial Intelligence , Peking University"}],"abstract":"Mimicking the sampling mechanism of the primate fovea, a retina-inspired vision sensor named spiking camera has been developed, which has shown great potential for capturing high-speed dynamic scenes with a sampling rate of 40,000 Hz. Unlike conventional digital cameras, the spiking camera continuously captures photons and outputs asynchronous binary spikes with various inter-spike intervals to record dynamic scenes. However, how to reconstruct dynamic scenes from asynchronous spike streams remains challenging. In this work, we propose a novel pretext task to build a self-supervised reconstruction framework for spiking cameras. Specifically, we utilize the blind-spot network commonly used in self-supervised denoising tasks as our backbone, and perform self-supervised learning by constructing proper pseudo-labels. In addition, in view of the poor scalability and insufficient information utilization of the blind-spot network, we present a mutual learning framework to improve the overall performance of the network through mutual distillation between a non-blind-spot network and a blind-spot network. This also enables the network to bypass constraints of the blind-spot network, allowing state-of-the-art modules to be used to further improve performance. The experimental results demonstrate that our methods evidently outperform previous unsupervised spiking camera reconstruction methods and achieve desirable results compared with supervised methods.","doi":"10.24963/ijcai.2022/396","url":"https://ijcai.org/proceedings/2022/396","pdf":"https://ijcai.org/proceedings/2022/0396.pdf","markdown":"https://ijcai.org/proceedings/2022/396.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/396","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":396,"pages":{"first":2859,"last":2866},"keywords":["Machine Learning: Unsupervised Learning","Machine Learning: Self-supervised Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986520","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984663","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Brandli et al., 2014] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240× 180 130 db 3 µs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10):2333-2341, 2014.","title":"A 240× 180 130 db 3 µs latency global shutter spatiotemporal vision sensor","authors":"Brandli","year":"2014","venue":"IEEE Journal of Solid-State Circuits","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Byun et al., 2021] Jaeseok Byun, Sungmin Cha, and Taesup Moon. FBI-Denoiser: Fast blind image denoiser for poisson- gaussian noise. In CVPR, pages 5768-5777, 2021.","title":"and Taesup Moon. FBI-Denoiser: Fast blind image denoiser for poissongaussian noise","authors":"Byun","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2020] Jonghyun Choi, Kuk-Jin Yoon, et al. Learning to super resolve intensity images from events. In CVPR, pages 2768-2776, 2020.","title":"Learning to super resolve intensity images from events","authors":"Choi","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2017] Siwei Dong, Tiejun Huang, and Yonghong Tian. Spike camera and its coding methods. DCC, 2017.","title":"Spike camera and its coding methods","authors":"Siwei Dong, Tiejun Huang, Yonghong Tian","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2019] Siwei Dong, Lin Zhu, Daoyuan Xu, Yonghong Tian, and Tiejun Huang. An efficient coding method for spike camera using inter-spike intervals. DCC, 2019.","title":"An efficient coding method for spike camera using inter-spike intervals","authors":"Siwei Dong, Lin Zhu, Daoyuan Xu, Yonghong Tian, Tiejun Huang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, pages 3146-3154, 2019.","title":"Dual attention network for scene segmentation","authors":"Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, Hanqing Lu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gallego et al., 2022] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, et al. Event-based vi- sion: A survey. IEEE Transactions on PAMI, 44(1):154-180, 2022.","title":"Event-based vision: A survey","authors":"Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba","year":"2022","venue":"IEEE Transactions on PAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2017] Jing Huang, Menghan Guo, and Shoushun Chen. A dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand. In ISCAS, pages 1-4, 2017.","title":"A dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand","authors":"Jing Huang, Menghan Guo, Shoushun Chen","year":"2017","venue":"ISCAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2019] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In ICCV, pages 603-612, 2019.","title":"Ccnet: Criss-cross attention for semantic segmentation","authors":"Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, Wenyu Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2021] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighbor: Self-supervised de- noising from single noisy images. In CVPR, pages 14781-14790, 2021.","title":"Neighbor2neighbor: Self-supervised denoising from single noisy images","authors":"Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, Jianzhuang Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krull et al., 2019] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In CVPR, pages 2129-2137, 2019.","title":"Noise2void-learning denoising from single noisy images","authors":"Alexander Krull, Tim-Oliver Buchholz, Florian Jug","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Laine et al., 2019] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denois- ing. NeurIPS, 32:6970-6980, 2019.","title":"High-quality self-supervised deep image denoising","authors":"Samuli Laine, Tero Karras, Jaakko Lehtinen, Timo Aila","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lehtinen et al., 2018] Jaakko Lehtinen, Jacob Munkberg, Jon Has- selgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In ICML, pages 2971-2980, 2018.","title":"Noise2noise: Learning image restoration without clean data","authors":"Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, Timo Aila","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Shih-Chii Liu, Bodo Rueckauer, Enea Ceolini, Adrian Huber, and Tobi Delbruck. Event-driven sensing for ef- ficient perception: Vision and audition algorithms. IEEE Signal Processing Magazine, 36(6):29-37, 2019.","title":"Event-driven sensing for efficient perception: Vision and audition algorithms","authors":"Shih-Chii Liu, Bodo Rueckauer, Enea Ceolini, Adrian Huber, Tobi Delbruck","year":"2019","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Masland, 2012] Richard H Masland. The neuronal organization of the retina. Neuron, 76(2):266-280, 2012.","title":"The neuronal organization of the retina","authors":"H Richard, Masland","year":"2012","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nah et al., 2019] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super- resolution: Dataset and study. In CVPR Workshops, pages 0-0, 2019.","title":"Ntire 2019 challenge on video deblurring and superresolution: Dataset and study","authors":"Nah","year":"2019","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2021] Rui Peng, Ronggang Wang, Yawen Lai, Luyang Tang, and Yangang Cai. Excavating the potential capacity of self- supervised monocular depth estimation. In ICCV, pages 15560- 15569, 2021.","title":"Excavating the potential capacity of selfsupervised monocular depth estimation","authors":"Rui Peng, Ronggang Wang, Yawen Lai, Luyang Tang, Yangang Cai","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pini et al., 2018] Stefano Pini, Guido Borghi, and Roberto Vez- zani. Learn to see by events: Color frame synthesis from event and rgb cameras. arXiv preprint arXiv:1812.02041, 2018.","title":"Learn to see by events: Color frame synthesis from event and rgb cameras","authors":"Stefano Pini, Guido Borghi, Roberto Vezzani","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Posch et al., 2008] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt. An asynchronous time-based image sensor. In ISCAS, pages 2130-2133, 2008.","title":"An asynchronous time-based image sensor","authors":"Christoph Posch, Daniel Matolin, Rainer Wohlgenannt","year":"2008","venue":"ISCAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rebecq et al., 2019a] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. In CVPR, pages 3857-3866, 2019.","title":"Events-to-video: Bringing modern computer vision to event cameras","authors":"Rebecq","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rebecq et al., 2019b] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE Transactions on PAMI, 2019.","title":"High speed and high dynamic range video with an event camera","authors":"Rebecq","year":"2019","venue":"IEEE Transactions on PAMI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sheth et al., 2021] Dev Yashpal Sheth, Sreyas Mohan, Joshua L Vincent, Ramon Manzorro, Peter A Crozier, Mitesh M Khapra, Eero P Simoncelli, and Carlos Fernandez-Granda. Unsupervised deep video denoising. In ICCV, pages 1759-1768, 2021.","title":"Unsupervised deep video denoising","authors":"Sheth","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tassano et al., 2020] Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising with- out flow estimation. In CVPR, pages 1354-1363, 2020.","title":"Fastdvdnet: Towards real-time deep video denoising without flow estimation","authors":"Matias Tassano, Julie Delon, Thomas Veit","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, and Angela Yao. To- wards compact single image super-resolution via contrastive self- distillation. In IJCAI, pages 1122-1128, 2021.","title":"Towards compact single image super-resolution via contrastive selfdistillation","authors":"Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, Angela Yao","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wässle, 2004] Heinz Wässle. Parallel processing in the mam- malian retina. Nature Reviews Neuroscience, 5(10):747-757, 2004.","title":"Parallel processing in the mammalian retina","authors":"Heinz Wässle","year":"2004","venue":"Nature Reviews Neuroscience","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2020] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In ECCV, pages 352-368. Springer, 2020.","title":"Unpaired learning of deep image denoising","authors":"Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, Wangmeng Zuo","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2018] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pages 4320-4328, 2018.","title":"Deep mutual learning","authors":"Ying Zhang, Tao Xiang, Timothy M Hospedales, Huchuan Lu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2021] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR, pages 11996-12005, 2021.","title":"Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream","authors":"Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, Tiejun Huang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zheng et al., 2021] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In CVPR, pages 6358-6367, 2021.","title":"High-speed image reconstruction through short-term plasticity for spiking cameras","authors":"Zheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2019] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In ICME, pages 1432-1437. IEEE, 2019.","title":"A retina-inspired sampling method for visual texture reconstruction","authors":"Lin Zhu, Siwei Dong, Tiejun Huang, Yonghong Tian","year":"2019","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2020] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In CVPR, pages 1438-1446, 2020.","title":"Retina-like visual image reconstruction via spiking neural model","authors":"Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, Yonghong Tian","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2021] Lin Zhu, Jianing Li, Xiao Wang, Tiejun Huang, and Yonghong Tian. Neuspike-net: High speed video recon- struction via bio-inspired neuromorphic cameras. In ICCV, pages 2400-2409, 2021.","title":"Neuspike-net: High speed video reconstruction via bio-inspired neuromorphic cameras","authors":"Lin Zhu, Jianing Li, Xiao Wang, Tiejun Huang, Yonghong Tian","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}